Put Less in the Prompt, On Purpose

The call includes the system prompt, the whole chat, three policy PDFs, the user directory, and a request to "be careful." The answer misses the one clause that mattered, which was in the middle of the second PDF. The model was not asked a question it could locate. It was handed a pile.

Vendors publish different context limits, and those limits change. Do not write a design that depends on a number from a blog post. Write a budget for this feature: how much instruction, how much retrieved text, how much history, and what must never be included. When the budget is full, drop or summarize by a rule you can test, not by hoping the model will notice the important part.

A budget for one call

• Instructions: the task, the output shape, and what to do when the input is insufficient. Short enough that a reviewer can read them.
• Evidence: the passages or fields selected for this question, with ids, not the entire corpus.
• History: the turns that change the meaning of this turn. A long chat can be stored in your database and summarized or trimmed.
• Tools: the definitions the model may call on this step, not every API you own.
• Excluded: secrets, other tenants, raw credentials, and documents the caller may not see.
• A test: a question whose answer is only in a passage you included, and a question whose answer is only in a passage you excluded. The second must abstain or refuse.

Selection is the product. An Embedding Index Is Not Your Database is how you find candidates. Permission checks decide which candidates may enter the prompt. Cost controls decide whether you can afford the call. This decision is which bytes earn a place. Stuffing is not thoroughness. It is an unreviewed concat.

Illustrative scenario

This scenario is illustrative. A policy assistant retrieves at most four passages, each under a length cap, from documents the caller can open. The instruction says to answer only from those passages and to say when they do not contain the answer. The prior conversation is the last few turns, not the full year. A regression test asks a question whose only source was not retrieved and expects a refusal, which is the behavior in When AI Should Refuse. Nobody raises the provider’s maximum context in order to skip the cap. The cap is the design.

Checklist

• Write the budget: instruction, evidence, history, tools.
• Retrieve or load by ids. Do not paste the database.
• Apply permission before the text enters the prompt.
• Keep secrets and cross-tenant data out.
• Test a case that should be answered and a case that should not.
• Record which ids were sent, so a bad answer can be traced to a selection, not to a vibe.

Questions that usually come next

Should we use the largest context available?

Use the smallest set that contains the evidence for this step. A larger window can hold more mistakes, more stale text, and more data the caller should not see. Raise the budget when a measured task needs it, not as a default.

Does a summary of history count as the history?

A summary is another model output. It can drop a constraint the user stated. Keep the original turns you still need, or accept that the summary is lossy and test the losses you care about.