Tool Results Fill Every Later Turn: Truncate Before You Append

The model asked how many open cases EMEA had. The tool ran List rows with no row cap, serialized every column, and appended the blob as tool_result. The next messages.create was rejected: the prompt no longer fit. The user saw a generic failure. The cases were never counted. A tool result is not a log line. It is input on every later request.

The tool-use loop already tells you to return errors as tool_result and to honor stop_reason. This post is what happens after a successful call whose payload you should not have kept.

The transcript is the prompt

Every Messages API call sends the whole messages array again. System prompt, tool definitions, every assistant turn, every tool_result. There is no server-side log the model rereads for free. If you append a dump, you pay for that dump on the next turn, and the one after that, until you remove it.

That is a different tax from streaming. Streaming is about when a block is complete. This is about whether the completed block should exist in this shape at all.

A count question wants a number. A "find the account" question wants one id and a name. A "what did the ticket say" question wants the body, not the audit history, not @odata.context, not fifty sibling rows. The tool decides the shape. The model cannot un-send tokens you already put in the array.

Cap at the tool, not in the next prompt

Truncation after the fact is a safety net. The fix is the return value.

PAGE_SIZE = 20

def list_accounts(name_contains: str, page: int = 1) -> dict:
    rows, total = query_accounts(name_contains, page, PAGE_SIZE)
    return {
        "page": page,
        "page_size": PAGE_SIZE,
        "total": total,
        "has_more": (page * PAGE_SIZE) < total,
        "accounts": [
            {"id": row.id, "name": row.name, "number": row.account_number}
            for row in rows
        ],
    }

total answers "how many." The page answers "which ones." has_more tells the model to call again with page + 1 instead of inventing a wider filter. Columns are the ones a later get_account would need as arguments. The rest stays in Dataverse.

Do the same for files, HTML, and stack traces. Strip a browse result to text, then cap. An error tool_result is one sentence: which argument was wrong, and which tool to call instead. That is the same routing idea as tool descriptions. A 40-line exception does not teach the next call.

If a payload still arrives oversized — a document the user pasted, a connector you do not own — cap it before messages.append, and say that you capped it:

MAX_CHARS = 8000

def cap_tool_result(text: str) -> str:
    if len(text) <= MAX_CHARS:
        return text
    omitted = len(text) - MAX_CHARS
    return (
        text[:MAX_CHARS]
        + "\n\n[truncated: "
        + str(omitted)
        + " characters omitted. Ask for a narrower filter, a later page, or get-by-id.]"
    )

The note is load-bearing. Silent truncation makes the model treat a partial list as complete and skip the page that had the row.

MCP servers have a cleaner split when the blob is a document the host can pin: put it in a resource, not in a tool result. Resources can be omitted from the next request. Tool results cannot, not without surgery on the transcript.

Pairing is the constraint on surgery

You may shorten a tool_result. You may not orphan it.

An assistant turn that contains tool_use blocks must be followed by a user turn that contains a tool_result for every id from that turn, and only those ids. Drop the assistant turn and keep the results: the API rejects the request. Drop a result and keep the tool_use: same rejection. Invent a tool_result for a call that never closed — the streaming failure mode — and you are sending an id whose input never existed.

Compact content, not structure.

STUB = "[omitted: result was used in an earlier turn. Call the tool again if you need the rows.]"

def stub_old_tool_results(messages: list, keep_last: int = 2) -> None:
    """Replace tool_result bodies on completed turns, oldest first. Pairing stays."""
    tool_user_idxs = [
        i for i, m in enumerate(messages)
        if m["role"] == "user" and _is_tool_result_turn(m)
    ]
    drop_before = max(0, len(tool_user_idxs) - keep_last)
    for i in tool_user_idxs[:drop_before]:
        content = messages[i]["content"]
        if isinstance(content, str):
            continue
        for block in content:
            if block.get("type") == "tool_result":
                block["content"] = STUB

_is_tool_result_turn is "this user message is the reply to tool_use," not "any user message." Do not stub the live turn: the model has not read those results yet.

If that assistant turn also carried thinking blocks, leave them on the assistant message you keep. Stubbing results does not give you permission to strip thinking or redacted_thinking. The signature still has to round-trip for the current loop.

Do not compact in the middle of a loop. stop_reason is tool_use means the next user message is the results. Summarize only after end_turn, or after you have already attached every result for the latest assistant turn.

Summaries replace turns, they do not splice them

When the conversation is long, stubbing bodies is not enough. You drop a prefix of completed user/assistant pairs and replace them with one pair that states what was already decided.

user: [summary of turns 1–8: EMEA open-case count is 41; account US-001 is id 7f3a…]
assistant: Acknowledged. Continue from this summary.
user: (the current question)

Rules that keep this from 400ing:

• The summary pair is ordinary text. It is not a tool_result with no matching tool_use.
• You drop whole turns. You do not keep an assistant tool_use whose user reply you deleted.
• The live question stays. The summary is context, not a substitute for the thing the user just typed.
• Indirect prompt injection still applies: a stub or a summary of untrusted tool output is still untrusted. Do not promote it into the system prompt.

A unique key on the business record is a better memory than a transcript of every list. Once the model has accountid, later turns should get by id, not resend the search page. If the workflow is always search-then-get-then-act, that order belongs in code, not in a growing dump.

Mutation breaks the cache prefix

Prompt caching keys on the exact prefix. Stubbing a tool_result in the middle of messages changes bytes behind the current turn. Everything from that block forward is a cache miss.

Two consequences:

• Cap on the way in. A result that was always 20 short rows stays a stable prefix for later turns. A result you later rewrite from 200kb to a stub saves the window and spends a cache write.
• Drop-from-the-oldest on every message, without a stable summary breakpoint, defeats caching the same way the caching post already warns about. Summarize at a fixed seam (system prompt, then summary, then recent turns) so the seam can be marked cache_control.

Tool definitions are part of that prefix. A list_accounts schema that already has page and page_size does not need a second tool named list_all_accounts. Two list tools that both say "accounts" is how you get the dump you were trying to prevent.

What to return from a list

Question	Return
How many?	total, no rows
Which ones match X?	One page of id, name, key; has_more
Tell me about this one	get by id, selected columns
The PDF / the page / the trace	Extracted slice, then cap_tool_result
The tool failed	One-line is_error result, not a stack

first() of an uncapped list is the same class of bug as stuffing a business key into Get a row by ID: you asked the wrong API to mean "find." Here the wrong API is "return the table."

Retries do not shrink the payload. A context-length rejection is not a blip. Raising the retry count on messages.create only repeats a prompt that already does not fit. Shrink the array, then send.

The model in the first paragraph never answered "how many" because the tool answered "here they all are." Return total. Cap anything that still looks like a document. When you must forget older results, stub bodies or replace completed turns — never break tool_use / tool_result pairing, and never do it while the current loop is still waiting on ids. Then the next request fits, and the count is a number instead of a 400.