Four techniques that give Claude a reliable output shape
1. Wrong shape of output (got a sentence, wanted JSON)
You didn't tell Claude how to format the answer — only what to answer. Claude fills that gap with whatever looks reasonable. Fix: add an explicit output constraint ("respond only in JSON with these fields: ...").
A developer wants Claude to label support tickets as BILLING, TECHNICAL, or ESCALATION. The first attempt is just:
"You are a support classifier. Classify the ticket."
Claude returns "Billing" sometimes, "billing" other times, and occasionally a full sentence. The downstream code expects an exact label — so it breaks. This is exactly the first row explains : wrong output shape, missing an output constraint
2. Content drifts over a conversation
Your instructions were loose enough that Claude gradually shifted tone, scope, or focus across turns. Fix: write a proper system prompt that locks in the role, boundaries, and format rules upfront — those rules then apply to every single response, not just the first one.
Imagine you're building a customer support bot for a software product. Your prompt is just:
"You are a helpful assistant. Answer the user's questions."
Turn 1 — User asks about a billing issue. Claude answers correctly, stays on topic.
Turn 3 — User casually mentions they're stressed about work. Claude starts offering life advice and a sympathetic tone.
Turn 5 — User asks a vague question. Claude now answers it like a general-purpose assistant, not a support agent — recommending Google searches, going off-topic.
Nothing in the prompt said stay scoped to software support only, don't shift tone, don't answer questions outside this domain. So Claude drifted — not because it misunderstood, but because there was no standing rule to hold it in place across turns.
Fix: A system prompt that locks the contract:
"You are a support agent for Acme Software. Answer only questions about billing, accounts, and product features. Do not offer personal advice. Keep responses under 3 sentences."
Now those rules apply to every single response, regardless of where the conversation wanders.
3. Structure is right but invented (Few-shot)
Claude understood the task but made up a structure you never asked for. The problem is that describing a structure in words is imprecise — Claude interprets the description, not the actual shape you have in mind. Fix: give it one concrete example of an input and the exact output you want. Showing beats telling here.
Few-shot examples show Claude the exact label and casing to return, and XML tags keep those examples separate from the instruction so Claude does not read them as part of the task:
XML tags — the examples are wrapped in <sample_input> / <ideal_output> tags so Claude can clearly see where the instruction ends and where each example starts and stops. Without that separation, Claude might treat the examples as part of the task description.
Wrap them with descriptive tag names like <my_code> and <docs> and the boundary becomes unambiguous. You do not need to use official XML tag names; descriptive names that match your content work best.
4. Works on your test cases, breaks on edge cases
You validated the prompt against inputs you thought of, so it handles those fine. But the prompt has no rule for anything outside that set. Fix: when you spot a breaking variant, name it explicitly in the prompt ("if the field is empty, return null") or add an example that covers it.
Back to the ticket classifier. You've added the output constraint and few-shot examples. You test it on 20 tickets — works perfectly. You ship it.
A week later the router starts breaking. The input coming in looks like this:
"I don't know, maybe it's a billing thing? Or maybe technical? I'm not sure."
Claude returns: BILLING / TECHNICAL — because nothing in the prompt said what to do when a ticket is ambiguous. Your router expects exactly one label, so it breaks.
Your prompt handled the happy path — clear, unambiguous tickets. But it had no rule for the case it never saw during testing.
Fix: Name the variant explicitly in the prompt:
"If the ticket is ambiguous or spans multiple categories, return ESCALATION."
Or add a few-shot example that covers it:
<sample_input>Not sure if this is a billing or account issue.</sample_input>
<ideal_output>ESCALATION</ideal_output>
The prompt wasn't wrong — it just had a gap that your test inputs never exposed. The fix is closing that specific gap, not rewriting the whole prompt.
The common thread: Claude can only follow rules you actually wrote down. Every gap in the prompt is a decision Claude makes for you — sometimes correctly, sometimes not.
No comments:
Post a Comment