Common LLM Prompting Mistakes That Kill Accuracy in Production
Avoid the most damaging LLM prompting mistakes in production — from underspecified outputs to missing edge case handling — with practical fixes for each.
Most LLM accuracy problems are not model problems — they are prompt problems. The model is doing exactly what the prompt says. The prompt just does not say what the developer thought it said. These mismatches are invisible in prototypes and expensive in production.
Here are the mistakes that appear most often, and what to do instead.
Mistake 1: Underspecifying the Output Format
"Return the result as JSON" is not a sufficient output instruction. Which fields? What types? What should be returned when a field is absent? What format should dates use?
Underspecified output instructions produce output that looks correct in testing and drifts in production as the model encounters input distributions it was not tested on. The model fills in its own judgment for unspecified details, and that judgment is not consistent.
Fix: Define the complete output schema. Use JSON Schema notation or an explicit field-by-field specification in the system prompt. Include an example output. Use native structured output APIs (constrained decoding) so the schema is enforced at the token level, not just requested.
Mistake 2: No Instruction for What to Do When Data Is Absent
"Extract the customer's phone number from the following email" works when the email contains a phone number. What happens when it does not? Most prompts do not specify. The model will either hallucinate a plausible phone number or produce a response that breaks downstream parsing.
Fix: Add explicit absent-data instructions. "If the email does not contain a phone number, return null for that field." Test absent-data cases explicitly in your eval suite — they are often more common in production than developers expect.
Mistake 3: Accumulating Contradictory Instructions
Prompts grow over time. A rule gets added to handle one failure case. Then another rule handles an exception to that rule. Then a third rule adds nuance. After six months, the prompt contains instructions that contradict each other, and the model's behavior becomes unpredictable because it is resolving conflicts in ways you did not anticipate.
Fix: Audit prompt instructions periodically. When adding a new rule, check whether it conflicts with existing ones. If you find yourself writing "except when..." more than once, consider whether the exception should be handled in application logic instead of in the prompt.
Mistake 4: Using Vague Qualitative Instructions
"Be concise." "Be professional." "Be helpful." These instructions are not executable. What counts as concise? The model has no calibration point. Different inputs produce wildly different output lengths and tones, all of which technically comply with the instruction.
Fix: Replace qualitative instructions with quantitative or behavioral ones. "Respond in two sentences or fewer." "Use formal language — no contractions, no first-person." "If the answer requires more than 200 words, summarize the key point in the first sentence and elaborate below." Specific instructions produce consistent behavior.
Mistake 5: Testing Only on Clean, Representative Inputs
Developers test with inputs that look like what they expect to receive. The model performs well on those inputs. Production has inputs from users who misunderstand the interface, paste in the wrong content, write in unexpected languages, or submit edge cases the developer never considered.
Fix: Build adversarial cases into your eval suite deliberately. What happens if the user submits an empty string? A 10,000-word document? A document in a different language? An input that partially matches the expected format but includes irrelevant sections? Testing only clean inputs means your accuracy metrics overstate production performance.
Mistake 6: Floating Model Versions in Production
Pointing your production prompt at a floating alias like gpt-4o-latest or claude-3-5-sonnet-latest means the model can change under you at any time. Providers update these aliases, and model updates can change behavior on your specific prompts even when overall benchmarks improve.
Fix: Pin model versions in production. When a new model version is released, test it against your eval suite before updating. Treat a model upgrade the same way you treat a dependency upgrade — with an explicit test cycle, not automatic adoption.
Mistake 7: Ignoring the Context Window Budget
Long system prompts, large few-shot example sets, and injected documents can consume a large share of the context window before the user's input arrives. When inputs are long too, the total may approach or exceed the context limit, causing the model to truncate or degrade in ways that are hard to diagnose.
Fix: Track your token budget explicitly. For each prompt, know the fixed cost (system prompt + examples) and the variable cost (typical input). Ensure there is headroom for both typical and large inputs. If your system prompt has grown past 1,000-2,000 tokens, audit it for removable content.
Mistake 8: No Observability on Model Outputs
If you are not logging what the model actually returns in production, you cannot know when behavior degrades. Silent degradation — the model starts returning technically valid but semantically wrong outputs — is the hardest failure mode to catch without observability.
Fix: Log the raw model response for every production call. Log input token count, output token count, and latency. Set up anomaly detection on output length and format compliance. A sudden increase in output length often signals the model has started producing explanations or caveats it was not asked for.
Mistake 9: Treating the Prompt as Read-Only After Launch
A prompt deployed to production is not finished — it is in its first production iteration. User input distributions shift. The model provider updates the underlying model. New edge cases surface. Treating the prompt as read-only means these changes accumulate as silent accuracy degradation.
Fix: Schedule prompt reviews as a recurring engineering task. Treat accuracy metrics from production as signals, not just launch-time gates. When metrics move, investigate whether the prompt needs updating.
Avoiding these mistakes is easier when you have an eval framework and proper observability in place before launch, not retrofitted after. Clixo builds LLM systems designed to stay accurate over time — get in touch to discuss your architecture.