LLM Production Readiness Checklist: What to Verify Before Shipping

Shipping an LLM feature to production is different from shipping a conventional API integration. The failure modes are subtler, the behavior is non-deterministic, and the ways things go wrong are harder to catch with standard testing practices. The following checklist captures what needs to be in place before real users depend on your LLM system.

This is not a theoretical framework. It is a working checklist oriented toward the team about to push the deploy button.

Prompts and Model Configuration

• All prompts are in version control. No prompts live only in environment variables, dashboards, or developer notebooks.
• Model versions are pinned. Production calls specify an exact model version, not a floating alias.
• Temperature and other sampling parameters are set explicitly for every production prompt, not left at defaults.
• Output format is fully specified — field names, types, and behavior when data is absent are defined in the prompt, not left to the model's discretion.
• Context window budget is documented. The maximum expected input size plus the fixed prompt overhead leaves sufficient headroom.
• System prompts and user message templates are separated cleanly in code — no string concatenation across concerns.

Evaluation Coverage

• An eval dataset exists with at least 20 representative test cases including absent-data and adversarial cases.
• Eval results are documented for the current prompt version. You have a baseline.
• A regression test run is part of the deployment process. Prompt changes do not go to production without running evals.
• Key metrics are defined before launch — accuracy, format compliance, refusal rate, or whatever is task-appropriate. You know what "worse" means.
• Edge cases from the task domain are included in the eval set, not just clean representative samples.

Error Handling and Reliability

• Retry logic is implemented with a sensible maximum retry count and backoff. Not every retry is appropriate — the strategy is documented.
• Fallback behavior is defined. What happens when all retries are exhausted? Does the feature degrade gracefully or fail loudly?
• Timeout values are set on all LLM API calls. Long-running requests that never return are a real failure mode.
• The application handles provider outages without crashing. Circuit breakers or graceful degradation are in place.
• Streaming error handling is implemented if you use streaming — mid-stream failures are handled differently from full-response failures.
• Rate limit errors are handled explicitly — not caught by a generic exception handler.

Observability

• Raw model inputs and outputs are logged for every production call (with PII stripped or redacted as appropriate).
• Per-prompt token usage is tracked — input tokens, output tokens, and model version.
• Latency is instrumented — time-to-first-token and total response time are measured and stored.
• A cost dashboard or alert exists so unexpected spend spikes surface before the monthly bill.
• Output quality signals are collected where possible — user feedback, downstream success rates, format compliance rates.

Cost Controls

• Prompt caching is enabled for prompts with long, stable prefixes.
• Output length constraints are set via max_tokens and explicit instructions.
• Model selection is deliberate — you have verified that the chosen model tier is necessary for this task, not just the default.
• Batch API is used for offline or asynchronous processing where real-time is not required.
• A per-user or per-session rate limit exists to prevent runaway costs from a single user or misconfigured client.

Safety and Output Quality

• The model cannot be prompted into revealing system prompt contents by users. Test this explicitly.
• User-provided inputs are validated and sanitized before injection into prompts. Injection attacks via user inputs are a real risk.
• The model's output is not rendered as HTML or executed as code without validation. If users can influence prompt inputs, treat model outputs as untrusted.
• Refusal and error responses from the model are handled — the application does not treat a model refusal as a valid structured output.
• Content policy compliance is reviewed for your use case and provider. Some use cases require explicit approval.

Deployment and Operations

• Prompt changes go through code review like any other code change.
• A rollback plan exists. If a prompt change degrades production quality, how quickly can you revert?
• Staging environment mirrors production model configuration — same model version, same temperature, same prompt version.
• Documentation exists for each production prompt: what it does, what model it uses, what the eval baseline is, and who owns it.
• On-call responsibilities are clear for LLM-related incidents. Someone knows how to diagnose and address a production quality degradation.

After Launch

Production readiness does not end at launch. Plan for:

• Monthly prompt reviews against accumulated production failure cases
• Eval suite expansion as new edge cases surface
• Model version evaluation when providers release updates
• Cost trend monitoring and optimization cycles

A checklist like this is most useful when it prompts honest answers, not checkbox theater. If an item is unchecked because it was deprioritized, document that explicitly and create a follow-up task.

If your team is building an LLM feature and wants a technical review of your production readiness before launch, Clixo offers architecture and engineering support for exactly this.