LLM Output Validation with Pydantic and Zod: A Production Pattern

The model returned JSON. Your application parsed it without error. Three hours later, a downstream database write failed because the start_date field contained the string "as soon as possible" instead of an ISO date. The model's output was structurally valid JSON. It was semantically wrong. Your parser did not catch it. Your application did not catch it. It propagated into your database.

This is the failure mode that LLM output validation exists to prevent. Here is a practical pattern for implementing it with Pydantic in Python and Zod in TypeScript.

The Problem with Relying on JSON Parsing Alone

JSON parsing only tells you that the output is syntactically valid JSON. It does not tell you that the JSON contains the fields you need, that those fields have the right types, or that the values make sense in context.

A model that returns {"status": "sort of done", "count": "many"} passes JSON parsing with flying colors and breaks your application logic silently. Runtime type checking and semantic validation catch these failures before they propagate.

Schema-First Design: One Source of Truth

The right architecture defines the schema once and derives everything else from it:

1. The JSON Schema passed to the LLM API (for constrained generation)
2. The runtime validator applied to the model's response
3. The TypeScript or Python type used in application code

When these three are separate and maintained independently, they drift. The API call uses an old schema. The validator enforces a different schema. The application code expects a third variant. When a model output fails, you cannot tell which layer is wrong.

In Python with Pydantic

Define your model as a Pydantic class. Use model.model_json_schema() to generate the JSON Schema for the API call. Use Model.model_validate() to parse and validate the model's response.

from pydantic import BaseModel, Field
from typing import Optional
 
class ExtractedContact(BaseModel):
    name: str
    email: Optional[str] = None
    phone: Optional[str] = None
    action_items: list[str] = Field(default_factory=list)

With this model, ExtractedContact.model_json_schema() generates the JSON Schema to pass to the API. ExtractedContact.model_validate_json(response_text) validates the model's response and raises a ValidationError if it does not match. You never maintain two separate schema representations.

Pydantic's ValidationError contains field-level failure messages, which you can format and pass back to the model in a retry prompt. This makes retry logic precise rather than generic.

In TypeScript with Zod

The same pattern applies in TypeScript. Define your schema with Zod, use .toJsonSchema() (via a compatibility library) for the API call, and schema.parse() for runtime validation.

import { z } from "zod";
 
const ExtractedContact = z.object({
  name: z.string(),
  email: z.string().email().nullable(),
  phone: z.string().nullable(),
  action_items: z.array(z.string()),
});
 
type ExtractedContactType = z.infer<typeof ExtractedContact>;

ExtractedContact.parse(parsed_json) throws a ZodError with field-level failure details on validation failure. z.infer derives the TypeScript type — your application code uses the inferred type, so type safety flows from the schema.

Semantic Validation Beyond Type Checking

Type validation catches structural failures. Semantic validation catches value-level failures that are structurally valid but meaningless or incorrect.

Add validators for business rules that the schema cannot express:

• Date fields should be parseable as actual dates
• Numeric fields should be within plausible ranges
• Enum fields should map to known values
• URL fields should be parseable as valid URLs

In Pydantic, use @field_validator decorators. In Zod, use .refine(). These run after structural validation and before the value is used in application code.

Be selective about semantic validators. Over-validation creates false failures when the model's output is correct but in a slightly different format than your validator expects. Date validation that accepts only one ISO format will fail on a model that outputs a different valid ISO representation.

Retry Logic with Validation Feedback

When validation fails, you have two options: fail hard (raise an exception, return an error to the caller) or retry with feedback. Retry with feedback is appropriate when the failure mode is recoverable.

A retry prompt appends the validation failure to the conversation:

"Your previous response failed validation. The field email must be a valid email address or null. You returned "not provided". Please correct this field and return the full object again."

This single-retry pattern resolves most field-level failures without full re-generation. Limit retries to one or two — beyond that, the failure is likely systematic and retrying will not help.

Log every retry and every validation failure with the field-level details. These logs are your best signal for prompt improvements — recurring validation failures on a specific field usually mean the prompt's output specification is ambiguous.

Handling Constrained Decoding

If you use native constrained generation (OpenAI response_format: json_schema with strict mode, or Anthropic tool use with schema enforcement), structural failures are eliminated at generation time. The model cannot produce JSON that does not match your schema structure.

In this case, Pydantic or Zod validation shifts from structural enforcement to semantic validation. You are no longer checking for missing fields or wrong types — constrained generation handles that. You are checking for date formats, range constraints, and business rule compliance.

This is the correct layering: constrained generation for structure, runtime validation for semantics. Both are necessary. Neither is sufficient alone.

What to Do When Validation Keeps Failing

Persistent validation failures on the same field or same prompt usually indicate one of three problems:

The prompt's output specification is ambiguous. The model is making a reasonable interpretation of an underspecified instruction. Clarify the instruction or add an example.

The schema is too strict for the actual data. If email is required but emails are absent in a significant fraction of inputs, make it optional. Forcing the model to populate a field when no data exists causes hallucination.

The model is not capable of this extraction reliably. Test with a larger model. If a larger model succeeds and a smaller one fails consistently, this is a capability issue, not a prompt issue.

In all three cases, the validation failures are giving you diagnostic information. Log them, analyze the patterns, and treat them as prompt feedback rather than noise.

Reliable LLM output validation is the layer between a working prototype and a production system that handles the full range of real-world inputs. Clixo builds that layer as part of every LLM product — talk to us if you are designing for production from the start.