Building an LLM Evaluation Framework from Scratch

You shipped a prompt, users started using it, and now someone changed a single sentence in the system prompt and your extraction accuracy dropped by 15%. You only found out two days later from a support ticket. This is the standard consequence of shipping LLM features without an evaluation framework — changes are invisible, regressions are silent, and debugging is archaeology.

Building an eval framework is the work that makes everything else manageable. Here is how to build one from scratch.

What an LLM Evaluation Framework Actually Is

An eval framework is a repeatable process for answering the question: "Is this version of the prompt or model better or worse than the last?" It consists of four components:

1. A dataset of test cases with known inputs and expected outputs
2. A set of metrics that measure what "better" means for your task
3. An automated runner that executes test cases and scores them
4. A reporting layer that makes results comparable over time

None of this requires a third-party platform to start. A Python script that runs prompts against a CSV of test cases and writes a JSON results file is an eval framework. Add tooling only when the simple version becomes a bottleneck.

Step 1: Define What You Are Evaluating

Before writing a single test case, be explicit about your evaluation objectives. Different tasks need different eval designs.

Extraction tasks (pull structured data from unstructured text): measure field-level accuracy, recall for optional fields, and behavior on inputs that lack the target data.

Classification tasks (route inputs to one of N categories): measure accuracy, precision, recall, and F1 per class. Pay attention to minority class performance — aggregate accuracy hides failures on rare but important inputs.

Generation tasks (summarization, drafting, rewriting): measure task-specific properties. Factual consistency for summarization — does the output contradict the source? Format compliance — did the model follow the length and structure constraints? Goal completion — does the output accomplish the stated purpose?

Agentic tasks (multi-step, tool-using agents): measure per-step accuracy and end-to-end task completion rate. These are harder to automate; start with deterministic sub-tasks.

Step 2: Build Your Dataset

Your dataset is the most important and most underinvested part of any eval framework. A dataset of 20 carefully chosen test cases with correct labels is more valuable than a dataset of 200 cases that all look the same.

Principles for good eval datasets

Cover the distribution, not just the happy path. Include inputs where the expected output is empty, null, or a rejection. If your task involves optional data, include cases where that data is absent. If your task involves ambiguity, include ambiguous inputs.

Include at least one adversarial case per known failure mode. Every time the model fails on a real input, add that input (sanitized of PII) to the eval dataset. Your dataset should grow every time production fails.

Label conservatively. If you are not sure what the correct output should be for a given input, either get a second opinion or exclude that case. Noisy labels produce misleading metrics.

Separate development and holdout sets. Use the development set when designing and iterating on prompts. Keep the holdout set sealed — only run against it when you believe you have a candidate worth measuring. Otherwise you overfit to the holdout.

Step 3: Choose Metrics

For deterministic or near-deterministic outputs

Exact match, field-level accuracy, and F1 are appropriate for extraction and classification. These are easy to compute and unambiguous.

For generation outputs

You need either human review or an LLM judge. LLM-as-judge works well for consistency and coherence checks if you design the judge prompt carefully and use a model larger than the one being evaluated. The judge prompt should include a scoring rubric, not just "rate this output."

Run LLM judges on a sample of your dataset — not every case on every run. It costs money and adds latency to your eval pipeline.

Behavioral metrics

Beyond accuracy, track: refusal rate (is the model refusing valid inputs?), format compliance rate (is the output in the requested format?), and latency distribution. These operational metrics matter for production viability even when accuracy is acceptable.

Step 4: Automate the Runner

Your runner needs to do three things: load the dataset, call the model with each input, and score each output against the expected result. In Python:

• Load test cases from a CSV or JSON file
• For each case, call the LLM API with the prompt under test
• Score the response using your metric functions
• Write results to a timestamped output file

Pin the model version explicitly in your runner. Evals against a floating model alias (gpt-4o-latest) are not reproducible. If the model updates, your metrics change for reasons unrelated to your prompts.

Step 5: Integrate with CI/CD

An eval framework that requires manual execution will not be run consistently. The goal is automatic execution on every prompt or code change.

Set up your runner as a CI step. Configure it to fail the build if core metrics drop below a threshold. Start with a conservative threshold — your first goal is making evals run automatically, not catching every regression.

Gate prompt-change PRs on eval results. A PR that changes a prompt and does not include a description of its effect on eval metrics should not merge.

Common Mistakes When Building Eval Frameworks

Building the runner before the dataset. A runner with no good dataset tells you nothing. Invest in dataset quality first.

Measuring only what is easy to measure. Exact match is easy. Semantic correctness is hard. Do not let the easy metric crowd out the important one.

Not tracking results over time. A single eval run is a snapshot. Value comes from comparison. Store results with timestamps and prompt/model version metadata.

Treating evals as a one-time setup. Evals need maintenance. When your product changes, the dataset and metrics need to change with it.

Clixo builds LLM-powered products with eval infrastructure from the start, not retrofitted after problems surface. Start a conversation about what that looks like for your use case.