AI Campus • Practical resources
Automated LLM Evaluation & Benchmark Suite
Create a repeatable saved-output regression harness, then extend it with human ratings and operational measurements.
Build guide and prerequisites
Follow this roadmap to implement and validate your own application, then record your observed results.
- Python and JSON basics.
- Saved baseline/candidate outputs and a small manually labeled task set.
Required tools and free options
- Python
- JSON
- Optional SQLite
- Optional charting frontend
Cost: The supplied deterministic starter runs offline without model calls. Evaluating saved answers is free of API fees; generating new outputs or using an LLM judge may incur separate costs.
Start with a smaller working example
Downloadable offline output evaluator →This starter checks required and forbidden phrases with exit codes. It does not evaluate semantic truth, faithfulness or comprehensive model quality, and it does not call an LLM judge.
Build it step by step
1. Specify cases before candidate changes
Create ten stable case IDs with inputs, expected properties and known failure examples. Cover normal, ambiguous, unsupported and adversarial inputs. Freeze a holdout set that is not used to tune the prompt.
2. Run the deterministic starter
Download the evaluator and follow its cases.json and answers.json example. Confirm a known good answer exits 0, a deliberately wrong answer exits 1 and invalid input exits 2.
3. Compare saved output versions
Save baseline and candidate answers under different filenames, with the same case IDs and prompt/model metadata. Run both against identical checks. Do not regenerate answers silently during evaluation.
4. Add task-specific review
For citations, resolve source IDs and manually compare claims to passages. For structured output, validate JSON shape and required fields. Have a reviewer score a blinded sample against a written rubric; phrase matches alone are not truth assessment.
5. Capture operational measurements honestly
Record latency, errors and token usage only when actually observed during generation. Leave unavailable metrics blank rather than inventing zeros. Calculate estimated costs separately using the applicable provider rate and date, if a paid provider is later used.
6. Publish a bounded comparison
Create a table or dashboard with pass counts, changed cases, reviewer notes and missing metrics. Define acceptance thresholds before looking at the candidate. Retain examples of regressions; do not turn one aggregate score into a universal quality claim.
Expected output and validation
A reproducible comparison of saved outputs with transparent deterministic checks and optional human review. A complete LLM-as-judge benchmark is an extension, not supplied code.
- Deliberate regressions fail and malformed suites cannot pass.
- Case IDs align across versions and holdout cases stay separate.
- Reports distinguish phrase checks, human ratings and measured operational metrics.
Official references
Free to read. Software setup, model access and hosted services have their own requirements. References checked 3 November 2026.