LLM evals
Tests the quality of AI outputs against datasets and rules, so you know if a change made answers better or worse.
Search AirStackWhy teams use it
- Catch regressions before a prompt or model change ships
- Compare models on your own tasks, not public benchmarks
- Score answers for accuracy, safety and format
- A shared quality bar for product and engineering
Who uses it: AI engineer, ML engineer, QA engineer, Data scientist, Product manager, AI safety researcher. Watch for: A model grading another model can be wrong too; check a sample of scores by hand before you trust them. AirStack’s guide to the category.
10 products
| Product | What it says it does | Starting at |
|---|---|---|
| DeepEval | The LLM Evaluation Framework | Not observed |
| promptfoo | Automated testing that finds & fixes AI risk in development | Not observed |
| Giskard | Test your AI agents to detect issues before users or hackers do, and protect them continuously. | Not observed |
| Evidently | Evaluate, test, and monitor LLMs, RAG applications, AI agents, and ML models in a single framework. | Not observed |
| LM Evaluation Harness | A framework for few-shot evaluation of language models. | Not observed |
| TruLens | TruLens: Evals and Tracing for AI Agents | Not observed |
| OpenAI Evals | Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks. | Not observed |
| Autoevals | AutoEvals is a tool for quickly and easily evaluating AI model outputs using best practices. | Not observed |
| Inspect | An open-source framework for large language model evaluations | Not observed |
| Ragas | Ragas is a library that helps you move from "vibe checks" to systematic evaluation loops for your AI applications. | Not observed |
Prices are list prices as written on each pricing page; hover one for its quote. “Not observed” means AirStack has not found it on that product’s pages.