LLM evaluation
Software that tests AI model and agent outputs against datasets and scoring rules.
Includes: Evaluation frameworks, red-teaming and benchmark harnesses.
For example: promptfoo, Ragas, DeepEval.
10 listed products
| Product | What it says it does |
|---|---|
| Autoevals | AutoEvals is a tool for quickly and easily evaluating AI model outputs using best practices. |
| Evidently | Evaluate, test, and monitor LLMs, RAG applications, AI agents, and ML models in a single framework. |
| Giskard | Test your AI agents to detect issues before users or hackers do, and protect them continuously. |
| OpenAI Evals | Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks. |
| Ragas | Ragas is a library that helps you move from "vibe checks" to systematic evaluation loops for your AI applications. |
| Inspect | An open-source framework for large language model evaluations |
| LM Evaluation Harness | A framework for few-shot evaluation of language models. |
| promptfoo | Automated testing that finds & fixes AI risk in development |
| DeepEval | The LLM Evaluation Framework |
| TruLens | TruLens: Evals and Tracing for AI Agents |
10 of these are read from their own pages. A product appears where AirStack's category list places it; 7,865 of 13,386 listed products carry no category yet.