Tools for evaluating machine-learning models, prompts, agents, retrieval systems, and large language models.
A
Arize Phoenix
Use case: Trace, evaluate, and troubleshoot LLM and machine-learning applications.
Type: Open-source AI observability and evaluation platform
Access: Free and open source with commercial services
What It Does
According to the vendor or project, the tool's stated purpose is to trace, evaluate, and troubleshoot LLM and machine-learning applications.
Positive Community Commentary
Practitioners often recommend Phoenix when they want open-source tracing and evaluation without tying an application to one framework. Users particularly value inspecting multi-step agent traces, adding custom evaluators, and self-hosting the stack, which gives teams transparent debugging and control over their evaluation data.
Things to Know
- Product or project type: Open-source AI observability and evaluation platform.
- Current access model: Free and open source with commercial services.
- Review the official page for current platforms, integrations, system requirements, licensing, and product changes.
Visit the Arize Phoenix website
Information checked: August 5, 2026
Back to top
B
Braintrust
Use case: Manage datasets, experiments, evaluations, prompts, and production traces.
Type: Commercial AI evaluation platform
Access: Free and paid plans
What It Does
According to the vendor or project, the tool's stated purpose is to manage datasets, experiments, evaluations, prompts, and production traces.
Positive Community Commentary
Teams using Braintrust often highlight how quickly they can turn representative examples into datasets and compare prompt or model changes. Practitioners value its experiment history, trace inspection, and regression visibility, especially for classification and other workflows where output consistency matters across many runs.
Things to Know
- Product or project type: Commercial AI evaluation platform.
- Current access model: Free and paid plans.
- Review the official page for current platforms, integrations, system requirements, licensing, and product changes.
Visit the Braintrust website
Information checked: August 5, 2026
Back to top
D
DeepEval
Use case: Write automated evaluations for LLM applications using test-style metrics.
Type: Open-source LLM evaluation framework with hosted platform
Access: Free and open source with paid plans
What It Does
According to the vendor or project, the tool's stated purpose is to write automated evaluations for LLM applications using test-style metrics.
Positive Community Commentary
Developers appreciate DeepEval’s pytest-style workflow because LLM quality checks can live beside ordinary automated tests and run in CI. Community discussions often mention its broad metric set and custom evaluators, which help teams compare variants, explain individual scores, and catch regressions without building an evaluation harness from scratch.
Things to Know
- Product or project type: Open-source LLM evaluation framework with hosted platform.
- Current access model: Free and open source with paid plans.
- Review the official page for current platforms, integrations, system requirements, licensing, and product changes.
Visit the DeepEval website
Information checked: August 5, 2026
Back to top
DeepTeam
Use case: Red-team LLM applications and AI agents for security and safety weaknesses.
Type: Open-source Python LLM red-teaming framework
Access: Free and open source under the Apache License 2.0; model-provider costs may apply
What It Does
According to the project, DeepTeam helps teams test LLM systems for vulnerabilities such as prompt injection, bias, misinformation, harmful content, and personally identifiable information leakage. It supports reusable vulnerabilities, adversarial attack methods, single-turn and multi-turn testing, and assessments aligned with OWASP guidance and the NIST AI Risk Management Framework.
Positive Community Commentary
DeepTeam is best suited to teams performing security and safety red-team assessments rather than ordinary answer-quality evaluation. It complements general LLM evaluation frameworks such as DeepEval.
Things to Know
- Product or project type: Python framework for LLM and agent red teaming.
- Current access model: Free and open source under the Apache License 2.0.
- Supports configurable attacks, vulnerabilities, metrics, and model callbacks.
- External model or service usage may create additional costs.
- Review the official documentation for current supported attacks, vulnerabilities, and integrations.
Visit the DeepTeam website
Information checked: September 21, 2026
Back to top
G
Garak
Use case: Probe language models for security, safety, and robustness weaknesses.
Type: Open-source LLM vulnerability scanner
Access: Free and open source
What It Does
According to the vendor or project, the tool's stated purpose is to probe language models for security, safety, and robustness weaknesses.
Positive Community Commentary
Security practitioners value Garak as a practical way to probe models with a broad library of attacks rather than inventing every adversarial prompt manually. Its extensible probes and detectors are used for repeatable red-team baselines, research experiments, and comparisons of how different models respond to the same risk scenarios.
Things to Know
- Product or project type: Open-source LLM vulnerability scanner.
- Current access model: Free and open source.
- Review the official page for current platforms, integrations, system requirements, licensing, and product changes.
Visit the Garak website
Information checked: August 5, 2026
Back to top
Giskard
Use case: Test machine-learning and LLM applications for quality, safety, and bias.
Type: Open-source AI testing framework with commercial platform
Access: Free and open source with paid plans
What It Does
According to the vendor or project, the tool's stated purpose is to test machine-learning and LLM applications for quality, safety, and bias.
Positive Community Commentary
Practitioners looking for a more guided evaluation workflow often point to Giskard’s ready-made checks and test-generation approach. It is especially useful when teams want to turn business rules, RAG examples, and known failure patterns into repeatable quality and security tests that domain experts can review with engineers.
Things to Know
- Product or project type: Open-source AI testing framework with commercial platform.
- Current access model: Free and open source with paid plans.
- Review the official page for current platforms, integrations, system requirements, licensing, and product changes.
Visit the Giskard website
Information checked: August 5, 2026
Back to top
H
Helicone
Use case: Monitor, evaluate, and analyze requests to language-model providers.
Type: Open-source LLM observability platform
Access: Free and paid plans
What It Does
According to the vendor or project, the tool's stated purpose is to monitor, evaluate, and analyze requests to language-model providers.
Positive Community Commentary
Developers often describe Helicone as a straightforward way to add request logging, cost tracking, latency visibility, and provider analytics with little application change. It is valued by teams that want a quick operational view of model calls and a central place to inspect requests before adopting a more elaborate evaluation stack.
Things to Know
- Product or project type: Open-source LLM observability platform.
- Current access model: Free and paid plans.
- Review the official page for current platforms, integrations, system requirements, licensing, and product changes.
Visit the Helicone website
Information checked: August 5, 2026
Back to top
I
Inspect AI
Use case: Build and run reproducible evaluations of language models and agents.
Type: Open-source LLM evaluation framework
Access: Free and open source
What It Does
According to the vendor or project, the tool's stated purpose is to build and run reproducible evaluations of language models and agents.
Positive Community Commentary
Evaluation practitioners value Inspect AI for its composable task, solver, scorer, tool, and sandbox model, which supports reproducible work from simple benchmarks through agent evaluations. Its transparent transcripts, growing evaluation registry, and adoption by research organizations make it useful when rigor and inspectable execution matter more than a polished hosted dashboard.
Things to Know
- Product or project type: Open-source LLM evaluation framework.
- Current access model: Free and open source.
- Review the official page for current platforms, integrations, system requirements, licensing, and product changes.
Visit the Inspect AI website
Information checked: August 5, 2026
Back to top
L
Langfuse
Use case: Trace, evaluate, monitor, and manage prompts for LLM applications.
Type: Open-source LLM engineering platform
Access: Free open-source edition and paid cloud plans
What It Does
According to the vendor or project, the tool's stated purpose is to trace, evaluate, monitor, and manage prompts for LLM applications.
Positive Community Commentary
Practitioners frequently recommend Langfuse when self-hosting, data control, and framework-neutral tracing are priorities. Teams value the ability to follow complete LLM and agent traces, monitor token use and latency, manage prompts, and add evaluations while keeping the core observability platform in their own infrastructure.
Things to Know
- Product or project type: Open-source LLM engineering platform.
- Current access model: Free open-source edition and paid cloud plans.
- Review the official page for current platforms, integrations, system requirements, licensing, and product changes.
Visit the Langfuse website
Information checked: August 5, 2026
Back to top
LangSmith
Use case: Trace, test, evaluate, and monitor language-model applications.
Type: Commercial LLM development and evaluation platform
Access: Free and paid plans
What It Does
According to the vendor or project, the tool's stated purpose is to trace, test, evaluate, and monitor language-model applications.
Positive Community Commentary
Teams working with LangChain and LangGraph often value LangSmith’s native trace views because agent steps, tool calls, and graph execution are easy to follow. Practitioners also use its datasets and custom evaluators to compare prompt changes against production behavior and run repeatable regression checks before releases.
Things to Know
- Product or project type: Commercial LLM development and evaluation platform.
- Current access model: Free and paid plans.
- Review the official page for current platforms, integrations, system requirements, licensing, and product changes.
Visit the LangSmith website
Information checked: August 5, 2026
Back to top
lm-evaluation-harness
Use case: Evaluate language models across a large collection of academic benchmarks.
Type: Open-source language-model evaluation framework
Access: Free and open source
What It Does
According to the vendor or project, the tool's stated purpose is to evaluate language models across a large collection of academic benchmarks.
Positive Community Commentary
Researchers and model builders value lm-evaluation-harness as a common, reproducible way to run many established benchmarks across local models and APIs. Its broad task catalog, configurable prompts, multiple inference backends, and use behind public leaderboards make results easier to compare with published work and community baselines.
Things to Know
- Product or project type: Open-source language-model evaluation framework.
- Current access model: Free and open source.
- Review the official page for current platforms, integrations, system requirements, licensing, and product changes.
Visit the lm-evaluation-harness website
Information checked: August 5, 2026
Back to top
O
OpenAI Evals
Use case: Define and run evaluations for language-model behavior.
Type: Open-source evaluation framework and registry
Access: Free and open source
What It Does
According to the vendor or project, the tool's stated purpose is to define and run evaluations for language-model behavior.
Positive Community Commentary
Developers have valued OpenAI Evals for making model behavior testable through reusable registry entries and custom, use-case-specific evaluations. The framework is particularly useful for teams that want to compare model versions against private examples and contribute benchmark cases that describe important real-world failure patterns.
Things to Know
- Product or project type: Open-source evaluation framework and registry.
- Current access model: Free and open source.
- Review the official page for current platforms, integrations, system requirements, licensing, and product changes.
Visit the OpenAI Evals website
Information checked: August 5, 2026
Back to top
Opik
Use case: Trace, test, evaluate, and monitor LLM applications.
Type: Open-source LLM evaluation and observability platform
Access: Free open-source edition and paid cloud plans
What It Does
According to the vendor or project, the tool's stated purpose is to trace, test, evaluate, and monitor LLM applications.
Positive Community Commentary
Practitioners value Opik for combining an open-source backend and UI with tracing, datasets, experiments, model-based metrics, and pytest integration. Teams often point to self-hosting and step-by-step agent traces as useful when they want to debug individual components, run evaluations in CI, and keep application data inside their own environment.
Things to Know
- Product or project type: Open-source LLM evaluation and observability platform.
- Current access model: Free open-source edition and paid cloud plans.
- Review the official page for current platforms, integrations, system requirements, licensing, and product changes.
Visit the Opik website
Information checked: August 5, 2026
Back to top
P
Patronus AI
Use case: Evaluate LLM systems for quality, safety, compliance, and security.
Type: Commercial AI evaluation and security platform
Access: Contact vendor for pricing
What It Does
According to the vendor or project, the tool's stated purpose is to evaluate LLM systems for quality, safety, compliance, and security.
Positive Community Commentary
AI evaluation practitioners have shown particular interest in Patronus AI’s work on domain-specific benchmarks and evaluator models, including financial question answering and hallucination detection. It is most relevant to teams that want structured evaluation scenarios, research-backed judge models, and deeper testing of retrieval or agent behavior in high-stakes applications.
Things to Know
- Product or project type: Commercial AI evaluation and security platform.
- Current access model: Contact vendor for pricing.
- Review the official page for current platforms, integrations, system requirements, licensing, and product changes.
Visit the Patronus AI website
Information checked: August 5, 2026
Back to top
Promptfoo
Use case: Compare prompts and models, run assertions, and test security and red-team scenarios.
Type: Open-source prompt and LLM evaluation framework
Access: Free and open source with commercial platform
What It Does
According to the vendor or project, the tool's stated purpose is to compare prompts and models, run assertions, and test security and red-team scenarios.
Positive Community Commentary
Developers value Promptfoo’s local, configuration-driven workflow for comparing prompts, providers, and test cases in a clear matrix. Its command-line focus, flexible assertions, caching, and CI integration make it popular for repeatable prompt regression tests, while the same test suite can be extended into application-specific red teaming and security checks.
Things to Know
- Product or project type: Open-source prompt and LLM evaluation framework.
- Current access model: Free and open source with commercial platform.
- Review the official page for current platforms, integrations, system requirements, licensing, and product changes.
Visit the Promptfoo website
Information checked: August 5, 2026
Back to top
Pydantic Evals
Use case: Evaluate LLM applications, agents, and other systems against repeatable test datasets.
Type: Python AI evaluation framework
Access: Free and open source
What It Does
Pydantic Evals is an open-source Python evaluation framework from Pydantic for defining evaluation datasets, test cases, and evaluators and comparing results while refining LLM and agent behavior.
Positive Community Commentary
Code-defined datasets, cases, and evaluators support repeatable experiments on LLM, agent, and other AI behavior.
Things to Know
- This is a code-first library; teams must define meaningful cases and scoring criteria for their system.
- Review the official page for current platforms, integrations, licensing, and product changes.
Visit the Pydantic Evals website
Information checked: October 14, 2026
Back to top
PyRIT
Use case: Run automated and human-led red-team assessments of generative AI systems.
Type: Open-source Python generative-AI risk-identification and red-teaming framework
Access: Free and open source under the MIT License; model or service costs may apply
What It Does
According to Microsoft, PyRIT supports automated and human-led red teaming of generative-AI systems. It includes attack strategies, reusable scenarios, target connectors, scoring methods, command-line scanning, and CoPyRIT, a graphical interface for interactive testing.
Positive Community Commentary
PyRIT is a strong fit for security teams that need repeatable AI red-team workflows across hosted models, custom HTTP endpoints, WebSockets, and web applications. It is more security-assessment-oriented than a conventional LLM quality-evaluation tool.
Things to Know
- Product or project type: Python framework for generative-AI security and safety testing.
- Current access model: Free and open source under the MIT License.
- Supports single-turn and multi-turn attack strategies.
- Includes CLI, GUI, framework, scenario, memory, and scoring workflows.
- The source repository moved from Azure/PyRIT to Microsoft/PyRIT; use the current repository for source and issue information.
- Review the current documentation for supported targets, installation, and configuration requirements.
Visit the PyRIT documentation
Information checked: September 21, 2026
Back to top
T
TruLens
Use case: Instrument and evaluate LLM applications with feedback functions.
Type: Open-source LLM evaluation and tracing framework
Access: Free and open source
What It Does
According to the vendor or project, the tool's stated purpose is to instrument and evaluate LLM applications with feedback functions.
Positive Community Commentary
Practitioners value TruLens for the RAG Triad approach: measuring context relevance, groundedness, and answer relevance rather than treating a response as one opaque score. Its tracing and feedback functions are useful for teams that want to inspect retrieval quality, compare application versions, and display evaluation results alongside a working RAG application.
Things to Know
- Product or project type: Open-source LLM evaluation and tracing framework.
- Current access model: Free and open source.
- Review the official page for current platforms, integrations, system requirements, licensing, and product changes.
Visit the TruLens website
Information checked: August 5, 2026
Back to top
W
Weights & Biases Weave
Use case: Trace, evaluate, and manage generative-AI application experiments.
Type: Commercial AI evaluation and observability platform
Access: Free and paid plans
What It Does
According to the vendor or project, the tool's stated purpose is to trace, evaluate, and manage generative-AI application experiments.
Positive Community Commentary
Users value Weave’s low-friction instrumentation: a small amount of code can capture calls, versions, metrics, and feedback for an LLM or agent workflow. Practitioners also appreciate having traces, datasets, custom scorers, and evaluations connected in one experiment-oriented workspace, with support for both Python and TypeScript applications.
Things to Know
- Product or project type: Commercial AI evaluation and observability platform.
- Current access model: Free and paid plans.
- Review the official page for current platforms, integrations, system requirements, licensing, and product changes.
Visit the Weights & Biases Weave website
Information checked: August 5, 2026
Back to top
