Evals

Benchmark of real GitHub issues for evaluating coding agents.

UK AISI framework for LLM and agent evaluations with sandboxed tools.

Framework and registry for evaluating LLMs and LLM systems.

CLI for testing prompts, agents and RAG with assertions and red-teaming.

Pytest-like LLM evaluation framework with agentic metrics like task completion.

Open-source LLM observability: traces, evals and prompt management for agents.

Open-source tracing and evaluation for LLM apps built on OpenTelemetry.

Sierra benchmark for tool-agent-user interaction in realistic domains.

Benchmark for agents completing tasks in real terminal environments.

Eval and observability platform for iterating on AI products.

Evaluation toolkit for RAG and agent pipelines.

Practical guidance on designing graders and tasks for agent evaluation.

Tracing, evals and monitoring for LLM apps.

Weights & Biases toolkit for tracing and evaluating LLM apps.

Open-source LLM gateway and observability.