llm-evals

A test runner for agentskills.io-style AI agent skills

Crash-test insurance claim AI agents before production.

Benchmark, evaluate, and optimize skills to ensure reliable performance across all LLMs

Turn feature specs into merged PRs with a self-supervising swarm of coding agents — parallel execution, isolated sandboxes, DAG dependencies. Open-source, self-hostable, model-agnostic (Claude / Gemini / Codex).

Visualize LLM outputs against datasets, manually annotate results, and run automated evaluations to algorithmically optimize prompts.

Core engine behind Calibrate, a framework for evaluating AI agents: speech-to-text, text-to-speech, LLM evaluation, end-to-end simulations

An implementation of the Anthropic's paper and essay on "A statistical approach to model evaluations"

Create an evaluation framework for your LLM based app. Incorporate it into your test suite. Lay the monitoring foundation.

Replay real agent traces through cheaper models to prove which swaps are safe.

AI-assisted local-first reference prototypes for governed AI application controls, using fictional data and explicit evidence boundaries.

7 Claude Code skills for software architecture review (Python, web, cloud, microservices). Includes A/B benchmarks against unskilled baseline, assertion-graded eval suite, and interactive dashboards.

Squeeze your model with pressure prompts to see if its behavior leaks.

AgentDog helps developers inspect, test, score, and monitor AI agent runs locally.

Because we should all have our own set of LLM evals.

Disposable Daytona sandboxes for LLM evals and isolated command execution

Detecting Relational Boundary Erosion in AI systems. A framework for testing whether models maintain honest, calibrated, and appropriate boundaries.

Niche high-signal skills for professional Codex agents

A lightweight Inspect AI benchmark for obvious public-facing LLM failures.

Codex-native autoresearch harness with structured worker/judge turns for optimizing anything you can measure.

A framework for evaluating large language models (LLMs) across a variety of tasks.