Running Evaluations
Use ct run eval to run a policy or control protocol against one or more tasks. Start with a test-policy smoke run before spending model tokens:
ct run eval --policy test --task-file simple --tags test
For a full sabotage evaluation, use ct run sabotage-eval, which runs honest evals, attack evals, monitoring, and safety metrics in one command.
Guides
| Guide | What it covers |
|---|---|
| Setup and configuration | Environments, credentials, models, policies, and external sandboxes |
| Task selection | Exact tasks, task files, trajectories, replay, and expansion modes |
| Examples | Honest, attack, batch, and live-development runs |
| Config files | YAML configuration, composition, and the current schema |
| Your own Task | Compose Control Tower's solver, scorers and sandbox into an Inspect Task you own |
| EC2 fleet | Authentication, fleet options, streaming fleet events, and debugging |
| Output | Eval logs, Inspect's viewer, and trajectory conversion |
| Runs | Uploading, listing, inspecting, sampling, and downloading runs |
| Troubleshooting | Docker address pools and container service startup |
ct run eval CLI reference | Every current argument and option, generated from Click |
See Also
- Sabotage Evaluations - End-to-end sabotage evaluation pipeline
- Auditing Environments - Environment security and side task quality review
- Monitoring Trajectories - Evaluating trajectory suspiciousness
- Making Main Tasks - Creating main tasks
- Making Side Tasks - Creating side tasks
- Making Sabotage Trajectories - Creating sabotage trajectories (strajs)