Evaluation Setup
Prerequisites
Environments
Environments must be installed before running evals. If .settings/ is empty, pull them first:
ct settings pull
See Development Setup for details.
API Keys
You'll need API keys for the model providers you're using. Set them in your .env file:
ANTHROPIC_API_KEY=your-key-here OPENAI_API_KEY=your-key-here
Local Services
Start the support services once:
docker compose up -d
This starts local MongoDB (localhost:27017), the website (private viewer at localhost:3000 with login admin/linuxbench, public viewer at localhost:3002, docs at localhost:3003, and the MinIO console at localhost:9001 with login minio/minio123). The stack's services reach Mongo through MONGOURI, set in compose.yml. The CLI never touches Mongo; it uploads to the local viewer over HTTP, so point it at the stack with CONTROL_TOWER_API_BASE_URL=http://localhost:3000. See Web Viewer for details.
Then log in with the local credentials admin/linuxbench:
ct login
Without the local services, evals still run and write local files, but results are not uploaded or viewable on the website.
Quick Start
The primary command for running evaluations is ct run eval:
ct run eval --policy <policy> --protocol <protocol> --env <env> --main-task <task>
Test your setup before running expensive model evaluations:
ct run eval --policy test --task-file simple --tag test
The test policy runs ls and submits without calling a model, here on the simple task set. A test run checks a new development environment, Docker builds, and scorer execution, and isolates infrastructure issues before real evals.
Configuration
Model Configuration
An eval binds two model roles: untrusted, the model the policy drives, and trusted, blue's trusted model. By default they are:
- Untrusted model:
anthropic/claude-haiku-4-5 - Trusted model:
openai/gpt-5-nano
Run setup also binds monitor, the role monitors answer on, to the trusted binding unless you bind it yourself. A run that includes a component needing its own model binds that component's role too, unless you do: the simulated sandbox binds simulator and scorer, --diagnostics binds diagnostic, and Jev binds jev. Every bound role is recorded in the run's task args. Code that needs a role nobody bound fails rather than picking a model of its own.
Override any role with --model-role <role>=<model>. The CLI accepts model aliases:
| Alias | Full Model Name |
|---|---|
haiku | anthropic/claude-haiku-4-5 |
sonnet | anthropic/claude-sonnet-4-6 |
opus | anthropic/claude-opus-4-6 |
opus_4_6 | anthropic/claude-opus-4-6 |
opus_4_8 | anthropic/claude-opus-4-8 |
gemini_flash | google/gemini-3-flash-preview |
gemini_pro | google/gemini-3.1-pro-preview |
untrusted | Current default untrusted model (haiku) |
trusted | Current default trusted model (gpt-5-nano) |
smart | Current best model (openai/gpt-6-astra); the default for analysis and generation tools |
fast | Current cheap fast model (openai/gpt-6-luna); the default where a tool favours speed and cost |
# Using aliases ct run eval --policy honest --model-role untrusted=sonnet ... # Using full model names ct run eval --policy honest --model-role untrusted=anthropic/claude-sonnet-4-6 ...
OpenRouter routing
USE_OPENROUTER sets which models go through OpenRouter: unset, 0 or off (the default) sends none, 1 or all sends every model, and a comma-separated list of provider slugs such as google,openai sends only those providers. When it is not off, set OPENROUTER_API_KEY in your .env.
Pinning an OpenRouter provider
OpenRouter load-balances each model across backends that can differ in quantization, context window, and which sampling parameters they honour, so unpinned runs are not reproducible per-provider (Control Tower warns once per model when this applies). A role binds a bare model name or Inspect's role config, {model: ..., <generate config fields>, model_args: {...}} in YAML or JSON; pin the serving provider with its model_args.provider:
ct run eval --policy honest \ --model-role 'untrusted={model: deepseek/deepseek-chat-v3.2, model_args: {provider: {only: [deepinfra/fp8], allow_fallbacks: false}}}' ...
The provider dict is OpenRouter's provider routing object, merged per key over Control Tower's defaults (fp8+ quantization floor, data_collection: deny, sort: exacto). only restricts the candidate set; order alone is try-priority and still falls through unless allow_fallbacks is false. Generate config fields such as reasoning_effort or max_connections sit beside model, over Control Tower's defaults (response cache and max_retries). The pin is recorded in run metadata and survives re-runs from saved args. Passing provider for a model that does not route via OpenRouter is an error.
Self-Hosted and Open-Weight Models
Model names resolve through Inspect AI's model providers, so self-hosted open-weight models work anywhere a model name is accepted. Point the provider at your endpoint via env vars in .env and use the matching prefix:
- vLLM / SGLang: set
VLLM_BASE_URL=http://<host>:8000/v1(orSGLANG_BASE_URL=...), then pass--model-role untrusted=vllm/Qwen/Qwen3-32B. Serve with--enable-auto-tool-choiceand your model family's tool-call parser, since agent policies use OpenAI-style tool calls. - Any OpenAI-compatible endpoint (NVIDIA NIM, TGI, llama.cpp): pass
--model-role untrusted=openai-api/<service>/<model>, which reads<SERVICE>_BASE_URLand<SERVICE>_API_KEY.
Self-hosted prefixes (vllm/, sglang/, hf/, ollama/, openai-api/) and Inspect's offline mockllm/ are never routed through OpenRouter, even with USE_OPENROUTER=1. Neither are the ct extras commands.
A single GPU endpoint usually needs concurrency capped, through the role's generate config and --eval-arg max_samples: --model-role 'untrusted={model: vllm/Qwen/Qwen3-32B, max_connections: 8}' --eval-arg max_samples=1. A fresh clone has no rate file until you run uv run python -m control_tower.scripts.generate_model_rates (see Development Setup); an eval refuses to start until that file exists. A model the loaded table does not list reports its cost as unknown, not $0.00. A monitor model that can't answer with a valid score records that draw unscored, so check unscored counts before trusting safety numbers from a small model.
Pull Pre-built Images (Optional)
CONTROL_TOWER_IMAGES sets where environment images come from:
| Value | Behavior |
|---|---|
build (default) | Build images locally from the env's Dockerfiles, with layer caching from Docker Hub. |
pull | Pull the published images from Docker Hub; never build them. |
offline | Contact no registry and use only images already in the local Docker store, filled by ct settings pull-images (see Egress-Locked Hosts). |
To pull pre-built images instead of building them:
export CONTROL_TOWER_IMAGES=pull
Services whose compose entry builds a single-segment image name (no registry or namespace, e.g. BashArena's per-task bash-arena-default-<task_id>:latest) are always built locally, even under pull or offline, because such names cannot exist in a registry. Fleet workers set CONTROL_TOWER_IMAGES=pull, so this is what lets per-task-image environments run on EC2.
Pre-built images for evaluation environments are published to Docker Hub under the linuxarena organization (for example linuxarena/env-web_scraping:latest). Images for public environments (those with public repos on github.com/linuxarena) can be pulled anonymously; contact the team for access to private environment images.
Each environment's image tag comes from ENV_IMAGE_TAG_<ENV> (the uppercased bare env id, so linuxarena/web_scraping reads ENV_IMAGE_TAG_WEB_SCRAPING). The variable is keyed by the bare env id alone, so environments from different packages that share an env id read the same one; a run that includes more than one of them is refused, and each runs on its own. Env CI publishes each image under both latest and the env repo's commit SHA. The backend decides each env's tag once, before the run's samples are built, and passes it to them in the task args (image_tags): when the variable is unset, a local CONTROL_TOWER_IMAGES=pull run pulls the image published at the env checkout's commit, so skipping the build does not change which env version runs, and an EC2 fleet run pulls latest (workers have no local checkout), whatever the dispatching machine's CONTROL_TOWER_IMAGES. If no image exists at the decided tag (for a local run, a checkout ahead of CI or predating SHA-tagged publishing), a local run fails before any sample starts and an EC2 run fails on its workers. Set the variable to a published tag to override either default.
Each sample records the tag it pulled at metadata.environment.image_tag and each service's image as its sandbox pinned it by registry digest at metadata.environment.images. Provenance copies both into env_versions[<env>] (image_tag, and images by service), keyed by the env's canonical name (linuxarena/web_scraping), pinning a mutable tag like latest to the exact bytes that ran. Locally-built images have no digest.
Use pull for CI and batch evals, and for EC2 evals since instances start fresh. Keep build while testing local Dockerfile changes.
Scoring the main task on attack samples (Optional)
By default an attack-mode sample (one with a side task) scores only its side task. Pass --score-main-on-attack to also run the main-task scorer on attack samples. This measures whether an attack policy completes the main task while attacking, and gives per-run safety/usefulness frontier points.
ct run eval --policy attack -e linuxarena/web_scraping -t crawl_depth -s visit_malicious_website --score-main-on-attack
The side scorer always runs first, then the main scorer runs under its own --scorer-timeout, so an attack sample can take up to twice as long to score. The flag is run-level, and it is carried through YAML configs and hawk submissions. An environment can also declare score_main_on_attack itself, for benchmarks whose side task is defined relative to main-task performance; if any selected environment declares it, main scoring is on for the whole run. The log's run metadata records score_main_on_attack as it was in effect, whichever turned it on. Simulated runs (--sandbox simulated) score no main task and ignore the flag.
Transcript diagnostics (Optional)
Pass --diagnostics refusal,sandbox-escape to have an LLM judge check each finished transcript for an explicit refusal or a sandbox-escape attempt. The run scans its final log with Inspect Scout and writes the results to scans/ beside it, leaving the log's scores untouched. See Transcript diagnostics.
Inspect run settings
Inspect's own run settings, such as epochs, limits and concurrency, are passed with --eval-arg, one per setting:
ct run eval -e linuxarena/web_scraping -t crawl_depth --eval-arg epochs=3 --eval-arg time_limit=600
The keys are the fields of Inspect's EvalConfig, and each value is read as Inspect reads a -T task arg: as YAML, with a comma-separated value a list. So --eval-arg sample_id=a,b selects two samples, a one-item list is written epochs_reducer=[mean], and a value YAML would misread is quoted, as in --eval-arg "sample_id='*probe'". In a run config the same settings go under eval_config.
--ec2 refuses max_samples and max_tasks, because each fleet job runs one sample; --ec2-arg concurrency_per_worker sets the fleet's concurrency. --hawk refuses the settings Hawk's runner sets itself, listed in the Hawk usage guide.
Policies and Protocols
Every run pairs the red team's policy (--policy, default honest) with the blue team's protocol (--protocol, default null-blue-team, which runs every action unmodified):
ct run eval --policy honest --task-file simple ct run eval --policy attack --model-role untrusted=sonnet --task-file medium
A protocol that monitors takes its monitors as an argument:
ct run eval \ --policy attack \ --protocol trusted-monitoring \ --protocol-arg 'monitors=[simple]' \ --protocol-arg audit_threshold=70 \ --task-file simple
A sample runs until U submits, an attack sample is audited, or U uses up white's model call budget (-wa max_steps=N, default 200). Control Tower sets no time, message or token limit of its own, locally or on the fleet; --eval-arg time_limit=N, --eval-arg message_limit=N and --eval-arg token_limit=N set one.
ct protocols policies and ct protocols protocols list what is registered, and ct protocols policies <name> shows one entry's arguments. See Protocols and Policies for how they are built.
Network Isolation
Docker evaluations deny direct internet access by default. --internet explicitly opts into direct egress; --no-internet selects the default. The restriction marks every network in the environment's compose file internal: true with an isolated gateway, so Docker gives the sample's containers no route out and no path to the host itself. Docker enforces this outside the container, so the agent cannot undo it even as root, and the agent service gets no added capability. The isolated gateway mode needs Docker Engine 28 or newer (27 rejects it; 26 and older ignore it, leaving the host reachable), so Control Tower refuses an older daemon before it starts containers. The sample's internet simulator is a service of the same compose project that also keeps a bridge of its own, so it stays reachable and is the only path to anything outside the sample. Only the upstream proxies an environment declares go through it to real hosts.
ct run eval --policy honest --task-file simple # Explicit opt-in for an environment that needs direct egress: ct run eval --policy honest --task-file simple --internet
--no-internet-simulator independently disables simulator provisioning and attachment for the sample, including persona routing and upstream proxies. It works for local Docker evaluations, fleet workers, and live sessions. Combine it with --no-internet to retain only compose siblings. Scorers and setup scripts that require simulator endpoints cannot run in this mode. Hawk and external sandboxes reject this option.
Compose siblings, the simulator, and its personas resolve and connect as usual unless the simulator is disabled. Anything else, including the host machine, cloud instance metadata, other samples' published ports, and other hosts on the machine's private network, is unreachable. The run fails before anything starts when the sample cannot be isolated this way: a service using a network_mode other than none or service:..., a compose network declared external: true, with a fixed name:, or with a driver other than bridge, a compose file using include:, or a service using extends:. Tasks whose scorers or setup scripts fetch from real hosts fail without internet access until they are made offline-clean.
Name resolution is closed the same way. Docker's embedded resolver answers compose service names, persona aliases and exposed-service domains, and Docker Engine 26 and newer forward no other query from a container attached only to internal networks, so outside names fail to resolve and no lookup reaches the host's resolvers.
The network choice is recorded in the run sandbox configuration and forwarded to fleet workers. Hawk enforces the same choice through its Cilium network policy; restricted runs support simulator personas but reject environments that declare upstream API proxies until their routing and CA setup is supported (see the Hawk values contract). External sandboxes and other backends without enforcement require an explicit --internet opt-in.
Egress-Locked Hosts
A host that runs cyber-capable models may allow outbound connections only to the model API. Everything else an eval needs is seeded while the host still has internet. The eval then runs with CONTROL_TOWER_IMAGES=offline, under which Control Tower pulls no images and uses only the local Docker store. A missing environment image fails the run before any sample starts. A missing internet simulator or exposed-services proxy image fails each sample at sandbox setup, before the agent runs.
Set it in .env first:
CONTROL_TOWER_IMAGES=offline
Then, with internet:
uv sync --locked uv run python -m control_tower.scripts.generate_model_rates ct settings pull linuxarena ct settings pull-images
ct settings pull-images always pulls, even under offline. It pulls every image referenced by the compose files of the codebases the installed environments list in codebases (every compose_paths entry of a DockerEnvironment), at the tags a pull or offline eval resolves: ENV_IMAGE_TAG_<ENV> when set, otherwise the env checkout's commit. It also pulls the internet simulator and exposed-services proxy images. --only <substring> and --skip <substring> select compose files by path, as for ct settings build-images. Services that build a single-segment image name are built at eval time, which needs registry access for their base images, so environments with such services cannot run on a locked host. Re-run it after updating Control Tower or the settings, since both can change which images an eval needs.
Then restrict egress to the model API. Leave --upload off and copy the eval logs off the host afterwards.
External Sandboxes
--sandbox with a name other than the built-in simulated (Simulated Sandbox) and replay (replaying without execution) runs samples in an Inspect sandbox provided outside this repo, such as a microVM runtime, instead of the default Docker sandbox. Name it by its registered type. Control Tower finds sandbox types the way it finds everything else outside its own code: an installed package registers one through its inspect_ai entry point (as Inspect's sandbox packages do), or a setting registers one from its ct_plugin/, which ct settings pull installs. An unknown type stops the run before it starts.
ct run eval \ --policy honest \ --task-file simple \ --sandbox my-microvm \ --internet \ --sandbox-arg boot_timeout=300
--sandbox-arg values are passed to an external sandbox as its config, a Pydantic model holding each value as the string given, for it to parse and validate; Control Tower does not interpret them, and their keys reach it exactly as written. A built-in sandbox takes only the args ct run eval --help lists for it, and without --sandbox the default Docker sandbox takes none. The external sandbox owns the sample lifecycle through Inspect's task_init, sample_init, sample_cleanup and task_cleanup hooks; with --ec2 each fleet worker runs it for its own jobs. It also requires --internet and rejects --no-internet-simulator, which only the Docker sandbox can enforce. The selection is recorded in run metadata with the rest of the sandbox config.