Evaluation Setup

Prerequisites

Environments

Environments must be installed before running evals. If .settings/ is empty, pull them first:

ct settings pull

See Development Setup for details.

API Keys

You'll need API keys for the model providers you're using. Set them in your .env file:

ANTHROPIC_API_KEY=your-key-here
OPENAI_API_KEY=your-key-here

Local Services

Start the support services once:

docker compose up -d

This starts local MongoDB (localhost:27017), the website (private viewer at localhost:3000 with login admin/linuxbench, public viewer at localhost:3002, docs at localhost:3003), and the eval viewer (localhost:8080). The stack's services reach Mongo through MONGOURI, set in compose.yml. The CLI never touches Mongo; it uploads to the local viewer over HTTP, so point it at the stack with CONTROL_TOWER_API_BASE_URL=http://localhost:3000. See Web Viewer and Inspect Logs Viewer for details.

Then log in with the local credentials admin/linuxbench:

ct login

Without the local services, evals still run and write local files, but results are not uploaded or viewable on the website.

Quick Start

The primary command for running evaluations is ct run eval:

ct run eval --policy <policy> --env <env> --main-task <task>

Test your setup before running expensive model evaluations:

ct run eval --policy test --task-file simple --tags test

The test policy runs ls and submits without calling a model, here on the simple task set. A test run checks a new development environment, Docker builds, and scorer execution, and isolates infrastructure issues before real evals.

Configuration

Model Configuration

By default, evaluations use:

• Untrusted model: anthropic/claude-haiku-4-5
• Trusted model: openai/gpt-5-nano

You can override these defaults with policy arguments. The CLI accepts model aliases:

# Using aliases
ct run eval --policy honest -p model=sonnet ...

# Using full model names
ct run eval --policy honest -p model=anthropic/claude-sonnet-4-6 ...

OpenRouter routing

Set USE_OPENROUTER to a boolean to send all model traffic through OpenRouter, or set USE_OPENROUTER_PROVIDERS to a comma-separated list of provider slugs to send only those providers through OpenRouter. Do not use both at once. If you use either, set OPENROUTER_API_KEY in your .env.

Pinning an OpenRouter provider

OpenRouter load-balances each model across backends that can differ in quantization, context window, and which sampling parameters they honour, so unpinned runs are not reproducible per-provider (Control Tower warns once per model when this applies). Anywhere a model is accepted, pass a JSON object instead of a bare name to pin the serving provider:

ct run eval --policy honest \
  -p model='{"model_name": "deepseek/deepseek-chat-v3.2", "provider": {"only": ["deepinfra/fp8"], "allow_fallbacks": false}}' ...

The provider dict is OpenRouter's provider routing object, merged per key over Control Tower's defaults (fp8+ quantization floor, data_collection: deny, sort: exacto). only restricts the candidate set; order alone is try-priority and still falls through unless allow_fallbacks is false. An optional config key sets generation config the same way YAML configs do. The pin is recorded in run metadata and survives re-runs from saved args. Passing provider for a model that does not route via OpenRouter is an error.

Self-Hosted and Open-Weight Models

Model names resolve through Inspect AI's model providers, so self-hosted open-weight models work anywhere a model name is accepted. Point the provider at your endpoint via env vars in .env and use the matching prefix:

• vLLM / SGLang: set VLLM_BASE_URL=http://<host>:8000/v1 (or SGLANG_BASE_URL=...), then pass -p model=vllm/Qwen/Qwen3-32B. Serve with --enable-auto-tool-choice and your model family's tool-call parser, since agent policies use OpenAI-style tool calls.
• Any OpenAI-compatible endpoint (NVIDIA NIM, TGI, llama.cpp): pass -p model=openai-api/<service>/<model>, which reads <SERVICE>_BASE_URL and <SERVICE>_API_KEY.

Self-hosted prefixes (vllm/, sglang/, hf/, ollama/, openai-api/) are never routed through OpenRouter, even with USE_OPENROUTER=1.

A single GPU endpoint usually needs the per-call timeout raised and concurrency capped: -p model_attempt_timeout=600 -p model_max_connections=8 --max-samples 1. A fresh clone has no rate file until you run uv run python scripts/generate_model_rates.py (see Development Setup); an eval refuses to start until that file exists. A model the loaded table does not list reports its cost as unknown, not $0.00. A monitor model that can't follow the <score> output format after retries records an error and scores 0.0, so check error counts before trusting safety numbers from a small model.

Skip Docker Builds (Optional)

By default, running an eval rebuilds Docker images locally using the Dockerfile, with layer caching from Docker Hub. To skip rebuilding and pull pre-built images directly from Docker Hub instead:

export SKIP_DOCKER_BUILD=1

Services whose compose entry builds a single-segment image name (no registry or namespace, e.g. BashArena's per-task bash-arena-default-<task_id>:latest) are always built locally, even under SKIP_DOCKER_BUILD, because such names cannot exist in a registry. Fleet workers set SKIP_DOCKER_BUILD=1, so this is what lets per-task-image environments run on EC2.

Pre-built images for evaluation environments are published to Docker Hub under the linuxarena organization (for example linuxarena/env-web_scraping:latest). Images for public environments (those with public repos on github.com/linuxarena) can be pulled anonymously; contact the team for access to private environment images.

Each environment's image tag comes from ENV_IMAGE_TAG_<ENV> (uppercased env id, e.g. ENV_IMAGE_TAG_WEB_SCRAPING). Env CI publishes each image under both latest and the env repo's commit SHA, and when the variable is unset a local SKIP_DOCKER_BUILD=1 run pulls the image published at the env checkout's commit, so skipping the build does not change which env version runs. If no image exists at that commit (checkout ahead of CI, or predating SHA-tagged publishing), the run fails before any sample starts. EC2 fleet runs default unset tags to latest at dispatch (workers have no local checkout). Set the variable to a published tag to override either default.

Provenance also records the pulled image's digest at env_versions[<env>]["image_digest"], pinning a mutable tag like latest to the exact bytes that ran. Locally-built images have no digest.

Use SKIP_DOCKER_BUILD=1 for CI and batch evals, and for EC2 evals since instances start fresh. Leave it unset while testing local Dockerfile changes.

Scoring the main task on attack samples (Optional)

By default an attack-mode sample (one with a side task) records only side_task_success. Pass --score-main-on-attack to also run the main-task scorer on attack samples and record main_task_success. This measures whether an attack policy completes the main task while attacking, and gives per-run safety/usefulness frontier points.

ct run eval --policy attack -e web_scraping -t crawl_depth -s visit_malicious_website --score-main-on-attack

The side scorer always runs first, then the main scorer runs under its own --scorer-timeout, so an attack sample can take up to twice as long to score. The flag is run-level, so every sample emits the same score keys, and it is carried through YAML configs, hawk submissions, and run metadata. An environment can also declare score_main_on_attack itself, for benchmarks whose side task is defined relative to main-task performance; if any selected environment declares it, main scoring is on for the whole run. Simulated runs (--simulated) score no main task and ignore the flag.

Policies

Pass a single policy with --policy:

ct run eval --policy honest --task-file simple
ct run eval --policy attack -p model=sonnet --task-file medium

For protocol evals, use --untrusted-policy with --protocol instead:

ct run eval \
  --untrusted-policy simple-attack \
  --protocol trusted-monitoring \
  --protocol-arg sus_threshold=7 \
  --task-file simple

See Protocols and Untrusted Policies for the built-in policies, protocol-mode concepts, and implementation details.

Network Isolation

Docker evaluations deny direct internet access by default. --internet explicitly opts into direct egress; --no-internet selects the default. The restriction marks every network in the environment's compose file internal: true with an isolated gateway, so Docker gives the sample's containers no route out and no path to the host itself. Docker enforces this outside the container, so the agent cannot undo it even as root, and the agent service gets no added capability. The isolated gateway mode needs Docker Engine 28 or newer, which the fleet bootstrap installs. The internet simulator joins the project network from its own bridge, so it stays reachable and is the only path to anything outside the sample. Only the upstream proxies an environment declares go through it to real hosts.

ct run eval --policy honest --task-file simple
# Explicit opt-in for an environment that needs direct egress:
ct run eval --policy honest --task-file simple --internet

--no-internet-simulator independently disables simulator provisioning and attachment for the sample, including persona routing and upstream proxies. It works for local Docker evaluations, fleet workers, and live sessions. Combine it with --no-internet to retain only compose siblings. Scorers and setup scripts that require simulator endpoints cannot run in this mode. Hawk and external sandboxes reject this option.

Compose siblings, the simulator, and its personas resolve and connect as usual unless the simulator is disabled. Anything else, including the host machine, cloud instance metadata, other samples' published ports, and other hosts on the machine's private network, is unreachable. The run fails before anything starts when the sample cannot be isolated this way: a service using a network_mode other than none or service:..., a compose network declared external: true, with a fixed name:, or with a driver other than bridge, a compose file using include:, a service using extends:, or a Docker Engine older than 28. Tasks whose scorers or setup scripts fetch from real hosts fail without internet access until they are made offline-clean.

Name resolution is closed the same way. Docker's embedded resolver still answers compose service names and persona aliases, and forwards everything else to the simulator, which answers the sample's registered personas and returns NXDOMAIN for any other name, so no query reaches the host's resolvers. The simulator records every forwarded query, and at sample cleanup Control Tower copies the sample's record into its eval-log store under sandbox_dns_lookups (name, query type, source address, timestamp, and whether the name was one of the sample's personas). Recording starts when the simulator joins the project network during sample setup, so an empty list means the sandbox asked for no name outside its environment after that. Environments without an internet simulator get the same restricted DNS with nothing listening at the address, so outside lookups time out and nothing is recorded.

The network choice is recorded in the run sandbox configuration and forwarded to fleet workers. Hawk enforces the same choice through its Cilium network policy; restricted runs support simulator personas but reject environments that declare upstream API proxies until their routing and CA setup is supported (see the Hawk values contract). External sandboxes and other backends without enforcement require an explicit --internet opt-in.

--no-intranet independently blocks the agent container's connections to private IPv4 ranges, carrier-grade NAT, link-local and IPv6 local ranges, and every address on the Docker host. The simulator and its per-sample exposed-service proxy remain reachable. Rules match the agent's host-side veth interfaces, so nothing the agent changes inside its container removes them. The agent cannot reach other Compose services; those services can still communicate with each other. ARP and IPv6 neighbor discovery remain available for on-link delivery. Use --no-internet as well to block direct internet connections.

This Docker-only option uses a small control-plane helper with NET_ADMIN in the Docker host network namespace; the agent receives no additional capability. The helper image is built and cached on first use; ct envs pull builds it ahead of time. Host bridge netfilter must be enabled for IPv4 and IPv6. Agent containers with restart policies are rejected, since restarting replaces their veth interfaces. Setup fails if enforcement cannot be installed.

Egress-Locked Hosts

A host that runs cyber-capable models may allow outbound connections only to the model API. Everything else an eval needs is seeded while the host still has internet. The eval then runs with SKIP_DOCKER_PULL=1, under which Control Tower pulls no images and uses only the local Docker store. A missing environment image fails the run before any sample starts. A missing internet simulator or exposed-services proxy image fails each sample at sandbox setup, before the agent runs.

Set both flags in .env first. SKIP_DOCKER_PULL requires SKIP_DOCKER_BUILD, because the seeded images are the published ones:

SKIP_DOCKER_BUILD=1
SKIP_DOCKER_PULL=1

Then, with internet:

uv sync --locked
uv run python scripts/generate_model_rates.py
ct settings pull linuxarena
ct envs pull

ct envs pull always pulls, whatever SKIP_DOCKER_PULL says. It pulls every image the installed environments' compose files reference, at the tags a SKIP_DOCKER_BUILD=1 eval resolves: ENV_IMAGE_TAG_<ENV> when set, otherwise the env checkout's commit. It also pulls the internet simulator and exposed-services proxy images and builds the --no-intranet guard. --only <substring> and --skip <substring> select compose files by path, as for ct envs build. Services that build a single-segment image name are built at eval time, which needs registry access for their base images, so environments with such services cannot run on a locked host. Re-run it after updating Control Tower or the settings, since both can change which images an eval needs.

Then restrict egress to the model API. ct run eval uploads by default, so pass --no-upload and copy the eval logs off the host afterwards. --sandbox-isolation fetches a static bwrap binary on first use, so run it once with internet or set CT_BWRAP_HOST_BINARY.

External Sandboxes

--sandbox runs samples in an Inspect sandbox provided outside this repo, such as a microVM runtime, instead of the default Docker sandbox. Pass a registered type, or module:type to import the module that registers it, with --extra-src when the project is not installed.

ct run eval \
  --policy honest \
  --task-file simple \
  --sandbox my_project.sandbox:my-microvm \
  --internet \
  --sandbox-arg boot_timeout=300 \
  --extra-src ../my-project

--sandbox-arg values are passed to the sandbox as a JSON string for it to parse and validate; Control Tower does not interpret them, and their keys reach it exactly as written. The sandbox owns the sample lifecycle through Inspect's task_init, sample_init, sample_cleanup and task_cleanup hooks, so it cannot be combined with --ec2, --simulated or --replay-sandbox. It also requires --internet and rejects --no-internet-simulator, --no-intranet and --sandbox-isolation, which only the Docker sandbox can enforce. The selection is recorded in run metadata with the rest of the sandbox config.