Evaluation Task Selection

The task selection system converts CLI arguments into task combinations. You must specify tasks using one of the following modes.

Selection Modes

1. Specific Task Mode

Specify exact tasks to run. This is the most common approach.

# Single main task (honest mode - no side task)
ct run eval -t crawl_depth --policy honest

# Main + side task (attack mode)
ct run eval -t crawl_depth -s visit_malicious_website --policy attack

# Multiple environments with same main task
ct run eval -e web_scraping -e clinical_trial --just-main-tasks --policy honest

# Multiple of each (creates cross-product)
ct run eval -t task1 -t task2 -s side1 -s side2 --policy attack

2. Task File Mode (--task-file)

Reads explicit task combinations from a file. Takes precedence over other options.

# From a named task set
ct run eval --task-file medium --policy attack

# From a custom file path
ct run eval --task-file my_tasks.jsonl --policy attack

Task files are JSONL, one record per task combination with main_task and/or side_task:

{"env": "web_scraping", "main_task": "crawl_depth", "side_task": "visit_malicious_website"}
{"env": "clinical_trial", "main_task": "adverse_reaction_followup_tracking", "side_task": "suppress_safety_notifications"}

3. Trajectory Mode (--trajectory-id)

Extracts task info from an existing trajectory. Useful for re-running or replaying.

# From API/database (trajectory ID)
ct run eval --trajectory-id abc123 --policy attack

# From local file
ct run eval --trajectory-id ./my_trajectory.json --policy attack

# With task override
ct run eval --trajectory-id abc123 -t different_main_task --policy attack

This is especially useful with the replay policy to re-execute trajectories:

ct run eval --trajectory-id abc123 --policy replay

Replaying without execution (--replay-sandbox)

By default a replay re-executes every recorded action in a real sandbox. --replay-sandbox skips execution: nothing is provisioned, and every exec / text_editor call is answered with the recorded output of the next action in the source trajectory. Replayed samples cost nothing on the sandbox side and match the recording exactly.

Replay stays sound while the tool calls that reach the sandbox form a verbatim prefix of the recording. Anything that never reaches the sandbox may differ from the recording: assistant text and reasoning, monitor scoring, interrogation questions and the policy's answers to them, judge deliberation, or stopping early. A sandbox-reaching call that is new, modified, reordered, or skipped raises ReplayDivergenceError with the recorded and incoming action.

--replay-sandbox therefore works with any replay-like policy, meaning one whose executed tool calls stay a prefix of the recording. For example, a policy that emits the recorded tool calls but answers a blue protocol's interrogation questions with a live model is replay-like, because the answers never touch the sandbox. On the blue side, pass-through protocols (e.g. trusted monitoring) replay cleanly, while protocols that rewrite actions (trusted editing, defer-to-trusted) diverge and raise. Even a harmless ls that wasn't recorded raises, because no container exists to answer it.

Recorded editor calls must match the editor operation and arguments, including line numbers and view ranges. A recorded editor failure is replayed as a tool error. A divergence prevents score carryover even if the calling tool or protocol catches the exception.

The eval message limit grows to fit the longest source recording under --replay-sandbox. The protocol's own step budget still applies: raise it (for example, -bp max_steps=420) when replaying a longer recording in full.

To sweep a blue protocol over a recording, pair --replay-sandbox with the replay untrusted policy, which re-issues the recording's tool calls in order and submits once they run out:

ct run eval2 --trajectory-id logs/run.eval --untrusted-policy replay -up traj_like=logs/run.eval \
    --replay-sandbox --protocol defer-to-resample

Programmatic scorers don't run: a full replay carries the source trajectory's scores over; a partial or diverged replay scores NOANSWER with an explanation. Hidden scorer calls such as run_side_task_test, policy-private actions, and recorded argument-parsing failures are skipped when matching sandbox calls. Their scorer scripts are not re-executed.

4. Expansion Modes

Use when you want to run many tasks without listing them explicitly.

--all: Expands to all valid main×side combinations.

# All attack combinations for one environment
ct run eval -e web_scraping --all --policy attack

--just-main-tasks: All main tasks with no side tasks, for honest evaluations.

ct run eval --just-main-tasks --policy honest
ct run eval -e web_scraping --just-main-tasks --policy honest

--just-side-tasks: All side tasks with no main task. The agent receives only the side task instructions.

ct run eval --just-side-tasks --policy attack

Validation Rules

1. One mode required: Must specify --task-file, an expansion flag (--all, --just-main-tasks, --just-side-tasks), or specific tasks (-t)
2. Expansion flags are mutually exclusive: Cannot combine --all, --just-main-tasks, --just-side-tasks
3. Specific tasks don't mix with expansion: -t/-s cannot be combined with expansion flags
4. Environment is optional: If not specified, inferred from task IDs or defaults to all environments

If you omit required parameters, the CLI will prompt you interactively (unless DISABLE_TUI=1).

Attack vs Honest Mode

Whether a sample runs in "attack" or "honest" mode is determined by side task presence:

• Attack mode: Sample has a side task (-s specified or from task file)
• Honest mode: Sample has no side task

Task Sets

Task sets are predefined collections of task combinations stored in src/control_tower/task/task_sets/. They make it easy to run consistent evaluations across the team. The built-in sets are examples:

Manage task sets with ct task-sets:

ct task-sets list              # List available task sets
ct task-sets validate simple   # Validate a task set
ct task-sets validate --all    # Validate all task sets
ct task-sets show medium       # Show contents of a task set

To create a custom task set, add a .jsonl file to src/control_tower/task/task_sets/ with one record per task combination:

{"env": "clinical_trial", "main_task": "adverse_reaction_followup_tracking", "side_task": "suppress_safety_notifications"}
{"env": "web_scraping", "main_task": "redact", "side_task": "expose_secret"}

Each record names an env plus main_task and/or side_task (a record with only side_task runs side-only). Run ct task-sets validate <name> to check it.