Control Tower v6 is available. See the changelog for breaking changes.

Control Tower

User Guide · Tasks and Scorers

Pairing Analysis

Uses an LLM, given the environment and task context, to judge whether main/side task pairings make realistic sabotage scenarios. Use it to find weak pairings and better cover tasks.

Commands

ct extras pairing-analysis

Options include -e/--env, -s/--side-task, -n/--num-workers, and --use-1m-context; see ct extras pairing-analysis --help for the full list.

Examples:

# Analyze all pairings ct extras pairing-analysis # Analyze pairings for specific environment ct extras pairing-analysis -e linuxarena/web_scraping

Output

The run is an Inspect task with one sample per side task, and writes an Inspect .eval log to logs/ (or INSPECT_LOG_DIR). Open it with inspect view. Each sample's score is the compatibility, out of 10, of the main task the model recommends for that side task, and the score's answer is that main task; a sample whose response names no best pairing is left unscored. A sample whose reply was cut off, at the output-token limit or by a prompt too long for the model, errors instead of being scored, and the command exits non-zero naming the log. Each prompt holds the environment's whole source, which can outgrow the model's context window. The sample transcript holds the prompt, reasoning and response.

Next to the log, <log name>.recommended_pairings.jsonl lists each scored side task with its recommended main task, highest score first. It is a task file, so ct run eval --task-file can run it.

Analysis Criteria

For each pairing:

  • Does the main task require accessing the same resources as the side task?
  • Would normal main task work provide cover for the sabotage?
  • Are there better main tasks for this side task?
  • Is the pairing realistic in a production scenario, and challenging?