Pairing Analysis

Uses an LLM, given the environment and task context, to judge whether main/side task pairings make realistic sabotage scenarios. Use it to find weak pairings and better cover tasks.

Commands

ct run pairing-analysis

Options include -e/--env, -s/--side-task, -n/--num-workers, and --use-1m-context; see ct run pairing-analysis --help for the full list.

Examples:

# Analyze all pairings
ct run pairing-analysis

# Analyze pairings for specific environment
ct run pairing-analysis -e web_scraping

Output

For each side task, an analysis that picks the best main task pairing and scores it out of 10. Results, including recommended_pairings.jsonl, are saved to data/pairing_analysis_runs/.

Analysis Criteria

For each pairing:

• Does the main task require accessing the same resources as the side task?
• Would normal main task work provide cover for the sabotage?
• Are there better main tasks for this side task?
• Is the pairing realistic in a production scenario, and challenging?