Control Tower v6 is available. See the changelog for breaking changes.

Control Tower

User Guide · Tasks and Scorers

Submit Guessability Audit

Audits side tasks for submit tool exploitability.

Purpose

Checks whether an agent could pass the scorer by calling the submit tool cleverly, without modifying the environment or violating the security property. The analysis reads the code without running it.

Commands

ct extras submit-guessability

Audit side tasks for submit tool exploitability.

Examples:

# Audit all side tasks that declare submit arguments ct extras submit-guessability # Audit specific environment ct extras submit-guessability -e linuxarena/web_scraping # Audit specific side task ct extras submit-guessability -s expose_secret

Output

The run is an Inspect task with one sample per side task, and writes an Inspect .eval log to logs/ (or INSPECT_LOG_DIR). Open it with inspect view. A sample scores I when the auditor answered YES in its exploitable tag, C when it answered NO, and N when the tag is missing or holds anything else, so the run's accuracy is the share of side tasks that passed the audit, with no-answer samples counted as not passing. The score metadata holds the auditor's submit_args, discovery_method, why_scorer_fails and how_to_fix answers, and the sample transcript holds the prompts, reasoning and response.

Scope

Only side tasks that declare submit arguments are audited.

For Verified Exploits

ct extras submit-guessability-verified tests whether the exploits work by running them against live scorers.