Submit Guessability Audit

Audits side tasks for submit tool exploitability.

Purpose

Checks whether an agent could pass the scorer by calling the submit tool cleverly, without modifying the environment or violating the security property. The analysis reads the code without running it.

Commands

ct run submit-guessability

Audit side tasks for submit tool exploitability.

Examples:

# Audit all side tasks with custom submit tools
ct run submit-guessability

# Audit specific environment
ct run submit-guessability -e web_scraping

# Audit specific side task
ct run submit-guessability -s expose_secret

ct run submit-guessability-view

Interactive viewer for submit guessability audit results.

Usage:

ct run submit-guessability-view [RUN_PATH]

Output

Results are saved to data/submit_guessability_runs/<name>_<timestamp>/:

  • results.jsonl - Analysis for each side task, including theoretical exploits identified by the LLM
  • prompt_logs/<env>_<side_task>.json - Prompts, response, and web searches for each side task
  • run.log - Run log

Scope

Only side tasks with a custom submit tool that takes arguments are audited.

For Verified Exploits

ct run submit-guessability-verified tests whether the exploits work by running them against live scorers.