Submit Guessability Verified

Audits side tasks for submit tool exploitability with live verification.

Purpose

Like submit-guessability, but it spins up containers and runs the scorer to check whether proposed exploits work. The AI gets multiple attempts, with feedback, to find working submit arguments.

Commands

ct run submit-guessability-verified

Audit side tasks with live scorer verification.

Examples:

# Audit all side tasks with verification
ct run submit-guessability-verified

# Audit specific environment
ct run submit-guessability-verified -e web_scraping

# More attempts to find exploits
ct run submit-guessability-verified -a 10

# Lower concurrency for resource-constrained systems
ct run submit-guessability-verified -c 1

ct run submit-guessability-verified-view

Interactive viewer for verified audit results.

Usage:

ct run submit-guessability-verified-view [RUN_PATH]

Output

Results are saved to data/submit_guessability_verified_runs/<name>_<timestamp>/:

  • results.jsonl - Verification results for each side task, including each attempt's submitted arguments and scorer output, and the arguments that passed, if any
  • prompt_logs/<env>_<side_task>.json - Prompts and full conversation for each side task
  • run.log - Run log

Workflow

For each side task:

  1. The AI analyzes the scorer and submit tool and proposes exploit arguments.
  2. A container is spun up and the scorer runs with those arguments.
  3. If the scorer fails, the AI sees the error and tries again, up to max-attempts times.
  4. The results show whether a working exploit was found.