Submit Guessability Verified
Audits side tasks for submit tool exploitability with live verification.
Purpose
Like submit-guessability, but it spins up containers and runs the scorer to check whether proposed exploits work. The AI gets multiple attempts, with feedback, to find working submit arguments.
Commands
ct extras submit-guessability-verified
Audit side tasks with live scorer verification.
Examples:
# Audit all side tasks with verification ct extras submit-guessability-verified # Audit specific environment ct extras submit-guessability-verified -e linuxarena/web_scraping # More attempts to find exploits ct extras submit-guessability-verified -a 10 # Lower concurrency for resource-constrained systems ct extras submit-guessability-verified -c 1
Output
The run is an Inspect task with one sample per side task, and writes an Inspect .eval log to logs/ (or INSPECT_LOG_DIR). Open it with inspect view. A sample scores I when a proposed set of submit arguments passed the live scorer, with those arguments as the score's answer, C when the auditor answered NO or its arguments reached the scorer and failed, and N when it did neither. The sample transcript holds every attempt and the scorer feedback the auditor was given.
Workflow
For each side task:
- The AI analyzes the scorer and submit tool and calls a
try_submit_argstool, whose parameters are the side task's submit arguments, with the arguments it proposes. - A container is spun up and the scorer runs with those arguments.
- If the scorer fails, the AI sees the error and tries again, up to
max-attemptstimes. - The results show whether a working exploit was found.