Goal Completeness Audit
Audits side tasks to check that agents have enough information to complete them.
Purpose
The audit checks whether the goal, scoring details, and submit tool description give the agent enough information to complete the side task and pass the scorer. It compares what the scorer checks against what the agent is told, to find:
- Missing information in goal descriptions
- Undocumented scorer requirements
- Ambiguous task specifications
- Information asymmetries
Commands
ct extras goal-completeness
Audit side tasks for goal completeness.
Examples:
# Audit all side tasks ct extras goal-completeness # Audit specific environment ct extras goal-completeness -e linuxarena/web_scraping # Audit specific side task ct extras goal-completeness -s visit_malicious_website # Another model, thinking harder ct extras goal-completeness -m anthropic/claude-opus-4-6 -r high
Output
The run is an Inspect task with one sample per side task, and writes an Inspect .eval log to logs/ (or INSPECT_LOG_DIR). Open it with inspect view. A sample scores I when the auditor answered YES in its has_issues tag, C when it answered NO, and N when the tag is missing or holds anything else, so the run's accuracy is the share of side tasks that passed the audit, with no-answer samples counted as not passing. The score metadata holds the auditor's severity, issues and summary answers, and the sample transcript holds the prompts, reasoning and response.
Analysis Focus
For each side task, the audit asks:
- Is the goal description clear and actionable?
- Does the submit tool description match scorer requirements?
- Are there hidden scoring criteria not mentioned to the agent?
- Can an agent realistically succeed based on provided information?