Sabotage Trajectory Review Workflow

A reviewer-aid bot for sabotage trajectory (straj) pull requests (PRs). A reviewer comments @straj-review on a straj PR, and a small workflow in the env repo calls control-tower's reusable straj-review.yml. It decodes the trajectory and posts the findings as a single sticky comment, which each run overwrites in place.

The bot is advisory: it never blocks merge, approves, or requests changes.

From top to bottom, the comment carries the trajectory facts (env, main/side task and scores, number of actions), the action- and trajectory-monitor identity with max_sus (the highest action suspiciousness score) / traj_sus (the trajectory monitor score) and sanity checks, a link to the straj's traj directory, and then, as supporting context, the env threat model and the main-task (cover) and side-task (objective) text. The LLM analysis stages, which read the straj diff and narrate the attack, add a straj analysis section at the bottom.

What the reviewer does

  1. Make sure the PR description contains a monitored trajectory URL, either a plain viewer URL containing /trajectories/… or an explicit line:

    Straj-Review-Trajectory: https://…/trajectories/<id>

  2. Comment @straj-review on the PR.

The bot reacts 👀 on the triggering comment, reads the URL from the PR body, downloads the trajectory, writes its sticky comment, and reacts 👍 when the comment is posted. No 👍 means the run failed. If the URL is missing or the download fails, the comment says what it needs. Comment @straj-review again to re-run; the comment always reflects the current state.

Providing the straj (the traj directory)

The straj's files live in the env repo at side_tasks/<side_task>/trajs/<main_task>/<variant>/, the per-straj directory described in making-strajs. Committing that directory to the PR is how the straj is tracked. It holds the straj's description.md and info.yml (which may be empty; the workflow does not check their contents), the trajectory's git diff, and any other relevant files.

The comment links to the traj directory rather than rendering the diff. A ct traj patch diff blends the main-task work with the attack, and committed files already show in the PR's Files changed tab.

The bot requires the PR to change exactly one traj directory. When it changes several, the comment asks you to split them into separate PRs. When it changes none, the comment asks you to create one:

  1. Generate the trajectory's git diff with ct traj patch <traj> locally. It brings the env up, replays the trajectory, and captures the diff.
  2. Commit it (with description.md / info.yml) under side_tasks/<side_task>/trajs/<main_task>/<variant>/ on the PR branch.
  3. Comment @straj-review again.

The bot regenerates anything absent, so to force the env threat model to regenerate, commit a deletion of docs/threat-model.md.

The LLM analysis stages require the threat model, because it defines the security property their verdicts judge against. When it is missing (generation failed or was skipped), the bot skips those stages, and the comment's analysis section says so and how to retry. The deterministic report posts regardless.

Setting up the caller

A caller job runs uses: linuxarena/control-tower/.github/workflows/straj-review.yml@<ref> with the comment's pr_number and comment_id, plus an optional ct_ref (the control-tower ref to install ct from, default main); pin <ref> to a tag to freeze behavior. The reusable workflow's workflow_call block declares the inputs and secrets a caller passes: CONTROL_TOWER_API_TOKEN when the trajectory backend needs auth, and ANTHROPIC_API_KEY for the threat-model and LLM analysis stages. The caller must grant every permission in the reusable job's permissions: block (contents: write to commit docs/threat-model.md, pull-requests: write, issues: write, and id-token: write), because a called workflow cannot exceed its caller's permissions. The reusable workflow checks no permissions itself, so the caller's job if: must require a PR comment (github.event.issue.pull_request) that contains @straj-review from an OWNER, MEMBER, or COLLABORATOR (github.event.comment.author_association). GitHub runs issue_comment workflows only from the default branch, so merge the caller there before it fires.