Running evals on Hawk
Submit evals to a shared Hawk cluster and watch them in the viewer.
0. Prerequisites
- Ubuntu 24.04 (or similar), with Python 3.13+ and git.
uv, the Python project manager.- Optional: the GitHub CLI (
gh), authenticated withgh auth login.ct settings pull(step 3) clones the public environment repos anonymously over https;ghis only needed for private repos.
1. Clone control-tower
git clone https://github.com/linuxarena/control-tower.git cd control-tower
2. Install
uv sync make hawk-cli
make hawk-cli installs the Hawk CLI as a standalone tool from PyPI. It compares its version with the server on every command and asks for confirmation when they differ; set HAWK_VERSION_CHECK=warn to only warn. Confirm:
ct --help hawk --help
3. Settings and versions
A Hawk run needs no setting checked out: the runner clones each setting the run names, installs its ct_plugin, clones the envs its envs.lock.yml pins that the run uses, and resolves the tasks, policies and protocol there. Name the envs in full (-e linuxarena/clinical_trial) or with a task set (--task-file linuxarena/simple), since they resolve on the runner. --all runs every task of the -e envs; without -e it takes every env of every setting the run clones, each at its lockfile pin unless use names the env, and needs --hawk-arg use= to name a setting. A misspelled name fails the eval set before any sample starts.
Each repo runs at the first of:
| repo | --hawk-arg use=… | else | else |
|---|---|---|---|
| Control Tower | control_tower@<ref> | the Control Tower you run, from its checkout's commit, git commit or release | |
| a setting | linuxarena@<ref>, or <setting>=<owner>/<repo>@<ref> to name its repo (one the registry doesn't list, or a fork) | its checkout in .settings/, at HEAD | the ref the settings registry gives it (main unless pinned) |
| an env | linuxarena/clinical_trial@<ref> | its checkout in .settings/<setting>/, at HEAD | the commit its setting's envs.lock.yml pins |
A ref is a tag, branch or SHA, and several go comma-separated: --hawk-arg use=linuxarena@v4.1.0,control_tower@v6.1.0. The log records the commit each setting and env ran at (setting_versions, env_versions) and Control Tower's (ct_git). A checkout ships exactly what it holds: --hawk refuses uncommitted changes and untracked files, and commits origin cannot serve, in Control Tower and in every setting and env checkout it ships. A task set comes from the cloned setting; any other task file ships by contents. Env keys go in the setting's values file in the settings dir (.settings/linuxarena.env), which ships with the run whether or not the setting is checked out (see Giving a Service a Variable).
At submit, --hawk reads each setting's envs.lock.yml with git at the commit its ref names, from your checkout or else the setting's origin, has the runner clone that commit, and declares the setting and every env repo the lock file pins as Hawk code_sources, which the submit summary lists.
To develop against the envs locally as well, pull them:
ct settings pull linuxarena
4. Configure (.env)
Add these to the .env in the control-tower root (or create one if you skipped the quickstart):
export HAWK_API_URL=https://api.hawk.production.redwood-hawk.com export CONTROL_TOWER_IMAGE_REGISTRY_PREFIX=linuxarena export USE_OPENROUTER=1 export OPENROUTER_API_KEY=<your OpenRouter API key>
Models route through OpenRouter, so OPENROUTER_API_KEY is the only model credential you need. CONTROL_TOWER_IMAGE_REGISTRY_PREFIX=linuxarena points image pulls at the public Docker Hub org. --hawk sends no Docker Hub credential to the runner: a private env's images come from the deployment's protected task images, which Hawk lets a run pull when your code groups cover the env's repo (see Private env images).
--hawk ships the rate table CONTROL_TOWER_PRICING names to the runner, as --ec2 does, and off turns pricing off there too. The default, file, needs a table from uv run python -m control_tower.scripts.generate_model_rates; without one, set CONTROL_TOWER_PRICING=api. A viewer token (ct login) is needed only for api pricing or a --from-run hosted on a private viewer; a public viewer needs none.
Private repos
The runner installs Control Tower and each setting's plugin from their origins and clones each setting and env the run uses. A Hawk deployment can hold a GitHub credential of its own for private repos. With a GitHub App, it serves a repo a code-access rule covers only if the run declares it and your code groups cover it. For private repos the deployment cannot read, set a GitHub PAT with read access to them in .env, and send it to the runner with --hawk-arg ship_git_token=true:
export CONTROL_TOWER_HAWK_GIT_TOKEN=github_pat_...
Without ship_git_token, the token stays on your machine, where --hawk uses it to read a private setting's envs.lock.yml at submit, and the runner clones with the deployment's credential. With it, --hawk ships the token as a name-only --secret, so it stays out of the eval-set config and the .eval log, and the runner's git uses it for github.com in place of the deployment's credential. It must then read every private repo the run uses, including those the deployment's credential covers, and a fine-grained PAT reads only one owner's repos.
Ship it only when the deployment's credential cannot read a repo. The deployment's credential gives a run a short-lived, read-only token for each repo it declared; your token sits in the runner for the whole run and reads everything it can, declared or not. To send it on every run, add ship_git_token=true to CONTROL_TOWER_HAWK_ARGS.
Restricted credentials
By default the runner works with your own Hawk login, so the run can read every eval set you can and call every model you can. CONTROL_TOWER_HAWK_RESTRICTED=1 in .env submits evals and monitor scans with Hawk's restricted credentials instead: the runner gets a token limited to the models the run declares, its own output, and the eval sets it reads, which Control Tower lists for it (a replay's recording, a scan's eval set). The run's results then carry those eval sets' model and code restrictions. A recording read from a viewer run is no Hawk eval set, so a replay of one inherits nothing from it.
Control Tower always states the mode, so a deployment's own default never decides it. Set the variable on a deployment that requires restricted credentials; a deployment whose token service is off refuses the submission.
5. Log in to Hawk
source .env hawk login
This prints a login URL. Open it in a browser and sign in with the Hawk account email and password the organizers issued you. The browser redirects to a local callback and the login completes.
Logging in from a remote or headless machine
If control-tower runs on a box with no local browser (an SSH session to a cloud VM, for example), use the device-code flow:
hawk login --device
This prints a code and a URL. Open the URL on any device with a browser and enter the code; the login completes on the headless machine once you confirm.
Confirm:
hawk list eval-sets
6. Run an eval on Hawk
This example submits the clinical_trial environment's suppress_warnings side task with the obvious-attack policy:
source .env ct run eval -e linuxarena/clinical_trial -s suppress_warnings \ --policy obvious-attack \ --model-role untrusted=openrouter/anthropic/claude-haiku-4.5 \ --protocol null-blue-team \ --main-tasks-with-no-scorers include \ --hawk --hawk-arg wait=false --hawk-arg open_browser=false
In this command:
--hawksubmits to the cluster instead of running locally.--hawk-arg wait=falsereturns right after submitting instead of blocking.--hawk-arg open_browser=falseskips auto-opening the viewer.--hawk-arg use=<repo>@<ref>,…picks the version of Control Tower, a setting or an env, as in Settings and versions.CONTROL_TOWER_HAWK=1makes--hawkthe default forct run evalandct run control-eval, andCONTROL_TOWER_HAWK_ARGSsets default--hawk-argvalues, space-separated (CONTROL_TOWER_HAWK_ARGS="wait=false").--no-hawk,--ec2and a--hawk-argoverride them for one run. A--run-configrun takes them too.--model-role untrusted=...binds the untrusted model; an OpenRouter slug (theopenrouter/prefix routes it through your key) works, e.g.openrouter/anthropic/claude-haiku-4.5. Unbound,untrustedandtrustedtake their Control Tower aliases, as in a localct run eval, and every bound role's model is declared to hawk. The runner refuses a model the launcher did not declare when a role binds it (as a protocol from a setting only the runner checks out may) or env code asks for it, and refuses every such model when the launcher sent no list (a Control Tower older than the runner's).
It prints an eval-set ID and a viewer URL. Watch it:
hawk status eval-set-<id> hawk logs -f eval-set-<id>
Open the viewer URL to see samples as they run.
Custom policies and protocols
A policy or protocol from a setting's ct_plugin runs on --hawk by its full name (--policy shade_arena/attack): the runner installs the plugin of every setting it clones. A monitor in a policy's or protocol's args names its setting the same way ({monitor: shade_arena/<monitor>}). ct run monitor --hawk clones the setting a monitor's full name or --hawk-arg use= names in the same way.
Filesystem .md monitor prompts are not shipped, so on --hawk select monitor prompts by packaged name or use a registered monitor factory.
Replaying a run
On --hawk, --from-run and the replay policy's traj_like take any source a local run takes, and the runner replays it in a real Kubernetes sandbox at no model cost. The runner fetches a viewer run itself and reads a hawk://<eval-set-id>[/<eval-id>] in place with your Hawk login, pinned to the log Control Tower read at launch. A local file or an s3:// log is imported into Hawk first with hawk import, as an eval set of its own (imported-<name>-<id>) holding a .eval copy under a fresh eval id, and the run reads it from there. Each replay of a local file imports it again: Hawk imports an eval id only once and won't say which eval set holds it, and its eval-set search lags imports by minutes, so an earlier import can't be found reliably. A -t or -s override from an env the recording does not use needs that env named with -e.
uv run ct run eval --hawk --from-run https://data.linuxarena.ai/runs/<run-id> --limit 1 --policy ct/replay
7. Viewing your runs
Samples are viewable live at viewer.hawk.production.redwood-hawk.com (or your deployment's viewer host); log in with the same account you used for hawk login.
Without --hawk-arg wait=false, Control Tower blocks until the eval set finishes. On success it runs hawk download, prints each task's metrics (for a control eval, usefulness and safety), and, with --upload, converts the .eval file(s) to Control Tower trajectories and uploads them; without --upload it leaves only the downloaded .eval files. Hawk retries a failed attempt in the same eval set, so the download can hold one log per attempt; Control Tower prints, converts and uploads only the logs that ended success and names the others it skips. On failure it exits non-zero without downloading anything. With --hawk-arg wait=false, Control Tower returns right after submitting and prints the follow-up commands for you to run, so it refuses --upload.
To fetch the raw .eval files yourself at any point, finished or still running:
hawk download eval-set-<id> -o <dir>
ct run monitor hawk://<eval-set-id> downloads and monitors a set; see Monitoring for --hawk.
8. Long runs and rate limits
Rate limits slow runs down but don't kill them. Inspect retries rate-limited requests indefinitely with exponential backoff (3s initial, capped at 30 minutes per wait, honoring Retry-After). Backoff does not consume a sample's working limit, but it does consume --eval-arg time_limit wall clock, so on long runs bound work with --eval-arg token_limit or --eval-arg message_limit rather than a tight time limit.
Samples that error for other reasons (sandbox failures, non-retryable API errors, limit overruns) don't abort the run either. The runner pod finishes the remaining samples, then re-runs only the errored ones, up to 10 attempts, reusing completed samples. This happens cluster-side with no action from you.
Throughput is capped at 10 concurrent connections per provider by default. Raising max_connections in your policy's model config helps only if your API tier has headroom; otherwise it just means more backoff.
For overnight runs, either keep the submitting terminal alive (in tmux or similar) so results are downloaded, converted, and uploaded on completion, or pass --hawk-arg wait=false and run the printed hawk download command yourself later.
To check whether a run is stalled on rate limits, watch hawk logs -f eval-set-<id> for retry messages; long-idle samples in the viewer with no new events are usually waiting out backoff. Errored samples appear in the downloaded .eval file with an error field and convert like any other sample, just without scores.
There is no command to rerun only failed samples. For samples that still end errored after the automatic retries, submit a fresh run scoped to those tasks.
Hawk's runner sets retry, failure, sample-logging and sandbox-cleanup settings itself and has no token limit type, so ct run eval --hawk rejects fail_on_error, log_buffer, log_images, log_samples, log_shared, retry_on_error, sample_id, sandbox_cleanup and token_limit_type, whether given with --eval-arg or in a run config's eval_config. Run locally when you need to control them. max_samples, max_tasks, max_sandboxes and max_subprocesses go to Hawk, which refuses a value over its deployment's ceiling. epochs_reducer is sent as Hawk's epochs: {epochs, reducer}.