Running evals on Hawk

Submit evals to a shared Hawk cluster and watch them in the viewer.

0. Prerequisites

  • Ubuntu 24.04 (or similar), with Python 3.13+ and git.
  • uv, the Python project manager.
  • Optional: the GitHub CLI (gh), authenticated with gh auth login. ct settings pull (step 3) clones the public environment repos anonymously over https; gh is only needed for private repos.

1. Clone control-tower

git clone https://github.com/linuxarena/control-tower.git
cd control-tower

2. Install

uv sync
make hawk-cli

make hawk-cli installs the Hawk CLI as a standalone tool from PyPI. It compares its version with the server on every command and asks for confirmation when they differ; set HAWK_VERSION_CHECK=warn to only warn. If you had a hawk 2.x login, run hawk login again: 3.x stores credentials differently. Confirm:

ct --help
hawk --help

3. Pull the environments

The env registry starts empty. Pull the public LinuxArena envs:

ct settings pull linuxarena

This clones the 10 public envs.

4. Configure (.env)

Add these to the .env in the control-tower root (or create one if you skipped the quickstart):

export HAWK_API_URL=https://api.hawk.production.redwood-hawk.com
export CT_IMAGE_REGISTRY_PREFIX=linuxarena
export USE_OPENROUTER=1
export OPENROUTER_API_KEY=<your OpenRouter API key>

Models route through OpenRouter, so OPENROUTER_API_KEY is the only model credential you need. CT_IMAGE_REGISTRY_PREFIX=linuxarena points image pulls at the public Docker Hub org. --hawk bills from GET /api/pricing on the runner, so run ct login before submitting.

5. Log in to Hawk

source .env
hawk login

This prints a login URL. Open it in a browser and sign in with the Hawk account email and password the organizers issued you. The browser redirects to a local callback and the login completes.

Logging in from a remote or headless machine

If control-tower runs on a box with no local browser (an SSH session to a cloud VM, for example), use the device-code flow:

hawk login --device

This prints a code and a URL. Open the URL on any device with a browser and enter the code; the login completes on the headless machine once you confirm.

Confirm:

hawk list eval-sets

6. Run an eval on Hawk

This example submits the clinical_trial environment's suppress_warnings side task with the obvious-attack policy:

source .env
ct run eval -e clinical_trial -s suppress_warnings \
  --untrusted-policy obvious-attack \
  --untrusted-policy-arg model=openrouter/anthropic/claude-haiku-4.5 \
  --protocol untrusted-agent \
  --main-tasks-with-no-scorers include \
  --hawk --hawk-no-wait --hawk-no-browser

In this command:

  • --hawk submits to the cluster instead of running locally.
  • --hawk-no-wait returns right after submitting instead of blocking.
  • --hawk-no-browser skips auto-opening the viewer.
  • model=... takes any OpenRouter model slug; the openrouter/ prefix routes it through your key.

It prints an eval-set ID and a viewer URL. Watch it:

hawk status eval-set-<id>
hawk logs -f eval-set-<id>

Open the viewer URL to see samples as they run.

Custom policies and protocols

Dotted-path selections from your own project work on --hawk with no extra flags. --untrusted-policy my_pkg.policies:my_attack resolves at submission, Control Tower ships your package's source with the submission (the serialized config is capped at ~900 KB), and the runner imports it. Dotted-path refs inside selection args, such as -bp action_monitor=my_pkg.monitors:my_monitor, work the same way.

Filesystem .md monitor assets are not shipped, so on --hawk select monitors by packaged name or module:function. The runner environment has control-tower and its dependencies installed and nothing else, so your package's third-party dependencies must be among them.

7. Viewing your runs

Samples are viewable live at viewer.hawk.production.redwood-hawk.com (or your deployment's viewer host); log in with the same account you used for hawk login.

Without --hawk-no-wait, Control Tower blocks until the eval set finishes. On success it runs hawk download, converts the .eval file(s) to Control Tower trajectories, and uploads them; --no-upload skips the conversion and upload and leaves only the downloaded .eval files. On failure it exits non-zero without downloading anything. With --hawk-no-wait, Control Tower returns right after submitting and prints the follow-up commands for you to run.

To fetch the raw .eval files yourself at any point, finished or still running:

hawk download eval-set-<id> -o <dir>

8. Long runs and rate limits

Rate limits slow runs down but don't kill them. Inspect retries rate-limited requests indefinitely with exponential backoff (3s initial, capped at 30 minutes per wait, honoring Retry-After). Backoff does not consume a sample's working limit, but it does consume --sample-time-limit wall clock, so on long runs bound work with --token-limit or --message-limit rather than a tight sample time limit.

Samples that error for other reasons (sandbox failures, non-retryable API errors, limit overruns) don't abort the run either. The runner pod finishes the remaining samples, then re-runs only the errored ones, up to 10 attempts, reusing completed samples. This happens cluster-side with no action from you.

Throughput is capped at 10 concurrent connections per provider by default. Raising max_connections in your policy's model config helps only if your API tier has headroom; otherwise it just means more backoff.

For overnight runs, either keep the submitting terminal alive (in tmux or similar) so results are downloaded, converted, and uploaded on completion, or pass --hawk-no-wait and run the printed hawk download command yourself later.

To check whether a run is stalled on rate limits, watch hawk logs -f eval-set-<id> for retry messages; long-idle samples in the viewer with no new events are usually waiting out backoff. Errored samples appear in the downloaded .eval file with an error field and convert like any other sample, just without scores.

There is no command to rerun only failed samples. For samples that still end errored after the automatic retries, submit a fresh run scoped to those tasks.

Hawk controls retry, failure, and concurrency settings cluster-side, so ct run eval --hawk rejects --retry-on-error, --no-fail-on-error, and --max-samples. Run locally when you need to control them.