Porting a Setting to Control Tower

Porting brings an existing benchmark or environment into Control Tower as a native setting and reproduces it faithfully. To author a new environment from scratch, read making-environments.md instead. A project that keeps its own Inspect Task and only uses Control Tower's solver, scorers, and sandbox needs no port; see Your Own Task.

The one rule

Port thin. Reproduce the source faithfully, reuse Control Tower's own utilities, and keep the integration as small as it can be. A port is a translation, not a redesign: the setting should behave the way it does at the source, with the least Control Tower glue around it.

In practice that means preferring the smaller option at each step. Build the env config directly instead of wrapping it in a factory. Reach for an existing Control Tower utility before writing your own. Add configuration only where the source has it. Keep docs and tests proportionate to what the setting actually needs.

Fidelity comes before cleanup. If you or a reviewer spot a bug or bad practice in code you are copying verbatim, keep the logic unchanged so results stay reproducible, and flag it (e.g. in a Linear issue). Fix a clear bug that is worth fixing in a separate, clearly versioned change, so people can replicate either the original or the corrected behavior.

Layout

There is no required file layout. A setting can be a single setting.py that builds everything, which is usually the cleanest result. Split logic into more files only when it earns its keep, for example to stay close to the structure of the setting you are porting. Putting each piece (dataset loading, tasks, scorers, the submit tool) in its own file is not a goal in itself.

Every setting satisfies the same ControlEnvironment protocol (see env-architecture.md): an id, docs, a task_space of main/side tasks, and a sandbox spec. Within that protocol, ports differ along independent axes that combine freely:

  • How tasks are defined: on disk (main_tasks/, side_tasks/) or built in code from a dataset or generator.
  • Which in-container tools the agent gets: the env's base_tools set, drawn from the standard bash, text_editor, and python. Unset means the default bash and text_editor. An empty frozenset() drops them, leaving a custom-only agent that still gets its custom_tools (as in shade_arena) or, with no custom tools, a submit-only agent.
  • Whether and how Docker is used: a full agent sandbox, a minimal container just for scoring, or none at all (a local sandbox plus an LLM judge).

For example, apps is dataset-backed and submit-only but still runs a Docker scoring sandbox; sae_interp_sabotage is dataset-backed, submit-only, and Docker-free; basharena_public vendors its tasks in code yet gives the agent in-container tools and per-task sandboxes.

Discovery

A packaged setting can register its get_environments() function in pyproject.toml:

[project.entry-points."control_tower.settings"]
my_setting = "ct_my_setting.setting:get_environments"

The entry-point name is the setting ID. An installed package is loaded directly. A pulled setting that declares control-tower as a dependency is isolated instead: Control Tower runs enumeration, validation, and eval execution in that project's own uv environment, so its package and third-party dependencies do not need to be installed in Control Tower's environment. Source checkouts that are not isolated are discovered through a root setting.py or environment packages below .settings/<setting>/.

Reuse the framework

Before writing a utility, check whether Control Tower already ships it. A JSON disk cache (control_tower.utils.cache.Cache) and an LLM judge (control_tower.environments.llm_judge.llm_score) already exist, for example. If something broadly useful is missing from core, prefer adding it to Control Tower over keeping a private copy in the setting, so the next port can share it.

Keep docs thin

  • description.md (or an inline description string) is a concise description of the environment, not instructions for how to do the task. The task instructions come from the tasks.
  • A single-doc setting does not need the docs/ folder convention. Pass description inline on the env config, or keep just docs/description.md.
  • Describe the attack surface in detail on the side task, in its attack_surface / scorer_description field; a thin port does not need a separate docs/attack_surface.md. Attack-mode scoring depends on side_task_success (plus the main-task scorer when the run passes --score-main-on-attack), so the side task is where that detail is useful.
  • Keep the README short. Avoid hardcoded paths, and avoid file-tree diagrams or option lists that drift out of date; point at description.md instead.

make_environment_config_from_path() reads docs/description.md and docs/attack_surface.md from disk, so the folder convention suits settings whose tasks live on disk. Ports that build tasks in code pass description inline, as apps does.

Scorers

  • Register the main and side scorers separately (for example make_main_task_scorer() / make_side_task_scorer()); see making-scorers.md.
  • A scorer must test real behavior. Watch the dataset filter in particular: if it admits empty or mismatched test cases, scoring can run no tests and still report success, so make sure a pass means something actually ran.

Tasks and pairing

  • Build native MainTask / SideTask objects, and keep the source's prompts and task names.
  • A benchmark that crosses task templates with dataset samples becomes one MainTask per (template, sample) pair; see Main Tasks Built from a Dataset for how to keep the template and sample on each result.
  • env.task_space() is the single source of truth for valid (main, side) pairings:
    • Use ExplicitSpace when each side task is bound to one specific main (1-to-1 dataset rows, as in sae_interp_sabotage).
    • Use CrossProduct otherwise, and narrow it by declaring compatible_main_tasks / incompatible_main_tasks on the side task (see making-side-tasks.md) rather than enumerating pairs by hand.

Dependencies

A setting owns its own dependencies in its own pyproject.toml. Avoid adding a setting's third-party dependency to Control Tower's core pyproject.toml.

Validation

  • Be precise about what changed from the source. If a prompt or schema was adjusted, describe it as adapted rather than identical. Record these changes in the porting pull request (PR) description for review, and capture material deviations in the setting's per-setting reference (e.g. its Linear entry) so they stay discoverable after merge.
  • Keep tests proportionate; a test that restates the implementation is not worth carrying (see the testing guidance in the repo guidelines).
  • Validate end to end with a real ct run eval before opening the PR, and include the honest and attack results.
  • Run the evals on both the source and the Control Tower port and confirm the results are aligned. Where the source only reports numbers, check that you can roughly reproduce them.

PR checklist

  • Bridge is thin and reuses Control Tower utilities.
  • description describes the environment, not the task.
  • Scorers test real behavior, with no vacuous success.
  • Pairing is correct (ExplicitSpace for 1-to-1, otherwise CrossProduct plus compatibility).
  • The setting owns its dependencies; Control Tower's pyproject.toml is unchanged.
  • Changes from the source are described accurately.
  • ct envs validate passes and the PR shows a real ct run eval.
  • Links the source.