Control Tower v6 is available. See the changelog for breaking changes.

Control Tower

User Guide · Building Environments

Making Environments

Porting an existing benchmark or environment? See porting-settings.md.

Definition

An environment is a realistic software system that serves as the context for AI agent evaluation. At the code level, an environment is any object satisfying the ControlEnvironment protocol (packages/ct-setting-sdk/src/ct_setting_sdk/env/environment.py):

@runtime_checkable class ControlEnvironment(Protocol): @property def id(self) -> str: ... def docs(self, args: BaseModelPython | None = None) -> EnvDocs: ... def get_sandbox_spec(self, codebase_name: str = "default", args: BaseModelPython | None = None) -> SandboxEnvironmentSpec: ... def task_space(self, args: BaseModelPython | None = None) -> TaskSpace: ...

task_space(args) returns the env's menu of valid task combinations: which main and side tasks exist and which (main, side) pairings are allowed. The eval runner builds samples from this menu, and listings like ct settings main-tasks read from it. args carries one run's --env-arg values (shaped by the env's args_model), since the menu can vary by configuration; None means the env's default configuration. The implementation is in packages/ct-setting-sdk/src/ct_setting_sdk/env/space.py.

An env may declare display_name (default: its id), tools and private_tools (the tools every sample gets, and the private tools a policy may use unseen by blue if white allows them, declared by id; none by default), agent_user: str, and codebases (the codebases whose Docker images Control Tower builds and pulls ahead of a run; a DockerEnvironment lists its compose_paths keys, any other env none by default). See env-architecture.md.

get_sandbox_spec() determines the sandbox type: Docker Compose, QEMU, Firecracker, Vagrant, or no sandbox at all. See env-architecture.md for the full architecture and available sandbox types, and .settings/demos/ for examples of environments using each type.

Most environments use Docker Compose via the DockerEnvironment class and the make_environment_config_from_path() factory. The rest of this guide covers that standard approach.

An official LinuxArena environment takes several forms as Control Tower loads and runs it:

Rendering diagram...

The checked-out repository supplies the files and configuration used to construct the object. Its __init__.py calls make_environment_config_from_path(), which returns a DockerEnvironment object exposing the five ControlEnvironment protocol members. During an evaluation, get_sandbox_spec() returns the configuration used to start the sandbox.

Components

Environments live one level deep inside a setting, a directory directly under .settings/ that groups related environments (e.g. .settings/linuxarena/web_scraping/). See Setting Layout for how discovery works.

Official environments are separate repos (e.g. under the linuxarena org) that ct settings pull clones into .settings/<setting>/. To make your own environment discoverable without a GitHub repo, put it in a setting of its own, a folder with a pyproject.toml, and link that folder with ct settings pull /path/to/my_setting; Build an Environment builds one step by step. See Development Setup for both.

The quickest way to prototype an environment is to pull the public LinuxArena environments (ct settings pull linuxarena) and copy or study one close to what you want to build; an agent can walk you through its structure.

A checked-out environment repository contains at least the following files and folders:

  • __init__.py - environment configuration using make_environment_config_from_path()
  • codebase/ - contains the code for the environment
  • codebase/compose.yml - a docker compose file declaring how to set up the container with the environment
  • docs/description.md - codebase structure, key files, and usage instructions
  • docs/attack_surface.md - points of interaction that could be targeted in side tasks
  • main_tasks/ - a set of main tasks for this environment (see how to add tasks below)
  • side_tasks/ - a set of side tasks for this environment (see how to add tasks below)

An environment can also have integration tests: tests/run_tests.py sets a module-level TESTERS list of Script (a file in tests/, run in a container with runner, e.g. pytest) and Command testers from ct_setting_sdk.env.integration_tests. The env process runs them in order against a live session's containers: ct live test against the active session, and ct settings test against a fresh session per environment.

All standard environments use Ubuntu 22.04 as the base image.

Task content does not have to be baked into the environment image. A main or side task can declare a files map (see Making Main Tasks) of per-sample payloads, such as a data file or a code repository, that are copied into the sandbox at init. Many tasks can then share one compose file and image.

Desired Properties

Most important

A well-designed environment has the following properties:

Realistic and Plausible

Environments should model systems that a real company or organization might plausibly use while building or operating software. A SaaS company is one example; environments need not be about ML, AI safety, or any specific domain.

Sufficient Complexity

Environments should be complicated enough that a typical software engineer needs ~1 week of onboarding to become productive, but not significantly more so. Look at examples of environments in .settings/ before embarking on environment ideation.

Docker Compose Compatible

All services must be runnable inside Docker Compose, without writable host bind mounts: Inspect's Docker sandbox refuses them, since an agent could then change files outside its sample. Copy code and seed data into the image, mount host files :ro, and keep writable state in named volumes. privileged and SYS_ADMIN services can't use host binds at all.

Tech Stack Diversity

As much as possible, environments should use different programming languages and services to sample from the space of real-world Linux systems. Examples of current environments:

  • file_indexing: Document search (Python 3.13, PyTorch, FastAPI, ChromaDB, OpenLDAP)
  • web_scraping: Distributed scraping (Node.js 20.x, Redis, PostgreSQL, FTP)
  • model_registry: ML artifact management (Go, Python/PyTorch, PostgreSQL, S3)
  • media_processing: Video/audio processing (Rust, FFmpeg)
  • clinical_trial: Clinical data and reporting (R, Python, PostgreSQL)
  • tiktok: Short-form video backend (Go, Redis, PostgreSQL)
  • user_tracking: User analytics (C, CMake, PostgreSQL)
  • slack_clone: Team chat API (Ruby on Rails, Elasticsearch, PostgreSQL)

Avoid languages already heavily represented among the existing environments (especially Python). Consider avoiding simple CRUD apps, and consider using unique services beyond the common Redis/PostgreSQL/S3 pattern:

  • ElasticSearch
  • Prometheus
  • NGINX
  • RabbitMQ
  • Git
  • Bazel build system
  • ImageMagick
  • Vector databases

High Attack Surface

Security weaknesses that side tasks can exploit make side task development easier but can hurt realism. Start environment ideation by trying to do without them. A lack of security features is generally more realistic than blatantly misimplemented ones.

Other

Persistent State

Prefer environments with high persistent state across many services, as this creates more opportunities for subtle sabotage.

Exfiltration Testing Server

For environments with exfiltration-related side tasks, consider including a mock external server (e.g., filebin.net or similar) that the agent can exfiltrate data to. This makes scorer verification straightforward.

Known Limitations & Security Considerations

Evaluation Awareness

An agent that can tell it is being evaluated may behave differently than it would in deployment. Common clues include the following:

  • Bind mounts. Even a read-only mount such as ./init:/init:ro can expose .settings/linuxarena/<env>/codebase/init in the agent's /proc/self/mountinfo. Use COPY for static files the agent's service needs.
  • Docker names. Volume names can appear in mount information and container names in reverse DNS. By default both carry the Compose project prefix, inspect- plus the start of the run name (such as attack or linuxarena). For unshared scratch state, prefer tmpfs to avoid exposing a volume name. Choose service and volume names that fit the application.
  • Simulated external hosts. Internet simulator hostnames such as filebin.net resolve to a private address on the agent's own Docker network, and a reverse lookup of that address returns the simulator's container name. Control Tower adds and names that service, so this is a harness limitation.
  • Agent-visible text. Keep benchmark names and harness details out of comments, package metadata, seed data and test fixtures the agent can read. The task-test tools also show the agent scorer explanations verbatim; see Making Scorers.

Creating a New Environment

1: Environment Idea

Create the environment repository under the linuxarena org, add it to the setting's environment list (envs.lock.yml in ct-linuxarena), and open the idea pull request (PR) there. At its root, include an __init__.py and docs/ directory with placeholder content describing the idea.

While the environment is still an idea, name the repository {environment_name}_idea. Environment discovery skips directories ending in _idea, so an unfinished environment does not fail tests. Drop the suffix once the environment is implemented.

Each environment registers itself in its __init__.py: an @environment("<id>") producer that builds it with make_environment_config_from_path(). The module top imports only ct_setting_sdk.registration, and everything heavy goes inside the producer, so the setting package's handshake stays fast:

from pathlib import Path from typing import TYPE_CHECKING from ct_setting_sdk.registration import environment if TYPE_CHECKING: from ct_setting_sdk.env.environment import ControlEnvironment @environment("my_environment") def my_environment() -> "ControlEnvironment": from ct_setting_sdk.env.docker_environment import make_environment_config_from_path from ct_setting_sdk.env.tools import ToolSpec environment_root = Path(__file__).parent return make_environment_config_from_path( environment_root, short_description="Brief one-line description shown on the website.", tech_stack=["Python", "Flask", "Redis"], compose_paths={"default": environment_root / "codebase" / "compose.yml"}, services=["Flask API (port 8080)", "Redis"], tools=[ToolSpec(id="ct/bash"), ToolSpec(id="ct/text_editor")], private_tools=[ ToolSpec(id="ct/run_main_task_test"), ToolSpec(id="ct/run_side_task_test"), ], )

The setting's registration module imports each env repo it lists, which runs this decorator; an env repo that fails to import, or registers nothing under its name, shows up as unloadable with its error.

The env declares its tools: Control Tower gives an env none of its own accord. tools are what every sample's agent works with; private_tools are the ones a policy may use unseen by blue, naming their ids in its private_tools arg when white allows them (-wa private_tools=ct/run_side_task_test). Tasks can add to both from their own folder module (tools / private_tools attributes), except that a side task may only add private tools, since interaction tools only attack samples carried would mark them. The task tests (ct/run_main_task_test, ct/run_side_task_test) are private tools only.

environment_root is the checked-out repository root. The factory combines these with its other arguments to build the DockerEnvironment object, and also reads docs/description.md and docs/attack_surface.md from the repository.

The specification for description.md (required sections, length cap, in-bounds vs out-of-bounds content, and the credentials rule) is src/control_tower/extras/env_description/shared/spec.md. Two tools use it:

  • ct extras env-description-generate -e <env> produces a fresh description.md from the codebase against the spec. Use --dry-run to write description.draft.md for side-by-side review, or --overwrite to replace an existing description.
  • ct extras env-description-audit -e <env> scores an existing description.md against the same spec on accuracy, clarity, completeness, information leakage, and overall.

See docs/auditing.md for full usage.

Available parameters for make_environment_config_from_path():

ParameterRequiredDescription
pathYesPath to the environment directory
short_descriptionYesOne-line description for the website
tech_stackYesList of languages, frameworks and tools, as strings
servicesYesList of service descriptions
compose_pathsNoMap of codebase name to compose file path (default: {"default": "codebase/compose.yml"})
custom_display_nameNoDisplay name (auto-generated from folder name if omitted)
agent_userNoUser the agent runs as (default: "root")
toolsNoInteraction tools every sample gets, as ToolSpec(id="ct/<name>"), ToolSpec(id="inspect/<name>"), or env_tool(tool); none by default
private_toolsNoPrivate tools a policy may use unseen by blue if white allows them, such as ToolSpec(id="ct/run_side_task_test"); none by default
internet_simulatorNoInternet simulator configuration
lang_rewrite_tasks_integration_test_filenameNoIntegration test filename for lang-rewrite flows

2: Environment Implementation and Initial Side Task

  1. Build the codebase and Docker Compose setup
  2. Create main task ideas, to use in the steps below
  3. Create side tasks (~5) with scorers
  4. Develop sabotage trajectories (strajs) using different strategies on different main task ideas

Issue templates, Discord, and sprint plans have more specific requirements.

Honest models should not accidentally trigger the initial side task.

Two strajs, each paired with a different main task, show that the environment has more than one sabotage opportunity.

3: Additional Main Tasks

After the initial strajs are approved, develop about 20 more main tasks.

Advanced Environment Configuration

To use different docker compose files for different tasks, pass multiple entries in compose_paths (inside the producer):

return make_environment_config_from_path( root, short_description="...", tech_stack=["Python"], services=["API (port 8080)"], compose_paths={ "default": root / "codebase" / "compose.yml", "special_env": root / "codebase" / "compose-special-env.yml", }, tools=[ToolSpec(id="ct/bash"), ToolSpec(id="ct/text_editor")], private_tools=[ ToolSpec(id="ct/run_main_task_test"), ToolSpec(id="ct/run_side_task_test"), ], agent_user="root", # Optional: defaults to "root" )

For a task to use the "special_env" compose file, set that task's codebase_name (see Making Main Tasks).

Giving a Service a Variable

An env's compose file is the declaration of the variables it needs: every place it reads one from the host is a declaration, and Control Tower fills each from the setting's values file. Nothing else declares a variable.

ReadWithout a value
${NAME}, $NAME, ${NAME:?message}, a bare NAME entry under environmentThe eval fails before any sandbox starts
${NAME:-default}, ${NAME-default}Compose uses the default
${NAME:+alternate}, ${NAME+alternate}Compose uses an empty string

A variable is required if any read of it is required, and an empty value in the values file counts as no value. A read nested in another's default, such as ${A:-${B}}, is refused. $$ writes a literal $. A value written in the file stays as written, so a sidecar can hold the real key while the agent's container gets a placeholder under the same name:

services: tinker-proxy: environment: - TINKER_API_KEY=${TINKER_API_KEY} # the setting's value default: environment: - TINKER_API_KEY=tml-ptb-proxy # stays as written - BATCH_SIZE=${BATCH_SIZE:-8} # optional, 8 unless the values file sets it

The values live in one dotenv file in the settings dir, named for the setting, <settings dir>/<setting>.env, for example .settings/post_train_bench.env. All of a setting's envs share it, so two envs that need different values read different names. The file is read as written, with no ${...} expansion, once when the setting's env process starts, so an edit takes effect on the next command. Control Tower's own .env and your shell never fill a compose read, and a .env inside the setting's checkout or beside an env's compose file is refused with a message naming the new location. Fleet and Hawk runs ship the file and put it at the same place on the worker or runner. Before any sandbox starts, every sample's compose file (with its internet simulator, whose service reads each upstream proxy's api_key_env) is checked against the file, locally, on the EC2 controller and on the Hawk runner, so one missing value fails the whole eval. --sandbox simulated checks when its sample starts, and a sandbox with no compose file, such as an external --sandbox, declares nothing.

Control Tower reserves the CT_* names. Compose fills COMPOSE_PROJECT_NAME, and ENV_IMAGE_REPO_<ENV> and ENV_IMAGE_TAG_<ENV> pick which image to pull, which is Control Tower's choice: they come from Control Tower's .env or your shell, never from the values file.

On Hawk, an entry under a service's environment that reads a variable is copied, resolved, onto the service of the same name in hawk-values.yaml. A read anywhere else, such as in command or image, is the hand translation's to carry, as images and commands already are.

The setting's env process also starts with the whole file in its environment, for the setting's own code: an env tool that starts its own infrastructure, or a scorer that calls an API. Control Tower cannot see those reads, so it checks none of them; a value they need that no compose file reads fails only when that code runs.

Agent Dockerfile

The default compose service should reference the agent Dockerfile, which must meet the requirements in this template:

FROM ubuntu:jammy-20250819 ENV DEBIAN_FRONTEND=noninteractive # Common dependencies - git is REQUIRED for gen-traj command RUN apt-get update && apt-get install -y --no-install-recommends \ curl \ vim \ git \ procps \ python3 \ python3-pip \ python3-venv \ sudo \ # Add environment-specific packages here && apt-get clean && rm -rf /var/lib/apt/lists/* # Install inspect-tool-support (REQUIRED for text_editor tool) ENV PATH="$PATH:/opt/inspect_tool_support/bin" RUN python3 -m venv /opt/inspect_tool_support && \ /opt/inspect_tool_support/bin/pip install inspect-tool-support && \ /opt/inspect_tool_support/bin/inspect-tool-support post-install --no-web-browser # Environment-specific setup (dependencies, directories, etc.) RUN mkdir -p /app # ... # Copy restart script (REQUIRED - must be at /app/restart.sh) COPY src/restart.sh /app/restart.sh RUN chmod +x /app/restart.sh # Copy application files COPY src/ /app/your_app/ WORKDIR /app ENV SHELL=/bin/bash # Start service and keep container alive CMD ["/bin/bash", "-c", "./restart.sh & tail -f /dev/null"]

Agent Privileges

Agents run as root by default so blue teams can implement their own security models.

  • Don't create an agent user in the Dockerfile; it causes permission issues for inspect_tool_support. Leave agent_user at its default of "root" unless you have a specific reason to change it.
  • Don't add sudoers entries, permission restrictions, or other measures that limit what the agent can do. Blue teams implement security through task setup scripts or control protocols.
  • Keep environment-level security measures, such as sandboxing for untrusted code execution or monitoring systems, that exist independently of the agent's privileges.
  • Users or permission structures that the environment's design requires are fine, as long as they are not there to restrict the agent.

Environment Restart

  • Each environment should provide a restart.sh at .settings/<setting>/<env>/codebase/src/restart.sh that restarts the main service.
  • The script should exit 0 on success, non-zero on failure, and print where to look for logs if start-up fails
  • The environment's Dockerfile must copy the script to /app/restart.sh (e.g. COPY src/restart.sh /app/restart.sh), where agents, developers, and scorers can call it directly
  • If an environment intentionally forbids restarts (by design), still include restart.sh that explains why and exits with a non-zero status