CopperTide/Shakedown

Experiments for agent systems

Shakedown runs one to four agent setups on one world and scenario, N runs each, and lays the results out in columns.

A real experiment

Three setups, three runs each

Our pharma example. A Grade A environmental-monitoring sample fails during a fill, and the usual courier loses capacity. The baseline setup failed all three runs. Both fixes passed two of three.

Shakedown experiment comparing three setups: pass rate, runs, checks, safety violations, denied actions, cost and time
Each column shows one setup: pass rate with interval, a square per run, checks passed, safety violations, denied actions, flags for a human, cost and time per run.

Inputs

What defines an experiment

01

World

Systems composed from connectors. See Environments.

02

Scenario

A task plus faults scheduled on the world clock.

03

Setups

Model, team structure and rules per setup. One to four per experiment.

04

Checks

Goal and safety checks as code. Safety counts on a separate line.

05

Runs

1, 3, 5 or 10 from the UI. Any N from the API. Runs interleave across setups.

Bring your agent

Your agent, your framework

Run Claude Code or Codex headless against the world’s tools. Use the built-in loop with OpenRouter, OpenAI, the Claude API or a local OpenAI-compatible server.

Or register an HTTP agent. Each run posts the task with an MCP endpoint and a token. Your agent works through the endpoint, Shakedown records each call, and your agent answers the POST. Header values never reach a log.

from arena import Arena, Agent

arena = Arena("http://127.0.0.1:4510")   # your Shakedown server
branch = arena.run("72",
    team=[Agent.claude_code("support", model="haiku")],
    seed=7)
for event in branch.stream():       # each step, live
    print(event.summary())
print(branch.check())                 # verdict, named checks, safety
branch.export("data/run-72")          # JSONL with a schema

The Python SDK. Other languages use the HTTP API.

Suites

Suites and regressions

A suite runs one deployment over a task set, repeated, from one seeded world. Compare two suites and Shakedown marks each check as improved, regressed or unchanged.

Time, tokens and cost count as changed beyond 10% of the first suite’s mean.

  • Pass rate per task and per check
  • Spread of checks, time, tokens, cost and peak context: min, max, mean
  • A per-run spend cap, enforced mid-turn
  • Unscored runs kept apart from scored runs
Next

Environments

→