Architecture
Experiment Runner
How hypothesis batches become branches, arms, trials, replays, statistics, and effects.
Experiments automate repeated replay work. A hypothesis batch describes what to test; the runner materializes branches and arms, runs replays, aggregates requirements, and ranks effects.
Input Shape
An experiment plan includes:
- project ID and base run ID
- checkpoint selector
- replay policy
- hypotheses
- interventions
- trial plan
- validity gates
- metadata
Runner Flow
Resolve the checkpoint selector to a checkpoint.
Create the experiment record.
Create one arm per hypothesis or configured branch.
Create branches and interventions for each arm.
Run trials by creating replays.
Collect paused fixture requirements.
Build trial matrix, statistics, arm comparison output, and ranked effects.
Current Execution Model
The runner can execute inline for wait=true or through a durable Graphile Worker job for wait=false. Within one experiment, replay execution is still serial: concurrency, maxReplays, and stopRule are parsed as planning fields but are not yet full scheduling controls.
Validity Gates
Validity gates model evidence discipline:
| Gate | Purpose |
|---|---|
minimum_trials | Require enough repetitions before support. |
max_fixture_dependency_rate | Limit fixture-sensitive conclusions. |
max_non_comparable_rate | Reject experiments with too much drift. |
requires_effect_size | Require a minimum effect signal. |
What To Improve When Experiments Are Noisy
- Use a later checkpoint.
- Split bundled interventions.
- Add better context annotations so targets are precise.
- Resolve fixtures with narrower matchers.
- Increase repetitions when nondeterminism matters.
- Treat non-comparable arms as design feedback, not as evidence.