no runner neededScores
I already have predictions
Return one score for each frozen query and an ObserverCard describing the method and its access.
Prepare a submission →Local execution
Run your method where you already work.
ObserverBench does not need your model weights or execute uploaded code. Run the observer locally, then hand the benchmark predictions and a short description of how they were produced.
New here? Start with a simple safety task and saved measurements from Qwen2.5-7B base. One command reports prediction quality and the outcomes of the decisions those predictions guide. No GPU or model download is needed.
No model download
Fit an observer on cached Qwen measurements
The open Copy-v2 pack supplies 40 calibration interventions and 128 queries. It reports effect-prediction error and the loss of the action your predictions select.
The IOI replay returns chosen actions, loss, and paired comparisons with four observers and no-op. All practice tasks have public targets: they provide immediate feedback but do not enter private-test rankings.
CPU only · open practice
Buy evidence before choosing an action
A report may hide a disallowed operation or misstate where an object came from. Spend a limited budget on trusted facts before a fixed controller allows, blocks, or escalates the request.
python -m pip install -e .
make demo-integrity
make demo-relayRun these from the repository root. Edit the example's measurement-selection and risk-prediction functions to try your own observer. Each run compares it with four baselines and reports loss including inspection cost.
The relay history is scripted and shares the first task's generator. It is not an independent replication or live multi-agent experiment. Public local targets make these practice scores ineligible for private-test rankings.
Choose a boundary
What does your observer produce?
The method can use activations, attribution, logits, SAE features, text, or another signal. The task-facing output stays small.
inference-freeEffects
I predict intervention effects
Fit on permitted measurements, predict held-out interventions, and test both prediction error and the action chosen from the map.
Open the effect runner →local scoresSafety
I have a safety monitor
Supply risk scores. ObserverBench applies the same audit, block, or escalation budget and reports realized harm and utility on safe requests.
Open the safety runner →Python contractControl
I have a state estimator
Compose an estimator and edit direction with a fixed controller, intervention rule, and control loss.
Open the control runner →Common output
Different methods leave the same evidence.
This boundary makes comparisons possible without forcing every observer into one software stack.
Predictions
Scores or estimates tied to fixed query IDs.
ObserverCard
Access, fitting, cost, version, and known failures.
Task result
Prediction quality and downstream action loss reported separately.
Reproducibility
Task version and hashes needed to repeat the comparison.
The notebooks reproduce named studies.
The repository includes Qwen and Gemma notebooks for induction, attribution patching, SAE probes, and safety experiments. They are useful examples of model access and measurement hooks, but they are not one generic hosted runner. Browse the notebooks.
Public boundary
A local run is not automatically a public score.
Automatic preflight currently accepts only the target-free Qwen2.5 paired-scope v1 prediction table. It checks data files and never executes contributor code. Private-test evaluation is not active today, so a passing preflight is not yet a benchmark rank.