no runner neededScores
I already have predictions
Return one score for each frozen query and an ObserverCard describing the method and its access.
Prepare a submission →Local execution
Run your method where you already work.
ObserverBench does not need your model weights or execute uploaded code. Run the observer locally, then hand the benchmark predictions and a short description of how they were produced.
No model download
Fit an observer on cached Qwen measurements
The open Copy-v2 pack supplies 40 calibration interventions and 128 queries. It reports effect-prediction error and the loss of the action your predictions select.
Both are open practice tasks with public targets. They provide immediate feedback and do not enter the sealed leaderboards.
Choose a boundary
What does your observer produce?
The method can use activations, attribution, logits, SAE features, text, or another signal. The task-facing output stays small.
inference-freeEffects
I predict intervention effects
Fit on permitted measurements, predict held-out interventions, and test both prediction error and the action chosen from the map.
Open the effect runner →local scoresSafety
I have a safety monitor
Supply risk scores. ObserverBench applies the same audit, block, or escalation budget and reports realized harm and clean utility.
Open the safety runner →Python contractControl
I have a state estimator
Compose an estimator and actuation direction with a fixed controller, intervention rule, and control loss.
Open the control runner →Common output
Different methods leave the same evidence.
This boundary makes comparisons possible without forcing every observer into one software stack.
Predictions
Scores or estimates tied to fixed query IDs.
ObserverCard
Access, fitting, cost, version, and known failures.
Task result
Prediction quality and downstream action loss reported separately.
Reproducibility
Task version and hashes needed to repeat the comparison.
The notebooks reproduce named studies.
The repository includes Qwen and Gemma notebooks for induction, attribution patching, SAE probes, and safety experiments. They are useful examples of model access and measurement hooks, but they are not one generic hosted runner. Browse the notebooks.
Public boundary
A local run is not automatically a public score.
Automatic preflight currently accepts only the target-free Qwen2.5 paired-scope v1 prediction table. It checks data files and never executes contributor code. Sealed scoring is not active today, so a passing preflight is not yet a benchmark rank.