A task-relative observer workbench
ObserverBench compares internal-state estimates under a fixed task, intervention rule, budget, and loss. It reports both how well an observer predicts and what happens when a controller acts on it.
Mechanistic Tomography develops the measurement framework; ObserverBench tests the resulting estimate through action.
Task catalog
Reading activations next to the verdict prompt gives 1.976 mean violations, versus 6.601 in neutral context. AUROC also improves, from 0.936 to 0.977. The action advantage persists when source problems are resampled.
On the recorded panel, the detailed prompted monitor has higher AUROC than the Gemma Scope SAE probe (0.893 vs. 0.870), but more mean violations (10.80 vs. 9.07). The action ranking is uncertain when source problems are resampled.
The SAE readout uses 127 fitted coefficients instead of 3,584 residual coordinates, but mean violations rise from 2.05 to 4.67. Prediction quality also falls. This measures readout sparsity, not a reduction in total compute.
On the induction-copy task, an interaction-aware predictor lowers held-out MAE from 0.121 to 0.040 with the same 128-measurement budget.
Run and score locally
Open practice tasks return scores on your computer. You do not need a GPU, a submission review, or the private-test evaluator.
Start with saved Qwen2.5-7B copy measurements, or a safety practice task.
The Qwen example fits an observer on 40 measurements and predicts 128 new intervention effects.
The local scorer reports prediction error and the loss from the interventions those predictions select.
Change the example or supply your own predictions in the task's CSV format, then score again.
Keep the data, actions, and decision rule fixed. Did your change improve prediction, action, or both?
Tangible outcomes
Did this observer beat no action and the relevant existing methods on downstream loss?
Where does it sit among observers with the same information, budget, controller, and target?
Which targets, prompt families, action budgets, or operating conditions make it fail?
Save your predictions and score output with the task version and settings so another researcher can repeat the comparison.
New CPU practice tasks
Choose which trusted facts to buy, then compare missed harm, clean work retained, and measurement cost. No model download is needed.
These are open, scripted practice tasks, not private-test rankings or live-agent safety tests. The cross-model comparison is a workbench diagnostic, not a paper finding.
Submission and release status
The open tasks above score locally. Submitting to a task with private evaluator targets uses the public checks and separate scoring service described here.
This service is not active at public launch. The issue comment will state if it is enabled later. A preflight pass alone is not a score or rank.
Ranking rule
A score is task-relative. ObserverBench can return an official rank only when the task version, observation boundary, action budget, controller, and loss match. Cross-track results remain visible as comparisons, and evaluator oracles remain unranked reference bounds.