Mean task quality
Published grade, on a 0–1 scale
Click a model to inspect
THE MODEL OBSERVATORY VOL. 01 / OCT 2026
Better intelligence.
Measured.
One model, two working together, or memory layered on top. We test what makes the answer better—and what it costs to get there.
Published data snapshotLoading source…
—current native reports
—run summaries in the archive
—Savant on/off comparisons
01 / THE CURRENT FIELD
Capability has a cost.
Compare the published measurements within a run. Class champions below come from separate brackets, not a head-to-head 3B versus 8B trial.
Quality × latency
Higher and further left is preferable
Select a point for its measurement.
View exact comparison data
02 / THE FUSION QUESTION
More layers. Better answers?
A native model and a depth-expanded candidate, kept in their own smoke-test population.
Final profile composite
Scrub execution · reported final-round scores
Inspect fusion source rows
LAB NOTE / F-001
A fast empty answer
is still an empty answer.
A profile score includes a speed component. Positive composite scores alone cannot establish that a fused model works.
Available in this exportSplice40 final composites
Missing from this exportFused raw grades, latency and traces
First-generation fused artifacts failed twice over: a two-byte misaligned read corrupted every copied weight, and the sealed rows were transport errors under accelerator contention — not graded output. An identity-rebuild control isolated the writer bug; rebuilt candidates pass a complete per-tensor payload audit (verified by an independent auditor) and generate real, if degraded, output. Stacked52 has no measurement in this snapshot; the corrected full-card bout is pending.
Read the evidence limits03 / COLLECTIVE INFERENCE
Two runners. One useful answer.
The test is whether their combined work improves the answer under an explicit inference budget.
A+A
Same model, twice
Independent seeded proposals from the strongest small model, followed by a practical selector or synthesis step.
A+B
Different models
Two complementary proposals, with the same selection or synthesis rule and total budget as the control.
A→B
Draft, then refine
One model drafts; the second checks and revises. Test both orders and a same-model refinement control.
↗/B
Route by task
A frozen routing rule chooses a model before generation. Evaluate on tasks not used to choose that rule.
Next experiment: a selective cascade.
A small model answers first. A frozen rule may request refinement when a practical check fails. The rule cannot see the held-out answer or its grade. Measure every call and end-to-end latency; no two-model configuration has a fixed 2× cost.
Oracle selection is an analysis tool.
Choosing the better answer with its true grade is an upper bound for selecting those candidates. A deployable selector must work without the answer key. Synthesis can produce a new answer and is a different experiment.
04 / THE MEMORY LAYER
What does Savant add?
The same engine, the same task, with and without the memory and orchestration layer.
Measure the engine.
Then measure the system.
Useful context may help a smaller model answer a continuity task better. That is a system gain. It does not establish that the weights have acquired new reasoning ability.
The first pass freezes the retrieved context across engines. A second pass evaluates Savant end to end, including retrieval choices, latency and incorrect memories.
Download experiment protocolEngine / methodSavant offSavant onPaired gain
Native 3BAwaitingAwaiting—
Native 8BAwaitingAwaiting—
A + A / practicalAwaitingAwaiting—
A + B / practicalAwaitingAwaiting—
Refiner / routerAwaitingAwaiting—
Fused candidateQuality gateQuality gate—
The current battery export has no Savant condition labels. Its single-model measurements cannot be reused as the “off” side of a new paired study.
05 / THE EXPERIMENT RECORD
History stays in the picture.
Completed reports, partial attempts and invalidated runs remain inspectable.
Version comparison
Final scrub composites · common 0–1 scale
v1 · invalidatedv2 · report
V1 was invalidated after an endpoint fault. These separate runs are retained for audit; their differences are not a valid improvement estimate.
Run archive
Run dates come from filenames. The export timestamp does not establish when each experiment executed.
06 / RESEARCH WE BUILD ON
Evidence beyond our own bench.
Published studies inform the experiment design. These are external results, not Titan or Savant measurements.
SELF-MOA / 2025Table 1
Diversity has to beat proposal quality.
AlpacaEval 2.0 length-controlled win rate (%)
Ensembles: six proposals, one Qwen1.5-110B-Chat aggregator, two layers. Same-model proposals use WizardLM-2-8x22B. These large-model results do not predict Titan’s small-model scores.
Read the paperLONGMEMEVAL / 2024Table 3
Better memory design can help the same model.
LongMemEval-M · end-to-end QA accuracy (%)
Original keyKey + user facts
Llama 3.1 8B53.4→57.2
Llama 3.1 70B62.4→68.2
GPT-4o67.0→72.0
Round-level memory, top-10 retrieval. Only the indexing key changes from the original value to value + extracted user facts. This is not a Savant on/off experiment.
Read the paper07 / METHODS & PROVENANCE
Know what the number means.
Quality, speed and composite scores
Mean task quality is the exporter’s mean_grade, on a 0–1 rubric scale. Mean latency is mean_lat_ms, displayed in seconds. A final composite comes from final_scores and uses a different population.
Reported scoring weights: scrub = 80% quality + 20% normalized speed; sling = 40% quality + 60% normalized speed. The normalized speed function is not in this export, so this site never reconstructs composites from mean latency.
The worker confirms identical generation settings but separate executions for the two profiles. Switching profiles therefore changes the measured latency population.
Counts, rounds and unresolved differences
The source’s n counts are shown as aggregate counts, not unique tasks or total inference calls. Feather-v2’s exported counts sum to 352; the handoff describes 400 executions. Export inclusion rules are unavailable.
The published feather final report names Gemma 3 4B and Ministral 3B. The handoff names Qwen 1.7B and Ministral 3B as replayed finalists. Charts retain the published report membership without assuming the descriptions cover the same rounds.
Final composites have no sample size in this export. Missing grades, counts and latency stay missing.
What is verified here
The source is a publicly served aggregate export retrieved over HTTPS. Its bytes have a recorded SHA-256. This verifies which snapshot the charts use; it does not independently verify the underlying ledger chain, raw model output or model artifact.
A published report is labeled “report available,” not “independently sealed.” Partial and invalidated runs are excluded from default current comparisons.
Canonical public feed ↗ · Provenance record ↓
What would establish a Savant gain
Freeze a held-out task set, user histories and model versions. Run each eligible engine with Savant off and on. Average seeded repetitions within each task before paired task-level analysis.
Report quality, continuity, false recall, unsupported claims, task completion, all inference calls, total tokens, latency percentiles, loading time and peak memory. A positive composite cannot waive a failed output-quality gate.
Routing learned from the current task card must be tested on an untouched card. Practical selection stays separate from oracle analysis.