# Research protocol: native models, dual inference, and Savant

**Status: proposed experiments, not measured results.** This protocol separates model capability, inference coordination, and Savant's memory/context contribution. It is portable across inference backends and does not prescribe deployment changes.

## 1. Working forecast and evidence boundary

The strongest validated native model is the default comparator. The reported Ministral 3B and 8B bracket wins justify keeping both; separate bracket wins do not establish a direct ranking. A duo may improve selected tasks when its candidates contribute useful alternatives and its practical selector or refiner can use them. Different models are not automatically better than independent samples from one strong model: published MoA and Self-MoA results motivate testing both [2, 3].

**Forecast:** Savant should help most when correct answers depend on prior preferences, decisions, updated facts, or evidence across sessions. On a fully specified new problem, native reasoning and instruction-following remain central. Savant is a separable memory/retrieval layer; an improvement with added context is a system improvement, not evidence of changed model weights. LongMemEval supplies relevant memory task categories and distinguishes indexing, retrieval, and reading [5].

The expanded candidates' reported empty/EOS behavior remains a failed-generation observation with an unresolved cause. Public aggregates do not independently establish their artifact integrity. SOLAR recovered a depth expansion through continued pretraining; its success does not establish that an untrained expansion should improve immediately [1].

## 2. Cross every eligible engine with Savant off/on

Use **A = native Ministral 3B** and **B = native Ministral 8B** throughout this experiment; retain exact model, quantization, tokenizer, template, and runtime digests. A+A means independent samples, not duplicated deterministic output. Selectors and routing policies are frozen before evaluation.

| Savant | A solo | B solo | A+A practical best-of | A+B practical best-of | A → B refinement | B → A refinement | One-call router | Oracle pair selection | Fused candidates |
|---|---|---|---|---|---|---|---|---|---|
| Off | Run | Run | Run | Run | Run | Run | Run | Offline only | Gate first |
| On | Run | Run | Run | Run | Run | Run | Run | Offline only | Gate first |

- **Practical best-of:** generate two candidates, then use a fixed deployable selector without ground truth. Preserve both answers, selection, and decision traces. If selection calls a model, count that invocation. Add B+B as the compute control when B is the stronger development-set baseline; this prevents a weak A+A control from overstating diversity gains.
- **Refinement:** the second model receives the original question and first draft; it may retain or revise it. Record whether correct answers become wrong and wrong answers become correct. Evaluate both orders separately.
- **One-call router:** choose A or B from the question and permitted context before either generates. Exactly one answer-model invocation follows the routing decision; count router and retrieval overhead separately.
- **Oracle:** retrospectively choose the better answer from the exact saved pair using ground truth. This is an offline ceiling for selection from that pair, not deployable performance or a ceiling for refinement/synthesis that generates a new answer.

## 3. Frozen evaluation and memory controls

Build two held-out strata: **self-contained capability tasks**, where the prompt contains all required evidence, and **continuity tasks**, where answers require user history. Cover recall, preference application, multi-session reasoning, temporal reasoning, knowledge updates, and appropriate abstention. Include irrelevant memories, superseded facts, and unanswerable questions. Use synthetic or consented, de-identified histories; publish no personal history.

Freeze task IDs, rubrics, answers, history snapshots, time cutoffs, development/test split, prompt templates, and seed lists. Tune selectors, routing thresholds, retrieval budgets, and prompts on development data only. Hold out entire users or history groups, not merely questions from the same conversations. No future turns, test answers, evaluator feedback, or outputs from another arm may enter memory.

For the first off/on comparison, retrieve once per task from a fixed Savant snapshot and give every engine the same permitted evidence packet, preserving source IDs and timestamps. Off receives the current question without cross-session memory. Restart each arm from an isolated copy of the same initial history state; fork trajectories for multi-turn tests. Assert that every retrieved item belongs to the current test user. Record rejected cross-user or future-dated items as binding failures, not useful retrieval.

Then run a separately labeled end-to-end study allowing each engine its own retrieval queries. For a diagnostic subset, provide gold evidence to isolate retrieval misses from reading failures. This control must not be merged with deployable Savant scores.

## 4. Measure quality and actual resource use

Freeze a raw-grade rubric before testing, using executable checks or exact answers where suitable and blinded review for open responses. Count empty/EOS, malformed, and timed-out answers as failures with explicit reasons. Report raw grade, valid-answer rate, memory-category correctness, and unsupported personal claims before any composite.

Historical profiles are **sling: 40% quality / 60% speed** and **scrub: 80% quality / 20% speed**. They use separate executions even when generation settings match. Preserve profile-specific execution IDs and timings; do not pool them or infer identical responses from equal aggregate grades. For new composites, freeze and publish the speed-normalization function, reference population, weights, and denominator. Do not reconstruct historical composites from an undocumented formula or equate a positive composite with successful generation.

Record the following for every execution and stage:

| Dimension | Required measurement |
|---|---|
| Quality | Task grade, grader version, correctness/failure reason, memory sources used |
| Inference budget | All candidate, refinement, selector, and router calls; full input/output token counts, including repeated history, retrieved context, and visible intermediate generations |
| Timing | End-to-end latency; retrieval, routing, selection, prefill, and decoding where exposed; P50/P95 with sample counts |
| Loading | Model load/switch time and cold-start latency separately from warm/resident execution; cache state and concurrency |
| Memory | Peak total resident RAM over the complete request, measurement method, sampling interval, and accelerator memory if applicable |
| Other work | Indexing, embedding, extraction/update costs, and their amortization assumptions; observed energy or monetary cost only when measured or explicitly priced |

Compare both fixed configurations and matched inference budgets. A+B, refinement, or selection may have different stage counts and residency costs; **do not assume every cost doubles**. Two runner processes do not imply exactly two model calls. Do not derive throughput from inverse mean latency or label historical latency as end-to-end without timing-scope evidence.

## 5. Paired analysis and decision rules

Choose the held-out sample size and a minimum useful gain using a development pilot before unsealing results. Use at least three predeclared replicate seeds initially, with independent candidate sub-seeds. Average replicates **within task × configuration × Savant condition × profile first**. Compute paired task differences from those means; replicates are not independent tasks. For multiple questions per history/user, use paired bootstrap intervals clustered by history/user. Keep profile analyses separate and report missing executions and exclusions explicitly.

For engine e, Savant uplift is **mean paired grade(on − off)**. Compare that uplift between duo and solo to estimate the interaction. Separately report oracle-minus-best-solo headroom and oracle-minus-practical-selector loss. Oracle improvement alone is not evidence of a deployable duo advantage.

Promote a configuration only when it clears the predeclared useful-gain or noninferiority criterion at the intended resource budget, without unacceptable valid-answer, memory-binding, or latency regressions. Publish category results and quality–latency/memory trade-offs; distinguish inconclusive comparisons from equivalence. Existing aggregate exports cannot supply paired confidence intervals, significance, or these new ablations.

## 6. Physical validity gate for expanded models

Before repeated fused evaluations, require an independent audit of GGUF structure, tensor names/shapes/counts, offsets/alignment, data ranges, expected layer mapping, quantization metadata, tokenizer/special tokens, and chat template. Validate a native identity round-trip through the same writer and compare it with the source under identical prompts and settings, using appropriate numerical tolerance. Preserve file digests and validator output.

Only an artifact passing integrity checks proceeds to a small bounded generation smoke with verified native controls. A timeout, repeated premature EOS, invalid values, or load failure stops the repeated capability battery until the cause is understood. Preserve failures; do not substitute missing raw scores with zero or discard them from denominators. An integrity pass is not a capability pass, and a failed construction is not proof that every fusion method fails.

## 7. Public snapshot and research references

The captured public export is [data.js](https://research.titandigitalengineering.com/data.js), linked from the [research site](https://research.titandigitalengineering.com/). Its `generated_utc` is `2026-10-17T19:43:26.855326Z`; this is an export marker, not a native run/seal timestamp. SHA-256 of the captured raw JavaScript:

`48dec0121ff3a7c8980ae62e70bfb4beda8d07708c0d3c01207f0d02e4b11eff`

This verifies the identity of the captured public bytes, not raw-ledger correctness. Keep source file/model/profile keys, published mean grades, source counts, and profile composites distinct. Missing duo, Savant, and held-out results remain unmeasured. Preserve tombstoned v1 material only as invalidated history, outside valid comparisons.

1. [SOLAR 10.7B — depth up-scaling and continued pretraining](https://arxiv.org/html/2312.15166v3), 2023/2024.
2. [Mixture-of-Agents Enhances Large Language Model Capabilities](https://arxiv.org/html/2406.04692), 2024.
3. [Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial?](https://arxiv.org/pdf/2502.00674), 2025.
4. [RouteLLM — routing quality/cost evaluation](https://arxiv.org/html/2406.18665v3), 2024.
5. [LongMemEval — long-term interactive memory evaluation](https://arxiv.org/html/2410.10813), 2024/2025.

Published research motivates this design; its numerical gains are not forecasts or measurements of the local models or Savant.