Methodology
The benchmark asks whether Auwai is faster than a control on the same task. Every run is local and names its endpoint explicitly. The release repository does not ship a hosted benchmark service.
Method
- One prompt per scenario, byte-identical for every arm.
- The prompt does not mention Auwai or a read budget.
- The environment is the only difference:
- Hermes with the Auwai skill and plugin;
- Hermes vanilla without the skill, plugin, or index;
- OpenCode without Auwai.
- The cloud cell used a native 1,000,000 context.
- The Auwai per-turn read cap was 65,536.
- Local Hermes used the local model and the configured context.
- Local runs used
AUWAI_BENCH_URLandAUWAI_BENCH_MODEL; see local replication. - Controls ran with
.auwaimoved aside. - The gate was Auwai within 10 percent of the faster control on both tasks.
Answer quality
The benchmark also needs a score for the answers, not only a time.
The quality run uses a fixed rubric and the same local model. The 2026-11-09
run added a vanilla control arm with the repository copied without .auwai
and a full-window segment validation at 64K. The control has source access but
no Auwai index or skill.
benchmarks/quality/auwai-architecture.json is the first rubric.
It asks eight questions about Auwai's own coverage path.
Each question has three content criteria and one citation criterion, for four
points, and the total is 32.
scripts/quality_check.py score matches each answer and its cited paths
against explicit patterns. It is deterministic and offline.
scripts/quality_check.py run calls one local model, captures the answers,
and scores them. It writes the rubric window to a temporary config.toml
for the model run.
A citation point requires the named source files, not only a prose answer. The scorer accepts a paraphrase that names the same mechanism. It is a check on correctness and citation support, not a human review.
The runs are in results.
The rubric and its answers are in benchmarks/quality/.
Measurements
- wall time, API calls, fresh input, cache read, output, reasoning;
- prompt hash;
- first Auwai command;
- full-read estimate and actual reads;
- findings, decisions, and cited paths;
- what remained unread or unverified.
What the quality run does not measure
- Answer correctness on a full-window segment.
- More than one repository or model.
- A control arm on the same question set.
- Meander inside a long context.
- The new SQLite ledger under a full read.