Local benchmark replication

Auwai does not publish or depend on a benchmark endpoint. Every benchmark is run against a local model server or a deterministic local mock. The release repository contains no public benchmark hostname and no runnable command that defaults to one.

Harness flow

The runner and endpoint are explicit. The mock server exercises transport and parsing only; it is never a model-quality result.

Prerequisites

Python 3.10 or newer.
SQLite with FTS5.
Git.
For model-backed runs, a local server that exposes the protocol shape used by the harness you want to test.
A checkout of the benchmark data repository:

export AUWAI_BENCHMARK_DATA=/path/to/auwai-benchmark-data

The data checkout supplies the prompt, rubrics, fixture repositories, and the runner scripts. It is separate from the Auwai release so the wheel stays small.

Protocol shapes

A local server must answer one or more of:

Protocol	Path used by the runner
Anthropic Messages	`/v1/messages`
OpenAI Chat Completions	`/v1/chat/completions`
OpenAI Responses	`/v1/responses`

The runner sets AUWAI_BENCH_URL as the root URL, for example http://127.0.0.1:8000. It appends /v1 for the provider configuration.

Environment

Variable	Required	Meaning
`AUWAI_BENCH_URL`	yes	Local model root URL, no public default
`AUWAI_BENCH_MODEL`	yes	Model id served by the endpoint
`AUWAI_BENCHMARK_DATA`	yes	Path to the benchmark data checkout
`AUWAI_BENCH_API_KEY`	no	Only for a local server that checks a key

The v6 runner refuses to start without AUWAI_BENCH_URL or --endpoint:

python3 "$AUWAI_BENCHMARK_DATA/run_case_study_v6.py" --help

Smoke test with the mock server

The mock server is a deterministic fixture for CI and parser tests. It is not a model-quality benchmark.

./benchmarks/local/run_mock.sh --answer "OK"
# prints: http://127.0.0.1:<port>

In another shell:

export AUWAI_BENCH_URL=http://127.0.0.1:<port>
export AUWAI_BENCH_MODEL=mock
export AUWAI_BENCHMARK_DATA=/path/to/auwai-benchmark-data
./benchmarks/local/run_v6.sh --harness hermes --arms auwai --repos large \
  --contexts 64k --reps 1

This proves transport, prompt packing, parsing, and scoring. It does not prove answer quality.

v6 matrix

The v6 matrix is the release regression gate. It covers four harnesses, two repository sizes, two context windows, two arms, and repeated cells.

export AUWAI_BENCH_URL=http://127.0.0.1:8000
export AUWAI_BENCH_MODEL=<model-id>
export AUWAI_BENCHMARK_DATA=/path/to/auwai-benchmark-data
./benchmarks/local/run_v6.sh --reps 1 --parallel 1

Use --harness, --arms, --repos, and --contexts to narrow a run:

./benchmarks/local/run_v6.sh --harness hermes,opencode \
  --arms auwai,vanilla --repos larger --contexts 64k,cloud

The runner writes a case-study directory under the data checkout by default. Pass --out to choose another location.

Workflow probes

The workflow suite measures service and ledger behavior without a model answer:

./benchmarks/local/run_workflows.sh --agents 1,2,4,8,16,32 \
  --per-agent 50 --batch 10

It runs the deterministic workflow benchmark and the full-read concurrency probe. The script still requires AUWAI_BENCH_URL so a run cannot silently inherit a public default; the probes themselves do not call it.

Scoring and gates

After a v6 run:

python3 "$AUWAI_BENCHMARK_DATA/summarize_v6.py" --case-dir <case-dir>
python3 "$AUWAI_BENCHMARK_DATA/rubric_audit.py" --case-dir <case-dir>

The release gates are:

every cell parses;
all 48 Auwai cells reach 64/64;
coverage is 1.00;
the matrix reports 96/96 cells;
the deterministic mock path passes in CI without network.

Do not present mock results as model quality. Do not weaken a gate to make a run pass; record the failure and the smallest reproducing command.

Expected cost

On the reference workstation, a full v6 matrix with one repetition takes several hours because it runs both arms across four harnesses. The workflow probes take minutes. The mock smoke test takes seconds. Model throughput, context window, and provider queueing dominate the wall time.

Troubleshooting

Symptom	Check
`AUWAI_BENCH_URL is required`	Export the variable or pass `--endpoint`
Model not found	Match `AUWAI_BENCH_MODEL` to the server's model id
404 on a protocol path	The server does not expose that shape; narrow `--harness`
Context limit errors	Lower the context cell or raise the server window
Queueing or timeouts	Raise `--timeout`, lower `--parallel`
Empty or unparseable answers	Run the mock first; then inspect the raw harness log
Coverage below 1.00	Re-run the cell; do not edit the rubric

Historical results

Older case studies in the benchmark data checkout were produced against a hosted proxy that is no longer available. They are retained as historical evidence and are not runnable instructions. The release result is the local v6 run and the workflow probes recorded in results.