Auwai

Knowledge service

The default Auwai install is one SQLite ledger per repository. No server, queue, or database credentials are required.

The optional knowledge service shares facts and findings across a team.

Architecture

The repository ledger remains local and authoritative for reads. The service is the optional shared layer for facts, findings, claims, plans, and jobs.

Sync flow

Local-only

auwai index --repo .
auwai facts build --repo .
auwai facts search auth --repo .

Local facts live in .auwai/auwai.sqlite. Nothing leaves the machine.

Team mode

On the server:

export AUWAI_KNOWLEDGE_TOKEN='...'
auwai serve \
  --host 0.0.0.0 \
  --port 8765 \
  --database sqlite:///var/lib/auwai/knowledge.sqlite

For PostgreSQL:

pip install 'psycopg[binary]'
auwai serve --database postgresql://auwai:...@db.internal/auwai

For MySQL 8.4 or MariaDB 11.4:

pip install PyMySQL
auwai serve --database mysql://auwai:...@db.internal/auwai
auwai serve --database mariadb://auwai:...@db.internal/auwai

The database URL selects the storage adapter. MySQL/MariaDB use bounded VARCHAR keys, utf8mb4, and hashed primary keys for coverage rows so InnoDB stays within its 3072-byte index limit. The service prints database URLs with the password redacted. Every adapter keeps one connection and runs the HTTP server single-threaded, so statements cannot interleave on that connection. Scale the queue with additional service processes or a shared database rather than threads.

Schema migrations

The service records migrations in schema_migrations(version, name, applied_at, checksum). The current release is schema version 3:

Version Name Change
1 initial Plans, facts, findings, reads, jobs, sync events
2 claims Answer claims and verification state
3 fact_identity fact_key, revision, first/last seen, unique identity

Startup applies pending migrations and refuses a database newer than the release. The migration runner is idempotent and checks applied checksums:

auwai serve --database "$AUWAI_DATABASE" --check
auwai serve --database "$AUWAI_DATABASE" --migrate

--check reports the current version and pending versions without writing. --migrate waits for the database migration lock and applies pending steps. PostgreSQL uses an advisory lock; MySQL/MariaDB use GET_LOCK; SQLite uses BEGIN IMMEDIATE. A failed SQLite or PostgreSQL step rolls back to the prior version. MySQL DDL can implicitly commit, so take a backup before a migration and re-run --check after it.

A local repository ledger carries its own schema_version meta key and refuses a database that belongs to another root or a newer ledger. Facts, findings, reads, plans, and segments from an older ledger are preserved.

The adapter contract and migration tests run against POSTGRES_URL, MYSQL_URL, and MARIADB_URL when those variables are set and skip otherwise. CI runs PostgreSQL 16, MySQL 8.4, and MariaDB 11.4 service containers. Release validation also covered MySQL 8.4.13 and MariaDB 11.4.13 in Podman containers with a 512 MiB memory limit:

MYSQL_URL='mysql://auwai:...@127.0.0.1:3306/auwai' \
  python3 -m unittest tests.test_knowledge_service -v

On each developer machine:

[knowledge]
mode = "team"
url = "https://auwai.internal.example"
repo_id = "github.com/acme/telemetry"
token_env = "AUWAI_KNOWLEDGE_TOKEN"

Then:

auwai facts build --repo .
auwai sync --repo . --status
auwai sync --repo .

Facts, findings, and verification claims are queued in a local outbox and drained in batches. The service validates, deduplicates by event id, and merges into its database. Facts carry a stable fact_key and a revision; re-publishing a chunk replaces its old facts instead of appending duplicate rows. Agents never receive database credentials. In team mode, run auwai sync after a read batch and before the final answer so coverage and findings reach the shared store. A second agent can pull shared facts into its local cache:

auwai sync --repo . --pull
auwai facts search auth --repo .

Pulled facts are pointers, not evidence. Read the exact range with auwai read before citing it.

Multiple agents

Use one git worktree, one .auwai ledger, and one session per agent. A shared checkout serializes SQLite writes and can lose reads when several agents use the default session file. Named sessions keep agent state separate:

auwai begin "auth review" --repo . --session agent-a
auwai read <chunk-id> --repo . --session agent-a
auwai recap --repo . --session agent-a

AUWAI_SESSION_ID sets the default session, and AUWAI_AGENT_ID labels reads, findings, and claims with an author. Named sessions live under .auwai/sessions/.

Token scopes, repositories, and redaction

A single token is enough for a small team. For stronger isolation, pass an auth file:

auwai serve --database sqlite:///var/lib/auwai/knowledge.sqlite \
  --auth-file /etc/auwai/auth.json

{
  "tokens": {
    "token-a": {"principal": "alice", "repos": ["github.com/acme/telemetry"],
                "scopes": ["read", "write", "jobs"]},
    "token-b": {"principal": "bob", "repos": ["github.com/acme/telemetry"],
                "scopes": ["read", "jobs"]}
  }
}

repos and scopes accept ["*"]. The service checks the repository in the request body or query string and rejects cross-repository access with 403. Fact and finding text is redacted by default: AWS access keys, password=, secret=, api_key=, token=, sk-..., and private keys are replaced with [REDACTED] before storage. Redaction is not a substitute for keeping secrets out of source code, but it is on by default in team mode.

Shared plans and remote full-read

A plan id is derived from the repository, snapshot, pack, context window, and segment chunk lists, so two agents that plan the same repository get the same id. Publish the plan and create one job per read segment:

auwai fullread plan --repo . --context-window 128000 --share --jobs

Workers claim a job, read its segment locally, record a short summary, and complete the job:

auwai fullread claim --remote --repo . --worker agent-1 --json
auwai fullread read --remote --repo . --job JOB_ID --max-bytes 0
auwai fullread done --remote --repo . --job JOB_ID --summary "what the segment contains"

done writes the summary as a finding, drains the local outbox, and marks the job complete. The service stores read coverage per chunk and reports read_chunks, complete_chunks, covered_chunks, and semantic_coverage. semantic_coverage is the ratio of chunks that have both a fact and a read record. Facts and reads are joined by repository and content-addressed chunk id, not by exact snapshot, so a fact seeded from a builder checkout still covers the same chunk when a worker reads it from a different checkout.

The full-read read path is also available without the service. The service path only changes who owns the lease and where coverage is merged.

Service API

GET  /health
POST /v1/sync
GET  /v1/facts?repo_id=&snapshot=&chunk_id=&kind=&query=&limit=&include_stale=
GET  /v1/findings?repo_id=&snapshot=&chunk_id=&author=&query=&limit=
GET  /v1/claims?repo_id=&snapshot=&answer_id=&chunk_id=&status=&limit=
GET  /v1/coverage?repo_id=&snapshot=
GET  /v1/jobs?status=
POST /v1/jobs
POST /v1/jobs/claim
POST /v1/jobs/{job_id}/complete

The service requires Authorization: Bearer <token> unless started with --allow-anonymous.

Job queue

The service includes a database-backed job queue with worker leases:

curl -X POST https://auwai.internal.example/v1/jobs \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"repo_id":"github.com/acme/telemetry","kind":"fullread","payload":{"segment":"read-1"}}'

A worker claims a job with POST /v1/jobs/claim and completes it with POST /v1/jobs/{job_id}/complete. The lease expires automatically.

Queue backends

The database queue is the default. Select another adapter in config.toml:

[knowledge]
queue = "redis"          # db, redis, rabbitmq, or kafka
queue_url = "redis://127.0.0.1:6379/0"
queue_name = "auwai"    # stream, queue, and topic prefix

The service also accepts --queue and --queue-url. Environment variables override the config file: AUWAI_QUEUE, AUWAI_QUEUE_URL, REDIS_URL, RABBITMQ_URL, and KAFKA_BOOTSTRAP_SERVERS (or KAFKA_URL). Install the optional client for the selected backend:

pip install redis         # Redis Streams
pip install pika          # RabbitMQ
pip install kafka-python  # Kafka

The HTTP API and worker behavior are the same for every adapter. The claim is atomic, so concurrent workers cannot claim the same job twice; an expired lease is claimable again.

Backend Job state Delivery Lease recovery
db service database jobs table scan expired lease_until is selected
redis Redis hashes and sorted set one stream per repo, consumer group XPENDING + XCLAIM
rabbitmq service database durable repo and all queues database scan
kafka service database keyed job-id topic, consumer group database scan

RabbitMQ and Kafka use the service database as the canonical lease and state store, and the broker as the dispatch channel. That keeps claim, complete, list, and lease expiry identical across adapters even if a broker redelivers a message. Redis owns all job state itself.

Retry and dead letters

There is no automatic retry. A completed job is terminal (done or failed) and stays visible through GET /v1/jobs. To retry work, create a new job or reset it in the database. Redis acknowledges the stream entry on completion. RabbitMQ acknowledges a dispatch message once the database claim succeeds. Kafka commits the consumer offset after processing; a replayed record cannot claim an already-active job because the database update is conditional. None of the adapters writes to a dead-letter queue; failed jobs are kept as records for inspection instead.

Testing external adapters

The database conformance tests always run. Container-backed tests start only when their URL is set and skip cleanly otherwise:

REDIS_URL=redis://127.0.0.1:6379/0 make test
RABBITMQ_URL=amqp://user:pass@127.0.0.1:5672/%2F make test
KAFKA_BOOTSTRAP_SERVERS=127.0.0.1:9092 make test

Release validation used Redis 7.4.12 (redis:7-alpine, 256 MiB), RabbitMQ 4.4.0 (rabbitmq:4-management, 512 MiB), and Kafka 3.9.0 (apache/kafka:3.9.0, KRaft, 768 MiB) in Podman. Each adapter passed its conformance suite and its service API integration test with a real container. RabbitMQ and Kafka keep the database as the canonical lease store, so their broker-side clients do not need to be thread-safe for claim correctness.

See security for the threat model, TLS deployment, token scopes, and the release checklist.

Health

GET /health returns the service name, release version, schema version, storage backend, and queue name. It requires the same bearer token as the other routes unless the service is in explicit anonymous loopback mode.

{
  "ok": true,
  "service": "auwai-knowledge",
  "version": "1.0.0",
  "schema_version": 3,
  "database": "sqlite",
  "queue": "db"
}

Security

Known limits