amduat/docs/conformance-harness-spec.md

# Conformance Harness Specification

Status: Draft
Scope: Define the Phase 0 conformance harness goals, backend matrix, fixture format, and pass/fail gates for ASL and TGK store surfaces.

---

# 1. Purpose

The harness ensures that all backend adapters implement existing store semantics without drift.

It MUST:
1. Run the same test vectors across multiple backends.
2. Enforce deterministic ordering rules.
3. Validate replay equivalence.
4. Provide a single command entry point for CI.

---

# 2. Targets and Scope

## 2.1 Surfaces Under Test

1. ASL/1-STORE
2. TGK/STORE/1

Optional extensions:
1. ASL/STORE-INDEX/1 (index/log store)
2. TGK/PROV/1 (if provenance surface is exposed)

## 2.2 Conformance Mode

All tests are:
1. Deterministic.
2. Backend-agnostic.
3. Semantics-first (no timing assumptions).

---

# 3. Backend Matrix

The harness MUST be able to run each test suite against:

## 3.1 ASL Backends

1. `asl_store_fs`
2. `asl_store_index_fs`
3. Future: `asl_store_minio`

## 3.2 TGK Backends

1. `tgk_store_mem`
2. `tgk_store_fs`
3. `tgk_store_asl_index_fs`
4. Future: `tgk_store_janus`

---

# 4. Canonical Fixture Format

Fixtures are the source of truth for all conformance tests.

## 4.1 Requirements

1. Versioned fixture schema.
2. Stable ordering of artifacts and edges.
3. Explicit references encoded as canonical `ReferenceBytes` hex.
4. Deterministic content generation (no randomness at runtime).

## 4.2 Minimal Fields

For ASL fixtures:
1. `artifact_bytes` (hex or base64, fixed encoding).
2. `type_tag` (optional).
3. `expected_reference` (canonical Reference hex).

For TGK fixtures:
1. `edge_artifact_reference` (canonical Reference hex).
2. `edge_body` (decoded representation).
3. `edge_type_id`
4. `from_refs` and `to_refs` (ordered lists, canonical hex).

---

# 5. Test Suite Requirements

## 5.1 ASL/1-STORE Suite

1. Put/get round-trip.
2. Idempotent `put`.
3. `get` on missing ref yields `ERR_NOT_FOUND`.
4. Collision on same ref yields `ERR_INTEGRITY`.
5. Unsupported config yields `ERR_UNSUPPORTED`.
6. Optional I/O fault path yields `ERR_IO`.

## 5.2 TGK/STORE/1 Suite

1. `resolve_edge` for valid edge returns `EdgeBody`.
2. `resolve_edge` for non-edge returns `GS_ERR_NOT_EDGE`.
3. `resolve_edge` for artifact integrity failure yields `GS_ERR_ARTIFACT_ERROR`.
4. `edges_from`, `edges_to`, `edges_incident` return identical lists across backends.
5. Deterministic ordering per `TGK/STORE/1` canonical `(hash_id, digest)` rule.
6. Optional `neighbors` returns canonical node order.
7. Optional `scan_edges` enumerates all edges exactly once.

## 5.3 Replay Equivalence

1. Build a fixture ASL log/index state.
2. Replay into a fresh store.
3. Assert identical references and TGK results.

---

# 6. Ordering and Equivalence Rules

## 6.1 Edge Ordering

Edge lists MUST be sorted in ascending lexicographic order of:
`hash_id_bytes || digest_bytes`, with `hash_id` encoded as big-endian `u16`.

## 6.2 Node Ordering

Neighbor lists MUST be sorted in ascending lexicographic order of:
`hash_id_bytes || digest_bytes`, with `hash_id` encoded as big-endian `u16`.

## 6.3 Cross-Backend Equivalence

For a fixed snapshot and `GraphStoreConfig`, all backends MUST return:
1. Identical edge sets.
2. Identical ordered lists for adjacency queries.
3. Identical results for `resolve_edge`.

---

# 7. Harness Structure

Suggested structure:
1. `tests/conformance/asl_store/*.c`
2. `tests/conformance/tgk_store/*.c`
3. `tests/fixtures/*.json`
4. `tools/amduat_conformance_run`

The harness runner MUST:
1. Accept backend selector flags.
2. Accept fixture set selection.
3. Emit a machine-readable summary.

---

# 8. CI Gate

An adapter cannot be merged unless:
1. It passes its conformance suite.
2. Cross-backend equivalence is demonstrated.
3. Replay equivalence passes on at least one backend pair.

---

# 9. Out of Scope

1. Performance benchmarking.
2. Load testing.
3. Backend-specific tuning.

Those are separate gates after conformance is proven.