amduat/docs/conformance-harness-spec.md

Conformance Harness Specification

Status: Draft Scope: Define the Phase 0 conformance harness goals, backend matrix, fixture format, and pass/fail gates for ASL and TGK store surfaces.

---

1. Purpose

The harness ensures that all backend adapters implement existing store semantics without drift.

It MUST:

1. Run the same test vectors across multiple backends.
2. Enforce deterministic ordering rules.
3. Validate replay equivalence.
4. Provide a single command entry point for CI.

---

2. Targets and Scope

2.1 Surfaces Under Test

1. ASL/1-STORE
2. TGK/STORE/1

Optional extensions:

1. ASL/STORE-INDEX/1 (index/log store)
2. TGK/PROV/1 (if provenance surface is exposed)

2.2 Conformance Mode

All tests are:

1. Deterministic.
2. Backend-agnostic.
3. Semantics-first (no timing assumptions).

---

3. Backend Matrix

The harness MUST be able to run each test suite against:

3.1 ASL Backends

1. asl_store_fs
2. asl_store_index_fs
3. Future: asl_store_minio

3.2 TGK Backends

1. tgk_store_mem
2. tgk_store_fs
3. tgk_store_asl_index_fs
4. Future: tgk_store_janus

---

4. Canonical Fixture Format

Fixtures are the source of truth for all conformance tests.

4.1 Requirements

1. Versioned fixture schema.
2. Stable ordering of artifacts and edges.
3. Explicit references encoded as canonical ReferenceBytes hex.
4. Deterministic content generation (no randomness at runtime).

4.2 Minimal Fields

For ASL fixtures:

1. artifact_bytes (hex or base64, fixed encoding).
2. type_tag (optional).
3. expected_reference (canonical Reference hex).

For TGK fixtures:

1. edge_artifact_reference (canonical Reference hex).
2. edge_body (decoded representation).
3. edge_type_id
4. from_refs and to_refs (ordered lists, canonical hex).

---

5. Test Suite Requirements

5.1 ASL/1-STORE Suite

1. Put/get round-trip.
2. Idempotent put.
3. get on missing ref yields ERR_NOT_FOUND.
4. Collision on same ref yields ERR_INTEGRITY.
5. Unsupported config yields ERR_UNSUPPORTED.
6. Optional I/O fault path yields ERR_IO.

5.2 TGK/STORE/1 Suite

1. resolve_edge for valid edge returns EdgeBody.
2. resolve_edge for non-edge returns GS_ERR_NOT_EDGE.
3. resolve_edge for artifact integrity failure yields GS_ERR_ARTIFACT_ERROR.
4. edges_from, edges_to, edges_incident return identical lists across backends.
5. Deterministic ordering per TGK/STORE/1 canonical (hash_id, digest) rule.
6. Optional neighbors returns canonical node order.
7. Optional scan_edges enumerates all edges exactly once.

5.3 Replay Equivalence

1. Build a fixture ASL log/index state.
2. Replay into a fresh store.
3. Assert identical references and TGK results.

---

6. Ordering and Equivalence Rules

6.1 Edge Ordering

Edge lists MUST be sorted in ascending lexicographic order of: hash_id_bytes || digest_bytes, with hash_id encoded as big-endian u16.

6.2 Node Ordering

Neighbor lists MUST be sorted in ascending lexicographic order of: hash_id_bytes || digest_bytes, with hash_id encoded as big-endian u16.

6.3 Cross-Backend Equivalence

For a fixed snapshot and GraphStoreConfig, all backends MUST return:

1. Identical edge sets.
2. Identical ordered lists for adjacency queries.
3. Identical results for resolve_edge.

---

7. Harness Structure

Suggested structure:

1. tests/conformance/asl_store/*.c
2. tests/conformance/tgk_store/*.c
3. tests/fixtures/*.json
4. tools/amduat_conformance_run

The harness runner MUST:

1. Accept backend selector flags.
2. Accept fixture set selection.
3. Emit a machine-readable summary.

---

8. CI Gate

An adapter cannot be merged unless:

1. It passes its conformance suite.
2. Cross-backend equivalence is demonstrated.
3. Replay equivalence passes on at least one backend pair.

---

9. Out of Scope

1. Performance benchmarking.
2. Load testing.
3. Backend-specific tuning.

Those are separate gates after conformance is proven.