166 lines
3.9 KiB
Markdown
166 lines
3.9 KiB
Markdown
# Conformance Harness Specification
|
|
|
|
Status: Draft
|
|
Scope: Define the Phase 0 conformance harness goals, backend matrix, fixture format, and pass/fail gates for ASL and TGK store surfaces.
|
|
|
|
---
|
|
|
|
# 1. Purpose
|
|
|
|
The harness ensures that all backend adapters implement existing store semantics without drift.
|
|
|
|
It MUST:
|
|
1. Run the same test vectors across multiple backends.
|
|
2. Enforce deterministic ordering rules.
|
|
3. Validate replay equivalence.
|
|
4. Provide a single command entry point for CI.
|
|
|
|
---
|
|
|
|
# 2. Targets and Scope
|
|
|
|
## 2.1 Surfaces Under Test
|
|
|
|
1. ASL/1-STORE
|
|
2. TGK/STORE/1
|
|
|
|
Optional extensions:
|
|
1. ASL/STORE-INDEX/1 (index/log store)
|
|
2. TGK/PROV/1 (if provenance surface is exposed)
|
|
|
|
## 2.2 Conformance Mode
|
|
|
|
All tests are:
|
|
1. Deterministic.
|
|
2. Backend-agnostic.
|
|
3. Semantics-first (no timing assumptions).
|
|
|
|
---
|
|
|
|
# 3. Backend Matrix
|
|
|
|
The harness MUST be able to run each test suite against:
|
|
|
|
## 3.1 ASL Backends
|
|
|
|
1. `asl_store_fs`
|
|
2. `asl_store_index_fs`
|
|
3. Future: `asl_store_minio`
|
|
|
|
## 3.2 TGK Backends
|
|
|
|
1. `tgk_store_mem`
|
|
2. `tgk_store_fs`
|
|
3. `tgk_store_asl_index_fs`
|
|
4. Future: `tgk_store_janus`
|
|
|
|
---
|
|
|
|
# 4. Canonical Fixture Format
|
|
|
|
Fixtures are the source of truth for all conformance tests.
|
|
|
|
## 4.1 Requirements
|
|
|
|
1. Versioned fixture schema.
|
|
2. Stable ordering of artifacts and edges.
|
|
3. Explicit references encoded as canonical `ReferenceBytes` hex.
|
|
4. Deterministic content generation (no randomness at runtime).
|
|
|
|
## 4.2 Minimal Fields
|
|
|
|
For ASL fixtures:
|
|
1. `artifact_bytes` (hex or base64, fixed encoding).
|
|
2. `type_tag` (optional).
|
|
3. `expected_reference` (canonical Reference hex).
|
|
|
|
For TGK fixtures:
|
|
1. `edge_artifact_reference` (canonical Reference hex).
|
|
2. `edge_body` (decoded representation).
|
|
3. `edge_type_id`
|
|
4. `from_refs` and `to_refs` (ordered lists, canonical hex).
|
|
|
|
---
|
|
|
|
# 5. Test Suite Requirements
|
|
|
|
## 5.1 ASL/1-STORE Suite
|
|
|
|
1. Put/get round-trip.
|
|
2. Idempotent `put`.
|
|
3. `get` on missing ref yields `ERR_NOT_FOUND`.
|
|
4. Collision on same ref yields `ERR_INTEGRITY`.
|
|
5. Unsupported config yields `ERR_UNSUPPORTED`.
|
|
6. Optional I/O fault path yields `ERR_IO`.
|
|
|
|
## 5.2 TGK/STORE/1 Suite
|
|
|
|
1. `resolve_edge` for valid edge returns `EdgeBody`.
|
|
2. `resolve_edge` for non-edge returns `GS_ERR_NOT_EDGE`.
|
|
3. `resolve_edge` for artifact integrity failure yields `GS_ERR_ARTIFACT_ERROR`.
|
|
4. `edges_from`, `edges_to`, `edges_incident` return identical lists across backends.
|
|
5. Deterministic ordering per `TGK/STORE/1` canonical `(hash_id, digest)` rule.
|
|
6. Optional `neighbors` returns canonical node order.
|
|
7. Optional `scan_edges` enumerates all edges exactly once.
|
|
|
|
## 5.3 Replay Equivalence
|
|
|
|
1. Build a fixture ASL log/index state.
|
|
2. Replay into a fresh store.
|
|
3. Assert identical references and TGK results.
|
|
|
|
---
|
|
|
|
# 6. Ordering and Equivalence Rules
|
|
|
|
## 6.1 Edge Ordering
|
|
|
|
Edge lists MUST be sorted in ascending lexicographic order of:
|
|
`hash_id_bytes || digest_bytes`, with `hash_id` encoded as big-endian `u16`.
|
|
|
|
## 6.2 Node Ordering
|
|
|
|
Neighbor lists MUST be sorted in ascending lexicographic order of:
|
|
`hash_id_bytes || digest_bytes`, with `hash_id` encoded as big-endian `u16`.
|
|
|
|
## 6.3 Cross-Backend Equivalence
|
|
|
|
For a fixed snapshot and `GraphStoreConfig`, all backends MUST return:
|
|
1. Identical edge sets.
|
|
2. Identical ordered lists for adjacency queries.
|
|
3. Identical results for `resolve_edge`.
|
|
|
|
---
|
|
|
|
# 7. Harness Structure
|
|
|
|
Suggested structure:
|
|
1. `tests/conformance/asl_store/*.c`
|
|
2. `tests/conformance/tgk_store/*.c`
|
|
3. `tests/fixtures/*.json`
|
|
4. `tools/amduat_conformance_run`
|
|
|
|
The harness runner MUST:
|
|
1. Accept backend selector flags.
|
|
2. Accept fixture set selection.
|
|
3. Emit a machine-readable summary.
|
|
|
|
---
|
|
|
|
# 8. CI Gate
|
|
|
|
An adapter cannot be merged unless:
|
|
1. It passes its conformance suite.
|
|
2. Cross-backend equivalence is demonstrated.
|
|
3. Replay equivalence passes on at least one backend pair.
|
|
|
|
---
|
|
|
|
# 9. Out of Scope
|
|
|
|
1. Performance benchmarking.
|
|
2. Load testing.
|
|
3. Backend-specific tuning.
|
|
|
|
Those are separate gates after conformance is proven.
|