amduat/docs/indexing-requirements-and-projections.md

Indexing Requirements and Projections

Status: Draft Scope: Define indexing contract, authority boundaries, and projection invariants for derived search/index systems (e.g., Elasticsearch).

---

API Usage Rule (Normative)

1. Elasticsearch returns candidate References only.
2. Authoritative confirmation MUST occur via ASL/1-STORE and/or TGK/STORE/1.
3. Any ordering visible to clients MUST be computed using the canonical (hash_id, digest) ordering defined in TGK/STORE/1.
4. Elasticsearch MUST NOT be treated as a source of semantic truth.

---

1. Decision Checklist (Must Resolve Before Implementation)

1. Which artifact types are searchable.
2. Which fields are eligible for text extraction.
3. Whether created_at is intrinsic or ingestion-derived.
4. Checkpoint source of truth (ASL log position required).
5. Verification policy (which APIs require authoritative confirmation).
6. Edge indexing included in Phase A or deferred.
7. Redaction policy per TypeTag.
8. Projection schema versioning strategy.

No indexing implementation proceeds without resolving these.

---

2. Authority Boundary

2.1 Authoritative Sources

1. Artifact identity and content: ASL/1-STORE.
2. Graph semantics and ordering: TGK/STORE/1.
3. Canonical encoding: ENC/ASL1-CORE.
4. Identity derivation: ASL/1-CORE and HASH/ASL1.

Elasticsearch is a derived projection only.

2.2 No Semantic Drift Rule

1. No backend-specific interpretation of artifacts.
2. No inferred edges.
3. No implicit ordering.
4. No hidden type coercion.
5. All optimizations must be observationally equivalent to authoritative results.

---

3. Reference Encoding Rule

All reference fields indexed in ES MUST use canonical ReferenceBytes hex encoding as defined by ENC/ASL1-CORE.

This guarantees:

1. Unambiguous equality.
2. Stable identity across backends.
3. No ambiguity between (hash_id, digest) representations.

No alternative encodings are permitted.

---

4. Ordering Rule

Elasticsearch ordering MUST NOT be treated as authoritative.

Any client-visible ordering MUST be computed via the canonical (hash_id, digest) ordering defined in TGK/STORE/1.

If ES provides candidate edges, authoritative logic must re-order them.

---

5. Projection Contract

5.1 Replayability

Index state MUST be fully replayable from ASL log/index state, EdgeArtifacts, and deterministic decoding rules.

5.2 Observational Equivalence

Given identical ASL state, ES projection rebuilt from scratch MUST produce identical documents within a schema version.

5.3 Schema Versioning

Each projection index MUST have:

1. An explicit schema version, e.g., projection_schema_version.
2. The version recorded in a checkpoint artifact.
3. A versioned rebuild strategy.

Schema changes require full replay.

---

6. Lag Metric Definition

Lag is formally defined as:

Lag = ASL_head_log_position - indexer_checkpoint_log_position

Where:

1. ASL_head_log_position is authoritative.
2. indexer_checkpoint_log_position is recorded in a projection checkpoint artifact.

Lag MUST be observable via metrics.

---

7. Decoding and Extraction Rules

1. Artifact decoding MUST use only the declared encoding profile.
2. No heuristic decoding permitted.
3. Text extraction MUST be deterministic and versioned.
4. Extractor version MUST be stored in index documents.

---

8. Edge Indexing Guardrail

If edges are indexed:

1. Edge documents MUST be derived only from EdgeArtifacts.
2. No inferred adjacency allowed.
3. Edge semantics MUST respect TGK profile rules.
4. Ordering MUST be applied by the authoritative layer.

---

9. Security and Redaction

1. Per-TypeTag redaction policy is mandatory.
2. No secrets indexed unless explicitly approved.
3. Redaction rules MUST be declared before indexing begins.

---

10. Conformance Gates

1. System must function with ES disabled.
2. Full rebuild equivalence test required.
3. Ordering correctness test required.
4. Cross-backend equivalence test required.