146 lines
3.9 KiB
Markdown
146 lines
3.9 KiB
Markdown
# Indexing Requirements and Projections
|
|
|
|
Status: Draft
|
|
Scope: Define indexing contract, authority boundaries, and projection invariants for derived search/index systems (e.g., Elasticsearch).
|
|
|
|
---
|
|
|
|
# API Usage Rule (Normative)
|
|
|
|
1. Elasticsearch returns candidate References only.
|
|
2. Authoritative confirmation MUST occur via ASL/1-STORE and/or TGK/STORE/1.
|
|
3. Any ordering visible to clients MUST be computed using the canonical `(hash_id, digest)` ordering defined in TGK/STORE/1.
|
|
4. Elasticsearch MUST NOT be treated as a source of semantic truth.
|
|
|
|
---
|
|
|
|
# 1. Decision Checklist (Must Resolve Before Implementation)
|
|
|
|
1. Which artifact types are searchable.
|
|
2. Which fields are eligible for text extraction.
|
|
3. Whether `created_at` is intrinsic or ingestion-derived.
|
|
4. Checkpoint source of truth (ASL log position required).
|
|
5. Verification policy (which APIs require authoritative confirmation).
|
|
6. Edge indexing included in Phase A or deferred.
|
|
7. Redaction policy per TypeTag.
|
|
8. Projection schema versioning strategy.
|
|
|
|
No indexing implementation proceeds without resolving these.
|
|
|
|
---
|
|
|
|
# 2. Authority Boundary
|
|
|
|
## 2.1 Authoritative Sources
|
|
|
|
1. Artifact identity and content: ASL/1-STORE.
|
|
2. Graph semantics and ordering: TGK/STORE/1.
|
|
3. Canonical encoding: ENC/ASL1-CORE.
|
|
4. Identity derivation: ASL/1-CORE and HASH/ASL1.
|
|
|
|
Elasticsearch is a derived projection only.
|
|
|
|
## 2.2 No Semantic Drift Rule
|
|
|
|
1. No backend-specific interpretation of artifacts.
|
|
2. No inferred edges.
|
|
3. No implicit ordering.
|
|
4. No hidden type coercion.
|
|
5. All optimizations must be observationally equivalent to authoritative results.
|
|
|
|
---
|
|
|
|
# 3. Reference Encoding Rule
|
|
|
|
All `reference` fields indexed in ES MUST use canonical `ReferenceBytes` hex encoding as defined by ENC/ASL1-CORE.
|
|
|
|
This guarantees:
|
|
1. Unambiguous equality.
|
|
2. Stable identity across backends.
|
|
3. No ambiguity between `(hash_id, digest)` representations.
|
|
|
|
No alternative encodings are permitted.
|
|
|
|
---
|
|
|
|
# 4. Ordering Rule
|
|
|
|
Elasticsearch ordering MUST NOT be treated as authoritative.
|
|
|
|
Any client-visible ordering MUST be computed via the canonical `(hash_id, digest)` ordering defined in TGK/STORE/1.
|
|
|
|
If ES provides candidate edges, authoritative logic must re-order them.
|
|
|
|
---
|
|
|
|
# 5. Projection Contract
|
|
|
|
## 5.1 Replayability
|
|
|
|
Index state MUST be fully replayable from ASL log/index state, EdgeArtifacts, and deterministic decoding rules.
|
|
|
|
## 5.2 Observational Equivalence
|
|
|
|
Given identical ASL state, ES projection rebuilt from scratch MUST produce identical documents within a schema version.
|
|
|
|
## 5.3 Schema Versioning
|
|
|
|
Each projection index MUST have:
|
|
1. An explicit schema version, e.g., `projection_schema_version`.
|
|
2. The version recorded in a checkpoint artifact.
|
|
3. A versioned rebuild strategy.
|
|
|
|
Schema changes require full replay.
|
|
|
|
---
|
|
|
|
# 6. Lag Metric Definition
|
|
|
|
Lag is formally defined as:
|
|
|
|
```
|
|
Lag = ASL_head_log_position - indexer_checkpoint_log_position
|
|
```
|
|
|
|
Where:
|
|
1. `ASL_head_log_position` is authoritative.
|
|
2. `indexer_checkpoint_log_position` is recorded in a projection checkpoint artifact.
|
|
|
|
Lag MUST be observable via metrics.
|
|
|
|
---
|
|
|
|
# 7. Decoding and Extraction Rules
|
|
|
|
1. Artifact decoding MUST use only the declared encoding profile.
|
|
2. No heuristic decoding permitted.
|
|
3. Text extraction MUST be deterministic and versioned.
|
|
4. Extractor version MUST be stored in index documents.
|
|
|
|
---
|
|
|
|
# 8. Edge Indexing Guardrail
|
|
|
|
If edges are indexed:
|
|
1. Edge documents MUST be derived only from EdgeArtifacts.
|
|
2. No inferred adjacency allowed.
|
|
3. Edge semantics MUST respect TGK profile rules.
|
|
4. Ordering MUST be applied by the authoritative layer.
|
|
|
|
---
|
|
|
|
# 9. Security and Redaction
|
|
|
|
1. Per-TypeTag redaction policy is mandatory.
|
|
2. No secrets indexed unless explicitly approved.
|
|
3. Redaction rules MUST be declared before indexing begins.
|
|
|
|
---
|
|
|
|
# 10. Conformance Gates
|
|
|
|
1. System must function with ES disabled.
|
|
2. Full rebuild equivalence test required.
|
|
3. Ordering correctness test required.
|
|
4. Cross-backend equivalence test required.
|