amduat/docs/projection-checkpoint-schema.md

Projection Checkpoint Schema

Status: Draft Scope: Schema for projection checkpoint artifacts used by derived indexers (e.g., Elasticsearch) to bind schema versions, replay state, and lag metrics to authoritative ASL state.

---

1. Purpose

Projection checkpoints make derived indexes replayable, auditable, and bounded. They tie a projection schema version to a specific ASL log position and snapshot identity so that rebuilds are deterministic and lag can be measured.

---

2. Canonical Checkpoint Artifact

Checkpoint artifacts are ASL artifacts whose bytes encode the following fields. The encoding is deterministic and versioned.

2.1 Minimal Fields

1. checkpoint_version (u32)
2. projection_name (string)
3. projection_schema_version (u32)
4. asl_snapshot_id (u64, optional if not available)
5. asl_log_position (u64, required)
6. created_at (u64, epoch milliseconds)
7. builder_id (string, identifying the indexer instance)

2.2 Optional Fields

1. artifact_count (u64)
2. edge_count (u64)
3. error_count (u64)
4. lag_millis (u64, derived)
5. note (string)

---

3. Identity and Storage

1. Checkpoints are stored as ASL artifacts using ASL/1-STORE semantics.
2. projection_name plus projection_schema_version must uniquely identify a projection format.
3. The checkpoint artifact reference is the authoritative identifier for the projection state.

---

4. Lag Metric Binding

Lag is defined as:

Lag = ASL_head_log_position - checkpoint.asl_log_position

The ASL head position is authoritative. The checkpoint log position is the last fully processed position.

---

5. Replay Rules

1. Rebuild begins at checkpoint.asl_log_position + 1.
2. Any schema change requires full replay from the earliest available log position.
3. Checkpoints must be monotonic in asl_log_position for a given projection.

---

6. Validation Requirements

When loading a checkpoint, the indexer MUST verify:

1. projection_schema_version matches the current build.
2. asl_log_position is not ahead of the authoritative ASL head.
3. checkpoint_version is supported.
4. If present, asl_snapshot_id matches the expected snapshot boundary.

---

7. Operational Guidance

1. Checkpoints should be written after a successful batch commit.
2. Checkpoints should be immutable; new checkpoints are separate artifacts.
3. Retain checkpoints for audit and rollback.