82 lines
2.4 KiB
Markdown
82 lines
2.4 KiB
Markdown
|
# Projection Checkpoint Schema
|
||
|
|
|
||
|
|
Status: Draft
|
||
|
|
Scope: Schema for projection checkpoint artifacts used by derived indexers (e.g., Elasticsearch) to bind schema versions, replay state, and lag metrics to authoritative ASL state.
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
# 1. Purpose
|
||
|
|
|
||
|
|
Projection checkpoints make derived indexes replayable, auditable, and bounded. They tie a projection schema version to a specific ASL log position and snapshot identity so that rebuilds are deterministic and lag can be measured.
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
# 2. Canonical Checkpoint Artifact
|
||
|
|
|
||
|
|
Checkpoint artifacts are ASL artifacts whose bytes encode the following fields. The encoding is deterministic and versioned.
|
||
|
|
|
||
|
|
## 2.1 Minimal Fields
|
||
|
|
|
||
|
|
1. `checkpoint_version` (u32)
|
||
|
|
2. `projection_name` (string)
|
||
|
|
3. `projection_schema_version` (u32)
|
||
|
|
4. `asl_snapshot_id` (u64, optional if not available)
|
||
|
|
5. `asl_log_position` (u64, required)
|
||
|
|
6. `created_at` (u64, epoch milliseconds)
|
||
|
|
7. `builder_id` (string, identifying the indexer instance)
|
||
|
|
|
||
|
|
## 2.2 Optional Fields
|
||
|
|
|
||
|
|
1. `artifact_count` (u64)
|
||
|
|
2. `edge_count` (u64)
|
||
|
|
3. `error_count` (u64)
|
||
|
|
4. `lag_millis` (u64, derived)
|
||
|
|
5. `note` (string)
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
# 3. Identity and Storage
|
||
|
|
|
||
|
|
1. Checkpoints are stored as ASL artifacts using ASL/1-STORE semantics.
|
||
|
|
2. `projection_name` plus `projection_schema_version` must uniquely identify a projection format.
|
||
|
|
3. The checkpoint artifact reference is the authoritative identifier for the projection state.
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
# 4. Lag Metric Binding
|
||
|
|
|
||
|
|
Lag is defined as:
|
||
|
|
|
||
|
|
```
|
||
|
|
Lag = ASL_head_log_position - checkpoint.asl_log_position
|
||
|
|
```
|
||
|
|
|
||
|
|
The ASL head position is authoritative. The checkpoint log position is the last fully processed position.
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
# 5. Replay Rules
|
||
|
|
|
||
|
|
1. Rebuild begins at `checkpoint.asl_log_position + 1`.
|
||
|
|
2. Any schema change requires full replay from the earliest available log position.
|
||
|
|
3. Checkpoints must be monotonic in `asl_log_position` for a given projection.
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
# 6. Validation Requirements
|
||
|
|
|
||
|
|
When loading a checkpoint, the indexer MUST verify:
|
||
|
|
|
||
|
|
1. `projection_schema_version` matches the current build.
|
||
|
|
2. `asl_log_position` is not ahead of the authoritative ASL head.
|
||
|
|
3. `checkpoint_version` is supported.
|
||
|
|
4. If present, `asl_snapshot_id` matches the expected snapshot boundary.
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
# 7. Operational Guidance
|
||
|
|
|
||
|
|
1. Checkpoints should be written after a successful batch commit.
|
||
|
|
2. Checkpoints should be immutable; new checkpoints are separate artifacts.
|
||
|
|
3. Retain checkpoints for audit and rollback.
|