amduat/docs/projector-pattern.md

# Projector Pattern

**Status:** Draft
**Scope:** Define the minimal projector loop for derived projections in Amduat, using the repo’s cursor API and versioned checkpoint records stored as ASL records. This pattern applies to Elasticsearch indexing, lakehouse snapshots, IPFS exports, and other derived views.

---

## 1. Purpose

A **projector** is a deterministic process that:

1. reads authoritative ASL log/index state in a stable order,
2. applies a projection function to produce derived state (or derived side effects), and
3. persists progress as a **projection checkpoint record** stored in ASL.

Projectors accelerate discovery and analysis without becoming semantic authority.

---

## 2. Normative Rules

### 2.1 Authority

* **Authoritative truth** is the deployed `ASL/1-STORE` implementation plus deterministic TGK projection derived from ASL artifacts.
* Projectors produce **derived projections** only.

### 2.2 No Semantic Drift

* Projectors MUST NOT introduce backend-specific semantics.
* Projectors MUST NOT infer edges or meaning beyond declared encoding profiles and spec-defined decoding.
* Any optimization MUST be observationally equivalent to results obtainable from authoritative stores.

### 2.3 Disable-ability

* The system MUST remain correct when any projector is disabled.
* Projections may lag and may be eventually consistent.

---

## 3. Inputs, Outputs, and State

### 3.1 Inputs

A projector consumes:

* authoritative ASL index state (current head position)
* **ASL log records** read via cursor paging
* optional decoding of artifacts using only declared encoding profiles

**Important:** the ASL log is **not** a general “artifact publish/unpublish event stream.” Cursor replay iterates over **log records** (segment seals, tombstones, snapshot anchors, etc.), and any “what to project” logic must be derived from these records plus authoritative store state.

### 3.2 Outputs

A projector emits:

* derived side effects (e.g., write to Elasticsearch; export/pin to IPFS; write snapshot files)
* optional derived artifacts (if you store projection outputs back into ASL)
* **a projection checkpoint record** for restart/resume

### 3.3 Projector State Model

Projector state must be reconstructible from:

* authoritative ASL store + log/index state, and
* the last checkpoint record.

No hidden state is permitted.

---

## 4. Minimal Projector Loop

This is the canonical pattern. Implementations may wrap it in a service, cron job, or CLI, but the core loop remains the same.

### 4.1 Pseudocode (repo-aligned)

```c
// Inputs
amduat_asl_store_t *store = ...;

// Authoritative head
amduat_asl_index_state_t head = amduat_asl_index_current_state(store);

// Resume (optional)
projection_checkpoint_t ckpt;
bool have_ckpt = load_latest_checkpoint(store, /*projection_name*/ "...", &ckpt);

// Cursor start uses checkpoint log_position as after_logseq
uint64_t after_logseq = have_ckpt ? ckpt.log_position : 0;

for (;;) {
  // Cursor reads log records (not “publish events”)
  asl_log_page_t page = amduat_asl_log_read_since(store, after_logseq, /*limit*/ N);

  if (page.count == 0) break;

  // Project: interpret log records as needed, using only declared profiles for decoding.
  projection_apply(page.records, page.count, /*derived sinks/state*/ ...);

  // Advance deterministically to the returned next position
  after_logseq = page.next_log_position;

  // Persist checkpoint as an ASL record: schema = "projection/checkpoint"
  projection_checkpoint_t next = (projection_checkpoint_t){
    .projection_name = "...",                    // required
    .projection_schema_version = SCHEMA_VER,     // required
    .log_position = after_logseq,                // required
    .builder_id = "...",                         // required
    .created_at = 0,                             // required (0 allowed)
    // optional: .snapshot_id = ..., counts, note, lag_millis, etc.
  };

  asl_record_t rec = projection_checkpoint_encode(&next); // schema "projection/checkpoint"
  asl_put_record(store, &rec);
}
```

### 4.2 Determinism Requirements

* Given the same ASL store state and the same projection schema version, repeated runs MUST produce identical derived outputs (within the declared projection contract).
* Cursor paging boundaries MUST NOT affect results.
* A checkpoint MUST allow resume without changing derived outcomes.

### 4.3 Implementation Slice (Checklist + Stub)

**Checklist (minimal, repo-aligned):**

* Choose a stable `projection_name`, `projection_schema_version`, and `builder_id`.
* Read authoritative head state via `amduat_asl_index_current_state`.
* Page log records with `amduat_asl_log_read_since` (cursor-driven).
* Apply a deterministic projection step (even if it is just a counter in the stub).
* Encode a checkpoint via `amduat_asl_projection_checkpoint_encode_v1`.
* Store the checkpoint as an ASL record with schema `projection/checkpoint`.
* Verify the stored checkpoint decodes and matches the last processed `log_position`.

**Repo stub:** `tests/asl/test_asl_projection_replay.c` includes
`test_projector_loop_stub()` that runs the minimal loop and persists a checkpoint.

---

## 5. Projection Checkpoint Records

### 5.1 Storage Convention (Kernel Layer)

Projection checkpoints are stored as **ASL records** with schema string:

> `projection/checkpoint`

This is the kernel-layer storage convention for restart/resume progress tracking.

### 5.2 Required Fields (repo-aligned)

Current required fields are:

* `projection_name`
* `projection_schema_version`
* `log_position`
* `builder_id`
* `created_at` (may be `0`)

Optional fields may include (as supported by current implementation/schema):

* `snapshot_id`
* counts
* `note`
* `lag_millis`

(See `projection_checkpoint` implementation and schema doc.)

### 5.3 Lag Metric (repo-aligned)

Lag is computed against the authoritative ASL index head:

```text
Lag = head.log_position − checkpoint.log_position
```

Where `head.log_position` comes from `amduat_asl_index_current_state(store)`.

---

## 6. Projection Schema Versioning

The repository currently treats `projection_schema_version` as a versioned field, but does not define major/minor/patch semantics in core.

**Recommended policy (not a MUST):**

* Treat incompatible schema changes as requiring full replay.
* Record schema version in each checkpoint.
* If schema version changes, prefer rebuild or explicit migration procedures.

Any stricter version policy should live in governance / deployment policy docs, not kernel-level requirements.

---

## 7. Decoding and Extraction Rules

* Projectors MAY decode artifacts, but decoding MUST use **only the declared encoding profile**.
* No heuristic decoding is permitted.
* Any extraction (e.g., text extraction for ES) must be deterministic and should be versioned (extractor version treated as part of the projection schema contract).

---

## 8. Ordering and Verification Rules

### 8.1 TGK Ordering

Any ordering visible to clients MUST be computed using canonical ordering defined by `TGK/STORE/1` (e.g., `(hash_id, digest)`), not by derived store defaults (ES sorting, DB insertion order, etc.).

### 8.2 Verification

If a projector drives an API that returns candidates (e.g., ES search):

* Results MUST be treated as **candidates**
* Truth MUST be confirmed against authoritative `ASL/1-STORE` and/or `TGK/STORE/1` before returning semantically meaningful answers.

---

## 9. How Specific Backends Fit

### 9.1 Elasticsearch

* ES stores derived documents keyed by canonical Reference hex.
* Projector writes ES docs and persists checkpoint records (`projection/checkpoint`) in ASL.
* ES is never authoritative; APIs must verify.

### 9.2 Lakehouse Snapshots (Nessie/Spark)

* Projector produces snapshot datasets derived from authoritative sources.
* Snapshot identifiers should be tied to a checkpoint boundary (log_position) for reproducibility.

### 9.3 IPFS Anchoring

* IPFS pinning is an export/distribution projection.
* Projector should emit pin/export receipts as artifacts (recommended).
* IPFS availability is not semantic truth.

---

## 10. Operational Guidance

### 10.1 Failure Handling

* Projector must be restartable from the last checkpoint.
* Partial writes to derived stores must not corrupt checkpoint progression:

  * either write side effects then checkpoint, or
  * checkpoint only after side effects are durably confirmed (policy choice; document per projector)

### 10.2 Observability

Expose:

* head log_position
* checkpoint log_position
* lag
* projection error counts / dead-letter queue
* rebuild duration
* verification fallback rate (for ES-backed APIs)

---

## 11. Non-Goals

This pattern does not define:

* daemon orchestration
* distributed consensus
* tenancy/authz
* semantic promotion of projections (governance handles promotion explicitly)

---

## 12. Design Outcome

This projector pattern ensures:

* replayable derived systems,
* stable restart/resume via checkpoint records,
* deterministic behavior independent of paging,
* strict separation of truth vs acceleration,
* safe foundations for MinIO/Janus/ES/IPFS without semantic drift.