amduat-api/notes/overview.md

    ## 1. Layered Overview (Bottom-Up)

┌─────────────────────────────┐
│      Federation Layer       │
│  - Domains (local / remote) │
│  - Published vs internal    │
│  - Cross-domain references  │
│  - Federated snapshots      │
└─────────────┬──────────────┘
              │
┌─────────────▼──────────────┐
│     Provenance / PEL       │
│  - Execution receipts      │
│  - DAG execution           │
│  - Deterministic traces    │
│  - PEL-PROV / PEL-TRACE   │
└─────────────┬──────────────┘
              │
┌─────────────▼──────────────┐
│  Current / Snapshot Logic  │
│  - Checkpoint + append-only│
│    log → reconstruct CURRENT│
│  - Snapshot identity        │
└─────────────┬──────────────┘
              │
┌─────────────▼──────────────┐
│   Index Layer              │
│  - ASL-CORE-INDEX          │
│      • Artifact → Block    │
│      • Shadowing / tombstones│
│  - ASL-STORE-INDEX         │
│      • Block sealing       │
│      • Retention / GC      │
│      • Small/Large packing │
│  - ENC-ASL-CORE-INDEX      │
│      • On-disk record layout│
│      • Domain / visibility │
└─────────────┬──────────────┘
              │
┌─────────────▼──────────────┐
│  Artifact Storage Layer (ASL) │
│  - Blocks (immutable)          │
│  - BlockID → bytes mapping     │
│  - Small / large block handling│
│  - ZFS snapshot integration    │
│  - Append-only write log       │
└───────────────────────────────┘

---

2. Key Data Flows

2.1 Artifact Ingestion

1. Artifact created → broken into blocks (small or large).

2. Blocks stored in ASL (immutable).

3. Index record created:

   ArtifactKey → { (BlockID, offset, length), DomainID, Visibility }
   

4. Segment sealed → snapshot + log appended → CURRENT updated.

---

2.2 PEL Execution

1. PEL program DAG consumes artifacts (or receipts) from index.
2. Execution produces new artifacts → stored in ASL.
3. Receipts are generated → added to provenance trace.
4. Deterministic mapping preserved via index and snapshots.

---

2.3 Provenance Tracking

• Each artifact references:

  • Producing DAG program
  • Input artifacts (local or cross-domain published)
  • Snapshot in which artifact was created

• Trace graphs allow deterministic replay and verification.

---

2.4 Federation / Multi-Domain

• Domain-local artifacts: internal, invisible externally.
• Published artifacts: visible to other domains, read-only.
• Cross-domain references tracked in index (CrossDomainSource).
• Federated snapshots reconstructed by combining local + imported published artifacts.

---

2.5 Garbage Collection & Retention

• Blocks are pinned by:

  • CURRENT in snapshots
  • Published artifacts
  • Tombstones for shadowed artifacts

• GC may reclaim unreachable blocks without breaking provenance.

• Small packed blocks require careful per-artifact tracking.

---

3. Determinism & Guarantees

Layer	Determinism / Safety
ASL	Blocks immutable; snapshot + log deterministic
Index	Artifact → Block mapping stable per snapshot
PEL	DAG execution deterministic per artifact + snapshot
Provenance	Trace fully reconstructs execution DAG
Federation	Published artifact references deterministic across domains

---

4. Encoding & Storage

• Segments: units of storage with multiple records
• Records: (ArtifactKey, BlockID, offset, length, DomainID, Visibility, CrossDomainSource)
• Tombstones: track deleted/shadowed artifacts with domain/visibility metadata
• Small/Large block packing: store-level optimization, index unaffected
• ZFS snapshots: checkpoint state, append-only log ensures CURRENT reconstruction

---

5. Notes

• Small vs Large block handling: purely store-layer optimization
• Domain visibility: encoded in index, used for filtering CURRENT per domain
• Cross-domain references: read-only, preserve provenance, deterministic
• Execution receipts: stored as artifacts, may be consumed by other DAGs

---

This diagram and flow description captures:

• All layers from storage → index → PEL → provenance → federation
• Deterministic reconstruction from checkpoint + append-only log
• Block semantics, small/large handling, and domain visibility
• Integration of execution receipts into artifact flows and traces