## 1. Layered Overview (Bottom-Up) ``` ┌─────────────────────────────┐ │ Federation Layer │ │ - Domains (local / remote) │ │ - Published vs internal │ │ - Cross-domain references │ │ - Federated snapshots │ └─────────────┬──────────────┘ │ ┌─────────────▼──────────────┐ │ Provenance / PEL │ │ - Execution receipts │ │ - DAG execution │ │ - Deterministic traces │ │ - PEL-PROV / PEL-TRACE │ └─────────────┬──────────────┘ │ ┌─────────────▼──────────────┐ │ Current / Snapshot Logic │ │ - Checkpoint + append-only│ │ log → reconstruct CURRENT│ │ - Snapshot identity │ └─────────────┬──────────────┘ │ ┌─────────────▼──────────────┐ │ Index Layer │ │ - ASL-CORE-INDEX │ │ • Artifact → Block │ │ • Shadowing / tombstones│ │ - ASL-STORE-INDEX │ │ • Block sealing │ │ • Retention / GC │ │ • Small/Large packing │ │ - ENC-ASL-CORE-INDEX │ │ • On-disk record layout│ │ • Domain / visibility │ └─────────────┬──────────────┘ │ ┌─────────────▼──────────────┐ │ Artifact Storage Layer (ASL) │ │ - Blocks (immutable) │ │ - BlockID → bytes mapping │ │ - Small / large block handling│ │ - ZFS snapshot integration │ │ - Append-only write log │ └───────────────────────────────┘ ``` --- ## 2. Key Data Flows ### 2.1 Artifact Ingestion 1. Artifact created → broken into **blocks** (small or large). 2. Blocks stored in **ASL** (immutable). 3. Index record created: ``` ArtifactKey → { (BlockID, offset, length), DomainID, Visibility } ``` 4. Segment sealed → snapshot + log appended → CURRENT updated. --- ### 2.2 PEL Execution 1. PEL program DAG consumes **artifacts** (or receipts) from index. 2. Execution produces new artifacts → stored in ASL. 3. Receipts are generated → added to provenance trace. 4. Deterministic mapping preserved via index and snapshots. --- ### 2.3 Provenance Tracking * Each artifact references: * Producing DAG program * Input artifacts (local or cross-domain published) * Snapshot in which artifact was created * Trace graphs allow deterministic replay and verification. --- ### 2.4 Federation / Multi-Domain * Domain-local artifacts: internal, invisible externally. * Published artifacts: visible to other domains, read-only. * Cross-domain references tracked in index (`CrossDomainSource`). * Federated snapshots reconstructed by combining local + imported published artifacts. --- ### 2.5 Garbage Collection & Retention * Blocks are pinned by: * CURRENT in snapshots * Published artifacts * Tombstones for shadowed artifacts * GC may reclaim unreachable blocks without breaking provenance. * Small packed blocks require careful per-artifact tracking. --- ## 3. Determinism & Guarantees | Layer | Determinism / Safety | | ---------- | ---------------------------------------------------------- | | ASL | Blocks immutable; snapshot + log deterministic | | Index | Artifact → Block mapping stable per snapshot | | PEL | DAG execution deterministic per artifact + snapshot | | Provenance | Trace fully reconstructs execution DAG | | Federation | Published artifact references deterministic across domains | --- ## 4. Encoding & Storage * **Segments**: units of storage with multiple records * **Records**: `(ArtifactKey, BlockID, offset, length, DomainID, Visibility, CrossDomainSource)` * **Tombstones**: track deleted/shadowed artifacts with domain/visibility metadata * **Small/Large block packing**: store-level optimization, index unaffected * **ZFS snapshots**: checkpoint state, append-only log ensures CURRENT reconstruction --- ## 5. Notes * **Small vs Large block handling**: purely store-layer optimization * **Domain visibility**: encoded in index, used for filtering CURRENT per domain * **Cross-domain references**: read-only, preserve provenance, deterministic * **Execution receipts**: stored as artifacts, may be consumed by other DAGs --- This diagram and flow description captures: * All layers from **storage → index → PEL → provenance → federation** * Deterministic reconstruction from **checkpoint + append-only log** * Block semantics, small/large handling, and domain visibility * Integration of **execution receipts** into artifact flows and traces