148 lines
5.2 KiB
Markdown
148 lines
5.2 KiB
Markdown
|
## 1. Layered Overview (Bottom-Up)
|
||
|
|
|
||
|
|
```
|
||
|
|
┌─────────────────────────────┐
|
||
|
|
│ Federation Layer │
|
||
|
|
│ - Domains (local / remote) │
|
||
|
|
│ - Published vs internal │
|
||
|
|
│ - Cross-domain references │
|
||
|
|
│ - Federated snapshots │
|
||
|
|
└─────────────┬──────────────┘
|
||
|
|
│
|
||
|
|
┌─────────────▼──────────────┐
|
||
|
|
│ Provenance / PEL │
|
||
|
|
│ - Execution receipts │
|
||
|
|
│ - DAG execution │
|
||
|
|
│ - Deterministic traces │
|
||
|
|
│ - PEL-PROV / PEL-TRACE │
|
||
|
|
└─────────────┬──────────────┘
|
||
|
|
│
|
||
|
|
┌─────────────▼──────────────┐
|
||
|
|
│ Current / Snapshot Logic │
|
||
|
|
│ - Checkpoint + append-only│
|
||
|
|
│ log → reconstruct CURRENT│
|
||
|
|
│ - Snapshot identity │
|
||
|
|
└─────────────┬──────────────┘
|
||
|
|
│
|
||
|
|
┌─────────────▼──────────────┐
|
||
|
|
│ Index Layer │
|
||
|
|
│ - ASL-CORE-INDEX │
|
||
|
|
│ • Artifact → Block │
|
||
|
|
│ • Shadowing / tombstones│
|
||
|
|
│ - ASL-STORE-INDEX │
|
||
|
|
│ • Block sealing │
|
||
|
|
│ • Retention / GC │
|
||
|
|
│ • Small/Large packing │
|
||
|
|
│ - ENC-ASL-CORE-INDEX │
|
||
|
|
│ • On-disk record layout│
|
||
|
|
│ • Domain / visibility │
|
||
|
|
└─────────────┬──────────────┘
|
||
|
|
│
|
||
|
|
┌─────────────▼──────────────┐
|
||
|
|
│ Artifact Storage Layer (ASL) │
|
||
|
|
│ - Blocks (immutable) │
|
||
|
|
│ - BlockID → bytes mapping │
|
||
|
|
│ - Small / large block handling│
|
||
|
|
│ - ZFS snapshot integration │
|
||
|
|
│ - Append-only write log │
|
||
|
|
└───────────────────────────────┘
|
||
|
|
```
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## 2. Key Data Flows
|
||
|
|
|
||
|
|
### 2.1 Artifact Ingestion
|
||
|
|
|
||
|
|
1. Artifact created → broken into **blocks** (small or large).
|
||
|
|
2. Blocks stored in **ASL** (immutable).
|
||
|
|
3. Index record created:
|
||
|
|
|
||
|
|
```
|
||
|
|
ArtifactKey → { (BlockID, offset, length), DomainID, Visibility }
|
||
|
|
```
|
||
|
|
4. Segment sealed → snapshot + log appended → CURRENT updated.
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
### 2.2 PEL Execution
|
||
|
|
|
||
|
|
1. PEL program DAG consumes **artifacts** (or receipts) from index.
|
||
|
|
2. Execution produces new artifacts → stored in ASL.
|
||
|
|
3. Receipts are generated → added to provenance trace.
|
||
|
|
4. Deterministic mapping preserved via index and snapshots.
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
### 2.3 Provenance Tracking
|
||
|
|
|
||
|
|
* Each artifact references:
|
||
|
|
|
||
|
|
* Producing DAG program
|
||
|
|
* Input artifacts (local or cross-domain published)
|
||
|
|
* Snapshot in which artifact was created
|
||
|
|
* Trace graphs allow deterministic replay and verification.
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
### 2.4 Federation / Multi-Domain
|
||
|
|
|
||
|
|
* Domain-local artifacts: internal, invisible externally.
|
||
|
|
* Published artifacts: visible to other domains, read-only.
|
||
|
|
* Cross-domain references tracked in index (`CrossDomainSource`).
|
||
|
|
* Federated snapshots reconstructed by combining local + imported published artifacts.
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
### 2.5 Garbage Collection & Retention
|
||
|
|
|
||
|
|
* Blocks are pinned by:
|
||
|
|
|
||
|
|
* CURRENT in snapshots
|
||
|
|
* Published artifacts
|
||
|
|
* Tombstones for shadowed artifacts
|
||
|
|
* GC may reclaim unreachable blocks without breaking provenance.
|
||
|
|
* Small packed blocks require careful per-artifact tracking.
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## 3. Determinism & Guarantees
|
||
|
|
|
||
|
|
| Layer | Determinism / Safety |
|
||
|
|
| ---------- | ---------------------------------------------------------- |
|
||
|
|
| ASL | Blocks immutable; snapshot + log deterministic |
|
||
|
|
| Index | Artifact → Block mapping stable per snapshot |
|
||
|
|
| PEL | DAG execution deterministic per artifact + snapshot |
|
||
|
|
| Provenance | Trace fully reconstructs execution DAG |
|
||
|
|
| Federation | Published artifact references deterministic across domains |
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## 4. Encoding & Storage
|
||
|
|
|
||
|
|
* **Segments**: units of storage with multiple records
|
||
|
|
* **Records**: `(ArtifactKey, BlockID, offset, length, DomainID, Visibility, CrossDomainSource)`
|
||
|
|
* **Tombstones**: track deleted/shadowed artifacts with domain/visibility metadata
|
||
|
|
* **Small/Large block packing**: store-level optimization, index unaffected
|
||
|
|
* **ZFS snapshots**: checkpoint state, append-only log ensures CURRENT reconstruction
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## 5. Notes
|
||
|
|
|
||
|
|
* **Small vs Large block handling**: purely store-layer optimization
|
||
|
|
* **Domain visibility**: encoded in index, used for filtering CURRENT per domain
|
||
|
|
* **Cross-domain references**: read-only, preserve provenance, deterministic
|
||
|
|
* **Execution receipts**: stored as artifacts, may be consumed by other DAGs
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
This diagram and flow description captures:
|
||
|
|
|
||
|
|
* All layers from **storage → index → PEL → provenance → federation**
|
||
|
|
* Deterministic reconstruction from **checkpoint + append-only log**
|
||
|
|
* Block semantics, small/large handling, and domain visibility
|
||
|
|
* Integration of **execution receipts** into artifact flows and traces
|
||
|
|
|
||
|
|
|