206 lines
6.4 KiB
Markdown
206 lines
6.4 KiB
Markdown
Absolutely — here’s a **formal draft for integrating PEL deterministic DAG programs with the ASL + TGK execution plan**, including how PEL outputs feed into operators, while preserving snapshot safety, determinism, and memory efficiency.
|
||
|
||
---
|
||
|
||
# PEL-Integrated Execution Model with ASL + TGK
|
||
|
||
---
|
||
|
||
## 1. Purpose
|
||
|
||
This specification defines:
|
||
|
||
1. How **PEL programs** (deterministic DAG programs) are executed in conjunction with **ASL artifacts** and **TGK edges**.
|
||
2. How **PEL outputs (execution receipts)** feed into **index operators**.
|
||
3. How **snapshot safety, determinism, and DAG traversal** are preserved across layers.
|
||
4. Integration of **sharding, SIMD acceleration, and tombstone shadowing** in a unified execution model.
|
||
|
||
---
|
||
|
||
## 2. Key Concepts
|
||
|
||
* **PEL Program (PP)**: A deterministic DAG with **input artifacts**, **output artifacts**, and **computational nodes** (concat, slice, primitive ops).
|
||
* **PEL Execution Receipt (PER)**: Artifact recording program execution, including:
|
||
|
||
* Inputs consumed
|
||
* Outputs produced
|
||
* Canonical logseq / snapshot
|
||
* **Index Operators**: SegmentScan, IndexFilter, Merge, TGKTraversal, TombstoneShadow, Projection, Aggregation.
|
||
* **Snapshot Safety**: All reads of artifacts or TGK edges are constrained to `logseq ≤ snapshot`.
|
||
* **Determinism**: Execution order is fixed by **logseq ascending + canonical tie-breaker**.
|
||
|
||
---
|
||
|
||
## 3. Integration Principles
|
||
|
||
### 3.1 PEL Program Execution as Input
|
||
|
||
1. PEL program outputs (PER artifacts) are treated as **ASL artifacts** in execution plans.
|
||
2. Operators can consume **either raw artifacts or PERs**.
|
||
3. If the execution plan requires DAG traversal of PER-derived edges, treat **PER as a TGK edge node**.
|
||
|
||
```text
|
||
PEL program outputs → PER artifact → SegmentScan → IndexFilter → TGKTraversal
|
||
```
|
||
|
||
---
|
||
|
||
### 3.2 Deterministic DAG Mapping
|
||
|
||
1. Each PEL DAG node corresponds to a **logical operator in the execution plan**.
|
||
2. Execution plan DAG integrates **PEL DAG nodes** as **virtual operators**:
|
||
|
||
* Input nodes → SegmentScan / IndexFilter
|
||
* Computation nodes → Projection / Aggregation
|
||
* Outputs → Artifact storage in ASL
|
||
|
||
---
|
||
|
||
### 3.3 Snapshot Propagation
|
||
|
||
* **Input artifacts** for PEL programs are fetched with snapshot bounds:
|
||
|
||
```
|
||
artifact.logseq ≤ program.snapshot
|
||
```
|
||
* **Output PER artifacts** are written with:
|
||
|
||
```
|
||
logseq = max(input_logseq) + 1
|
||
```
|
||
* All downstream index operators inherit **snapshot constraints**.
|
||
|
||
---
|
||
|
||
## 4. Runtime Integration Flow
|
||
|
||
1. **Load PEL Program DAG**
|
||
|
||
* Validate deterministic operators
|
||
* Identify **input artifacts** (raw or PER)
|
||
|
||
2. **Execute PEL Program**
|
||
|
||
* Evaluate primitives (concat, slice, etc.)
|
||
* Generate output artifacts (PER)
|
||
* Each primitive produces deterministic outputs
|
||
|
||
3. **Register Outputs in Index**
|
||
|
||
* PER artifacts are **visible to SegmentScan**
|
||
* Type tag and canonical ID added to **shard-local buffers**
|
||
|
||
4. **Execute Index Operators**
|
||
|
||
* SegmentScan → IndexFilter → TGKTraversal
|
||
* Merge shards deterministically
|
||
* Apply TombstoneShadow
|
||
* Projection/Aggregation
|
||
|
||
5. **Return Results**
|
||
|
||
* Combined output includes:
|
||
|
||
* Raw ASL artifacts
|
||
* PER artifacts
|
||
* TGK traversal outputs
|
||
|
||
---
|
||
|
||
## 5. Pseudocode Sketch
|
||
|
||
```c
|
||
void ExecutePELProgramWithIndex(PELProgram *pp, snapshot_range_t snapshot, record_buffer_t *final_output) {
|
||
// Step 1: Load inputs (artifacts or PERs)
|
||
record_buffer_t input_buffer;
|
||
LoadPELInputs(pp->inputs, snapshot, &input_buffer);
|
||
|
||
// Step 2: Execute PEL DAG
|
||
record_buffer_t per_buffer;
|
||
ExecutePEL(pp, &input_buffer, snapshot, &per_buffer);
|
||
|
||
// Step 3: Register PERs in ASL/TGK buffers
|
||
record_buffer_t combined_buffer;
|
||
Merge(&input_buffer, &per_buffer, 2, &combined_buffer);
|
||
|
||
// Step 4: Run Index Operators
|
||
exec_plan_t *plan = BuildExecutionPlan(pp, &combined_buffer);
|
||
ExecutePlan(plan, snapshot, final_output);
|
||
}
|
||
```
|
||
|
||
* `LoadPELInputs` ensures **snapshot safety**
|
||
* `ExecutePEL` guarantees **deterministic outputs**
|
||
* `Merge` maintains **logseq + canonical ordering**
|
||
* `ExecutePlan` runs **ASL/TGK index operators with SIMD/shard acceleration**
|
||
|
||
---
|
||
|
||
## 6. Determinism Guarantees
|
||
|
||
1. **PEL DAG evaluation** deterministic per program snapshot
|
||
2. **PER artifacts** incorporated with canonical ID + logseq
|
||
3. **SegmentScan, Merge, TGKTraversal** maintain global ordering
|
||
4. **TombstoneShadow** ensures no resurrection of deleted artifacts
|
||
5. **SIMD/shard parallelism** does not affect final order
|
||
|
||
---
|
||
|
||
## 7. Memory Layout Integration
|
||
|
||
* PER artifacts stored in **shard-local buffers** alongside raw artifacts
|
||
* Type tags and canonical IDs mapped for **SIMD filtering**
|
||
* TGK traversal buffers extended to include **PER edges** where needed
|
||
* Snapshot ranges propagated to all buffers
|
||
|
||
```c
|
||
struct shard_buffer {
|
||
uint64_t *artifact_ids;
|
||
uint64_t *tgk_edge_ids;
|
||
uint32_t *type_tags;
|
||
uint8_t *roles;
|
||
uint64_t count;
|
||
snapshot_range_t snapshot;
|
||
};
|
||
```
|
||
|
||
---
|
||
|
||
## 8. DAG Operator Mapping Summary
|
||
|
||
| PEL DAG Node | Index Operator Equivalent | Notes |
|
||
| ------------------- | ------------------------- | ------------------------------ |
|
||
| Input Artifact | SegmentScan | Snapshot-limited |
|
||
| Concat / Slice | Projection | Deterministic fields selected |
|
||
| TGK Projection | TGKTraversal | DAG traversal depth controlled |
|
||
| Aggregate | Aggregation | Counts, sums, unions |
|
||
| PER Artifact Output | SegmentScan | Feed into downstream operators |
|
||
|
||
---
|
||
|
||
## 9. Federation & Multi-Machine Considerations
|
||
|
||
* PER artifacts can be **federated across nodes**
|
||
* Deterministic execution ensures **replayable results**
|
||
* Shard/SIMD acceleration applied locally
|
||
* Snapshot boundaries propagated across federation nodes
|
||
|
||
---
|
||
|
||
## 10. Summary
|
||
|
||
This integration:
|
||
|
||
* Treats **PEL programs as first-class DAG nodes** in execution plans
|
||
* Allows **PER artifacts** to feed seamlessly into index operators
|
||
* Preserves **snapshot safety, determinism, and shard/SIMD acceleration**
|
||
* Fully unifies **PEL execution** with **ASL artifact and TGK edge indexing**
|
||
* Enables **federated, replayable execution** across machines
|
||
|
||
---
|
||
|
||
Next step could be to **formalize federation propagation rules and cross-node deterministic replay**, ensuring PER artifacts are synchronized across pools and snapshots.
|
||
|
||
Do you want me to draft that next?
|
||
|