amduat-api/notes/PEL-Integrated Execution Model with ASL + TGK.md

Absolutely — here’s a **formal draft for integrating PEL deterministic DAG programs with the ASL + TGK execution plan**, including how PEL outputs feed into operators, while preserving snapshot safety, determinism, and memory efficiency.

---

# PEL-Integrated Execution Model with ASL + TGK

---

## 1. Purpose

This specification defines:

1. How **PEL programs** (deterministic DAG programs) are executed in conjunction with **ASL artifacts** and **TGK edges**.
2. How **PEL outputs (execution receipts)** feed into **index operators**.
3. How **snapshot safety, determinism, and DAG traversal** are preserved across layers.
4. Integration of **sharding, SIMD acceleration, and tombstone shadowing** in a unified execution model.

---

## 2. Key Concepts

* **PEL Program (PP)**: A deterministic DAG with **input artifacts**, **output artifacts**, and **computational nodes** (concat, slice, primitive ops).
* **PEL Execution Receipt (PER)**: Artifact recording program execution, including:

  * Inputs consumed
  * Outputs produced
  * Canonical logseq / snapshot
* **Index Operators**: SegmentScan, IndexFilter, Merge, TGKTraversal, TombstoneShadow, Projection, Aggregation.
* **Snapshot Safety**: All reads of artifacts or TGK edges are constrained to `logseq ≤ snapshot`.
* **Determinism**: Execution order is fixed by **logseq ascending + canonical tie-breaker**.

---

## 3. Integration Principles

### 3.1 PEL Program Execution as Input

1. PEL program outputs (PER artifacts) are treated as **ASL artifacts** in execution plans.
2. Operators can consume **either raw artifacts or PERs**.
3. If the execution plan requires DAG traversal of PER-derived edges, treat **PER as a TGK edge node**.

```text
PEL program outputs → PER artifact → SegmentScan → IndexFilter → TGKTraversal
```

---

### 3.2 Deterministic DAG Mapping

1. Each PEL DAG node corresponds to a **logical operator in the execution plan**.
2. Execution plan DAG integrates **PEL DAG nodes** as **virtual operators**:

   * Input nodes → SegmentScan / IndexFilter
   * Computation nodes → Projection / Aggregation
   * Outputs → Artifact storage in ASL

---

### 3.3 Snapshot Propagation

* **Input artifacts** for PEL programs are fetched with snapshot bounds:

  ```
  artifact.logseq ≤ program.snapshot
  ```
* **Output PER artifacts** are written with:

  ```
  logseq = max(input_logseq) + 1
  ```
* All downstream index operators inherit **snapshot constraints**.

---

## 4. Runtime Integration Flow

1. **Load PEL Program DAG**

   * Validate deterministic operators
   * Identify **input artifacts** (raw or PER)

2. **Execute PEL Program**

   * Evaluate primitives (concat, slice, etc.)
   * Generate output artifacts (PER)
   * Each primitive produces deterministic outputs

3. **Register Outputs in Index**

   * PER artifacts are **visible to SegmentScan**
   * Type tag and canonical ID added to **shard-local buffers**

4. **Execute Index Operators**

   * SegmentScan → IndexFilter → TGKTraversal
   * Merge shards deterministically
   * Apply TombstoneShadow
   * Projection/Aggregation

5. **Return Results**

   * Combined output includes:

     * Raw ASL artifacts
     * PER artifacts
     * TGK traversal outputs

---

## 5. Pseudocode Sketch

```c
void ExecutePELProgramWithIndex(PELProgram *pp, snapshot_range_t snapshot, record_buffer_t *final_output) {
    // Step 1: Load inputs (artifacts or PERs)
    record_buffer_t input_buffer;
    LoadPELInputs(pp->inputs, snapshot, &input_buffer);

    // Step 2: Execute PEL DAG
    record_buffer_t per_buffer;
    ExecutePEL(pp, &input_buffer, snapshot, &per_buffer);

    // Step 3: Register PERs in ASL/TGK buffers
    record_buffer_t combined_buffer;
    Merge(&input_buffer, &per_buffer, 2, &combined_buffer);

    // Step 4: Run Index Operators
    exec_plan_t *plan = BuildExecutionPlan(pp, &combined_buffer);
    ExecutePlan(plan, snapshot, final_output);
}
```

* `LoadPELInputs` ensures **snapshot safety**
* `ExecutePEL` guarantees **deterministic outputs**
* `Merge` maintains **logseq + canonical ordering**
* `ExecutePlan` runs **ASL/TGK index operators with SIMD/shard acceleration**

---

## 6. Determinism Guarantees

1. **PEL DAG evaluation** deterministic per program snapshot
2. **PER artifacts** incorporated with canonical ID + logseq
3. **SegmentScan, Merge, TGKTraversal** maintain global ordering
4. **TombstoneShadow** ensures no resurrection of deleted artifacts
5. **SIMD/shard parallelism** does not affect final order

---

## 7. Memory Layout Integration

* PER artifacts stored in **shard-local buffers** alongside raw artifacts
* Type tags and canonical IDs mapped for **SIMD filtering**
* TGK traversal buffers extended to include **PER edges** where needed
* Snapshot ranges propagated to all buffers

```c
struct shard_buffer {
    uint64_t *artifact_ids;
    uint64_t *tgk_edge_ids;
    uint32_t  *type_tags;
    uint8_t   *roles;
    uint64_t   count;
    snapshot_range_t snapshot;
};
```

---

## 8. DAG Operator Mapping Summary

| PEL DAG Node        | Index Operator Equivalent | Notes                          |
| ------------------- | ------------------------- | ------------------------------ |
| Input Artifact      | SegmentScan               | Snapshot-limited               |
| Concat / Slice      | Projection                | Deterministic fields selected  |
| TGK Projection      | TGKTraversal              | DAG traversal depth controlled |
| Aggregate           | Aggregation               | Counts, sums, unions           |
| PER Artifact Output | SegmentScan               | Feed into downstream operators |

---

## 9. Federation & Multi-Machine Considerations

* PER artifacts can be **federated across nodes**
* Deterministic execution ensures **replayable results**
* Shard/SIMD acceleration applied locally
* Snapshot boundaries propagated across federation nodes

---

## 10. Summary

This integration:

* Treats **PEL programs as first-class DAG nodes** in execution plans
* Allows **PER artifacts** to feed seamlessly into index operators
* Preserves **snapshot safety, determinism, and shard/SIMD acceleration**
* Fully unifies **PEL execution** with **ASL artifact and TGK edge indexing**
* Enables **federated, replayable execution** across machines

---

Next step could be to **formalize federation propagation rules and cross-node deterministic replay**, ensuring PER artifacts are synchronized across pools and snapshots.

Do you want me to draft that next?