amduat-api/notes/PEL-Integrated Execution Model with ASL + TGK.md

Absolutely — here’s a formal draft for integrating PEL deterministic DAG programs with the ASL + TGK execution plan, including how PEL outputs feed into operators, while preserving snapshot safety, determinism, and memory efficiency.

---

PEL-Integrated Execution Model with ASL + TGK

---

1. Purpose

This specification defines:

1. How PEL programs (deterministic DAG programs) are executed in conjunction with ASL artifacts and TGK edges.
2. How PEL outputs (execution receipts) feed into index operators.
3. How snapshot safety, determinism, and DAG traversal are preserved across layers.
4. Integration of sharding, SIMD acceleration, and tombstone shadowing in a unified execution model.

---

2. Key Concepts

• PEL Program (PP): A deterministic DAG with input artifacts, output artifacts, and computational nodes (concat, slice, primitive ops).

• PEL Execution Receipt (PER): Artifact recording program execution, including:

  • Inputs consumed
  • Outputs produced
  • Canonical logseq / snapshot

• Index Operators: SegmentScan, IndexFilter, Merge, TGKTraversal, TombstoneShadow, Projection, Aggregation.

• Snapshot Safety: All reads of artifacts or TGK edges are constrained to logseq ≤ snapshot.

• Determinism: Execution order is fixed by logseq ascending + canonical tie-breaker.

---

3. Integration Principles

3.1 PEL Program Execution as Input

1. PEL program outputs (PER artifacts) are treated as ASL artifacts in execution plans.
2. Operators can consume either raw artifacts or PERs.
3. If the execution plan requires DAG traversal of PER-derived edges, treat PER as a TGK edge node.

PEL program outputs → PER artifact → SegmentScan → IndexFilter → TGKTraversal

---

3.2 Deterministic DAG Mapping

1. Each PEL DAG node corresponds to a logical operator in the execution plan.

2. Execution plan DAG integrates PEL DAG nodes as virtual operators:

   • Input nodes → SegmentScan / IndexFilter
   • Computation nodes → Projection / Aggregation
   • Outputs → Artifact storage in ASL

---

3.3 Snapshot Propagation

• Input artifacts for PEL programs are fetched with snapshot bounds:

  artifact.logseq ≤ program.snapshot
  

• Output PER artifacts are written with:

  logseq = max(input_logseq) + 1
  

• All downstream index operators inherit snapshot constraints.

---

4. Runtime Integration Flow

1. Load PEL Program DAG

   • Validate deterministic operators
   • Identify input artifacts (raw or PER)

2. Execute PEL Program

   • Evaluate primitives (concat, slice, etc.)
   • Generate output artifacts (PER)
   • Each primitive produces deterministic outputs

3. Register Outputs in Index

   • PER artifacts are visible to SegmentScan
   • Type tag and canonical ID added to shard-local buffers

4. Execute Index Operators

   • SegmentScan → IndexFilter → TGKTraversal
   • Merge shards deterministically
   • Apply TombstoneShadow
   • Projection/Aggregation

5. Return Results

   • Combined output includes:

     • Raw ASL artifacts
     • PER artifacts
     • TGK traversal outputs

---

5. Pseudocode Sketch

void ExecutePELProgramWithIndex(PELProgram *pp, snapshot_range_t snapshot, record_buffer_t *final_output) {
    // Step 1: Load inputs (artifacts or PERs)
    record_buffer_t input_buffer;
    LoadPELInputs(pp->inputs, snapshot, &input_buffer);

    // Step 2: Execute PEL DAG
    record_buffer_t per_buffer;
    ExecutePEL(pp, &input_buffer, snapshot, &per_buffer);

    // Step 3: Register PERs in ASL/TGK buffers
    record_buffer_t combined_buffer;
    Merge(&input_buffer, &per_buffer, 2, &combined_buffer);

    // Step 4: Run Index Operators
    exec_plan_t *plan = BuildExecutionPlan(pp, &combined_buffer);
    ExecutePlan(plan, snapshot, final_output);
}

• LoadPELInputs ensures snapshot safety
• ExecutePEL guarantees deterministic outputs
• Merge maintains logseq + canonical ordering
• ExecutePlan runs ASL/TGK index operators with SIMD/shard acceleration

---

6. Determinism Guarantees

1. PEL DAG evaluation deterministic per program snapshot
2. PER artifacts incorporated with canonical ID + logseq
3. SegmentScan, Merge, TGKTraversal maintain global ordering
4. TombstoneShadow ensures no resurrection of deleted artifacts
5. SIMD/shard parallelism does not affect final order

---

7. Memory Layout Integration

• PER artifacts stored in shard-local buffers alongside raw artifacts
• Type tags and canonical IDs mapped for SIMD filtering
• TGK traversal buffers extended to include PER edges where needed
• Snapshot ranges propagated to all buffers

struct shard_buffer {
    uint64_t *artifact_ids;
    uint64_t *tgk_edge_ids;
    uint32_t  *type_tags;
    uint8_t   *roles;
    uint64_t   count;
    snapshot_range_t snapshot;
};

---

8. DAG Operator Mapping Summary

PEL DAG Node	Index Operator Equivalent	Notes
Input Artifact	SegmentScan	Snapshot-limited
Concat / Slice	Projection	Deterministic fields selected
TGK Projection	TGKTraversal	DAG traversal depth controlled
Aggregate	Aggregation	Counts, sums, unions
PER Artifact Output	SegmentScan	Feed into downstream operators

---

9. Federation & Multi-Machine Considerations

• PER artifacts can be federated across nodes
• Deterministic execution ensures replayable results
• Shard/SIMD acceleration applied locally
• Snapshot boundaries propagated across federation nodes

---

10. Summary

This integration:

• Treats PEL programs as first-class DAG nodes in execution plans
• Allows PER artifacts to feed seamlessly into index operators
• Preserves snapshot safety, determinism, and shard/SIMD acceleration
• Fully unifies PEL execution with ASL artifact and TGK edge indexing
• Enables federated, replayable execution across machines

---

Next step could be to formalize federation propagation rules and cross-node deterministic replay, ensuring PER artifacts are synchronized across pools and snapshots.

Do you want me to draft that next?