amduat/docs/implementation-plan.md

# Amduat Open-Source Backend

# Implementation Plan Proposal

**Status:** Architecture & Implementation Baseline v1
**Purpose:** Define phased implementation of the OSS backend stack as a spec-compliant implementation of Amduat store/query surfaces.

---

# 1. Goal and Scope

## 1.1 Purpose

This plan defines how to implement the open-source stack (MinIO, JanusGraph, Cassandra, Elasticsearch, Spark, Nessie, IPFS, Ollama) as a backend implementation of:

* `ASL/1-STORE`
* `TGK/STORE/1`
* Optional `TGK/PROV/1`

The plan ensures:

* Strict semantic conformance
* Deterministic reproducibility
* Backend substitutability
* Observational equivalence

## 1.2 Out of Scope

* Modifying core specifications
* Introducing new kernel ops
* Elevating projections to semantic authority
* Redefining identity or encoding rules

## 1.3 Target Tiers

This plan covers:

* Tier 0 — Embedded / Edge
* Tier 1 — Platform / Scale-Out
* Tier 2 — Enterprise / Audit-Grade

---

# 2. Semantic Invariants

All phases must preserve:

## 2.1 No Semantic Drift Rule

* No backend-specific interpretation of artifacts
* No implicit schema expansion
* No hidden type coercion
* No projection elevation without governance
* All optimizations must be observationally equivalent to spec-defined behavior

## 2.2 Non-Negotiable Properties

* Canonical encoding (`ENC/ASL1-CORE`)
* Deterministic identity (`ASL/1-CORE` + `HASH/ASL1`)
* Deterministic TGK projection (`TGK/1-CORE`)
* Store/query surface conformance
* Replay equivalence

---

# 3. Decision Checklist (Must Resolve Upfront)

These decisions must be explicitly documented before Phase 1 begins.

---

## 3.1 Conformance Tests

* Exact test suites for:

  * `ASL/1-STORE`
  * `TGK/STORE/1`
* Deterministic ordering checks
* Replay equivalence tests
* Acceptance criteria per tier

---

## 3.2 Backend Mapping

### ASL Mapping

* Artifact storage layout in MinIO
* Namespace strategy
* Object key derivation rules
* Versioning policy (if enabled)

### TGK Mapping

* Vertex/edge mapping strategy
* Deterministic ordering enforcement
* Cassandra schema design
* Index strategy in Elasticsearch (if used)

---

## 3.3 Consistency Model

Per tier:

* Write guarantees
* Read-after-write expectations
* Projection lag tolerance
* Eventual consistency boundaries
* Snapshot isolation model

---

## 3.4 Snapshot & Export Strategy

* Snapshot format (artifact set, graph export, columnar dataset)
* Snapshot identity model
* Export cadence (manual, periodic, event-triggered)
* Replay and equivalence validation rules

---

## 3.5 Governance & Projection Promotion

* Rules for elevating a projection to authoritative
* Audit requirements
* Approval workflow

---

## 3.6 Failure & Recovery

* Partial-write handling
* Corruption detection
* Divergence reconciliation
* Reprojection strategy
* Disaster recovery model

---

## 3.7 Pluggable Ops Registry

* Storage location
* Versioning strategy
* Discovery mechanism
* Trust policy

---

## 3.8 Execution Artifact Schemas

Define required fields for:

* ExecutionIntent
* ExecutionReceipt
* ExecutionOutput
* ExecutionObservation

Define canonical encoding and validation rules.

---

## 3.9 Security & Isolation

* Executor trust boundaries
* Secret management
* Artifact integrity validation
* Network isolation policies

---

## 3.10 Performance Targets

Define SLOs per tier:

* Ingest throughput
* Query latency
* Snapshot generation time
* Recovery time objectives

---

# 4. Architecture Mapping

---

## 4.1 ASL/1-STORE Implementation

### Responsibilities

* Canonical encoding validation
* Identity derivation
* Artifact immutability
* Retrieval guarantees

### Concrete Mapping

* MinIO object store
* Object key = Reference
* Metadata includes encoding profile version
* Optional replication policy

---

## 4.2 TGK/STORE/1 Implementation

### Responsibilities

* Deterministic graph projection
* Required query operations
* Error model conformance
* Deterministic ordering guarantees

### Concrete Mapping

* JanusGraph for query layer
* Cassandra for durable storage
* Explicit ordering enforcement in query results
* Elasticsearch as optional derived projection

---

## 4.3 Optional Projections

### Elasticsearch

* Full-text search
* Attribute filtering

### IPFS

* Artifact distribution and caching

### Lakehouse (MinIO + Nessie + Spark)

* Snapshot export
* Batch provenance analysis
* Audit workflows

All remain derived projections.

---

## 4.4 Provenance Strategy

`TGK/PROV/1` may be realized via:

* Native graph traversal
* Snapshot-based analytics
* Hybrid strategy

Must remain observationally equivalent.

---

# 5. Interfaces and Contracts

---

## 5.1 ASL Interface

* `put(Artifact) -> Reference`
* `get(Reference) -> Artifact`

Error model strictly per spec.

---

## 5.2 TGK Interface

* Required query operations per `TGK/STORE/1`
* Deterministic ordering rules
* Explicit error semantics

---

## 5.3 Serialization

* Canonical encoding at artifact boundary
* No alternative encoding permitted
* Versioned encoding profile tracking

---

# 6. Execution Model (Pluggable Ops)

---

## 6.1 Registry Structure

* External registry
* Versioned definitions
* Immutable op identifiers
* Policy-scoped availability

---

## 6.2 Executor Lifecycle

* Intent creation
* Dispatch
* Receipt capture
* Output capture
* Optional observation

---

## 6.3 Artifact Workflow

1. Write ExecutionIntent
2. Executor produces ExecutionReceipt
3. Executor produces ExecutionOutput
4. Optional ExecutionObservation
5. Link artifacts via EdgeArtifacts

All artifacts stored in ASL.

---

## 6.4 TGK Linking

* Intent → Receipt
* Receipt → Output
* Output → Observation

Graph projection ensures traceability.

---

# 7. Implementation Phases

---

## Phase 0 — Conformance Baseline

* Build ASL/TGK conformance harness
* Define canonical test datasets
* Establish replay checks

---

## Phase 1 — ASL Store

* MinIO integration
* Canonical encoding enforcement
* Hash-based identity derivation
* Conformance validation

---

## Phase 2 — TGK Projection

* Graph projection pipeline
* JanusGraph integration
* Deterministic ordering enforcement
* Conformance validation

---

## Phase 3 — Projections & Indexing

* Elasticsearch integration
* IPFS optional layer
* Monitoring of projection lag

---

## Phase 4 — Snapshot & Governance

* Snapshot export format
* Nessie integration
* Replay equivalence testing
* Governance workflows

---

## Phase 5 — Pluggable Ops & Executors

* Registry implementation
* Execution artifact schemas
* Executor runtime
* Traceability validation

---

# 8. Testing and Verification

Each phase must include:

* Deterministic replay checks
* Ordering validation tests
* Snapshot equivalence tests
* Cross-backend equivalence tests

No phase advances without passing conformance.

---

# 9. Operational Concerns

---

## 9.1 Deployment per Tier

Tier 0:

* Single-node embedded

Tier 1:

* Janus + Cassandra + ES
* MinIO
* Optional IPFS

Tier 2:

* Add Spark
* Add Nessie
* Governance workflows

---

## 9.2 Observability

* Artifact write metrics
* Graph projection metrics
* Projection lag monitoring
* Snapshot health indicators

---

## 9.3 Backup & Restore

* MinIO replication or erasure coding
* Cassandra repair strategy
* Snapshot archival policy

---

# 10. Open Questions and Risks

* Canonical test dataset size
* Cassandra compaction strategy
* Projection drift detection
* Snapshot storage growth
* Executor trust hardening

Each risk must have a mitigation strategy before Tier 2 rollout.

---

# Design Outcome

This implementation plan:

* Preserves core invariants
* Enables infrastructure scaling
* Prevents semantic leakage
* Structures phased, verifiable progress
* Makes the OSS stack a compliant backend implementation