# Amduat Open-Source Backend # Implementation Plan Proposal **Status:** Architecture & Implementation Baseline v1 **Purpose:** Define phased implementation of the OSS backend stack as a spec-compliant implementation of Amduat store/query surfaces. --- # 1. Goal and Scope ## 1.1 Purpose This plan defines how to implement the open-source stack (MinIO, JanusGraph, Cassandra, Elasticsearch, Spark, Nessie, IPFS, Ollama) as a backend implementation of: * `ASL/1-STORE` * `TGK/STORE/1` * Optional `TGK/PROV/1` The plan ensures: * Strict semantic conformance * Deterministic reproducibility * Backend substitutability * Observational equivalence ## 1.2 Out of Scope * Modifying core specifications * Introducing new kernel ops * Elevating projections to semantic authority * Redefining identity or encoding rules ## 1.3 Target Tiers This plan covers: * Tier 0 — Embedded / Edge * Tier 1 — Platform / Scale-Out * Tier 2 — Enterprise / Audit-Grade --- # 2. Semantic Invariants All phases must preserve: ## 2.1 No Semantic Drift Rule * No backend-specific interpretation of artifacts * No implicit schema expansion * No hidden type coercion * No projection elevation without governance * All optimizations must be observationally equivalent to spec-defined behavior ## 2.2 Non-Negotiable Properties * Canonical encoding (`ENC/ASL1-CORE`) * Deterministic identity (`ASL/1-CORE` + `HASH/ASL1`) * Deterministic TGK projection (`TGK/1-CORE`) * Store/query surface conformance * Replay equivalence --- # 3. Decision Checklist (Must Resolve Upfront) These decisions must be explicitly documented before Phase 1 begins. --- ## 3.1 Conformance Tests * Exact test suites for: * `ASL/1-STORE` * `TGK/STORE/1` * Deterministic ordering checks * Replay equivalence tests * Acceptance criteria per tier --- ## 3.2 Backend Mapping ### ASL Mapping * Artifact storage layout in MinIO * Namespace strategy * Object key derivation rules * Versioning policy (if enabled) ### TGK Mapping * Vertex/edge mapping strategy * Deterministic ordering enforcement * Cassandra schema design * Index strategy in Elasticsearch (if used) --- ## 3.3 Consistency Model Per tier: * Write guarantees * Read-after-write expectations * Projection lag tolerance * Eventual consistency boundaries * Snapshot isolation model --- ## 3.4 Snapshot & Export Strategy * Snapshot format (artifact set, graph export, columnar dataset) * Snapshot identity model * Export cadence (manual, periodic, event-triggered) * Replay and equivalence validation rules --- ## 3.5 Governance & Projection Promotion * Rules for elevating a projection to authoritative * Audit requirements * Approval workflow --- ## 3.6 Failure & Recovery * Partial-write handling * Corruption detection * Divergence reconciliation * Reprojection strategy * Disaster recovery model --- ## 3.7 Pluggable Ops Registry * Storage location * Versioning strategy * Discovery mechanism * Trust policy --- ## 3.8 Execution Artifact Schemas Define required fields for: * ExecutionIntent * ExecutionReceipt * ExecutionOutput * ExecutionObservation Define canonical encoding and validation rules. --- ## 3.9 Security & Isolation * Executor trust boundaries * Secret management * Artifact integrity validation * Network isolation policies --- ## 3.10 Performance Targets Define SLOs per tier: * Ingest throughput * Query latency * Snapshot generation time * Recovery time objectives --- # 4. Architecture Mapping --- ## 4.1 ASL/1-STORE Implementation ### Responsibilities * Canonical encoding validation * Identity derivation * Artifact immutability * Retrieval guarantees ### Concrete Mapping * MinIO object store * Object key = Reference * Metadata includes encoding profile version * Optional replication policy --- ## 4.2 TGK/STORE/1 Implementation ### Responsibilities * Deterministic graph projection * Required query operations * Error model conformance * Deterministic ordering guarantees ### Concrete Mapping * JanusGraph for query layer * Cassandra for durable storage * Explicit ordering enforcement in query results * Elasticsearch as optional derived projection --- ## 4.3 Optional Projections ### Elasticsearch * Full-text search * Attribute filtering ### IPFS * Artifact distribution and caching ### Lakehouse (MinIO + Nessie + Spark) * Snapshot export * Batch provenance analysis * Audit workflows All remain derived projections. --- ## 4.4 Provenance Strategy `TGK/PROV/1` may be realized via: * Native graph traversal * Snapshot-based analytics * Hybrid strategy Must remain observationally equivalent. --- # 5. Interfaces and Contracts --- ## 5.1 ASL Interface * `put(Artifact) -> Reference` * `get(Reference) -> Artifact` Error model strictly per spec. --- ## 5.2 TGK Interface * Required query operations per `TGK/STORE/1` * Deterministic ordering rules * Explicit error semantics --- ## 5.3 Serialization * Canonical encoding at artifact boundary * No alternative encoding permitted * Versioned encoding profile tracking --- # 6. Execution Model (Pluggable Ops) --- ## 6.1 Registry Structure * External registry * Versioned definitions * Immutable op identifiers * Policy-scoped availability --- ## 6.2 Executor Lifecycle * Intent creation * Dispatch * Receipt capture * Output capture * Optional observation --- ## 6.3 Artifact Workflow 1. Write ExecutionIntent 2. Executor produces ExecutionReceipt 3. Executor produces ExecutionOutput 4. Optional ExecutionObservation 5. Link artifacts via EdgeArtifacts All artifacts stored in ASL. --- ## 6.4 TGK Linking * Intent → Receipt * Receipt → Output * Output → Observation Graph projection ensures traceability. --- # 7. Implementation Phases --- ## Phase 0 — Conformance Baseline * Build ASL/TGK conformance harness * Define canonical test datasets * Establish replay checks --- ## Phase 1 — ASL Store * MinIO integration * Canonical encoding enforcement * Hash-based identity derivation * Conformance validation --- ## Phase 2 — TGK Projection * Graph projection pipeline * JanusGraph integration * Deterministic ordering enforcement * Conformance validation --- ## Phase 3 — Projections & Indexing * Elasticsearch integration * IPFS optional layer * Monitoring of projection lag --- ## Phase 4 — Snapshot & Governance * Snapshot export format * Nessie integration * Replay equivalence testing * Governance workflows --- ## Phase 5 — Pluggable Ops & Executors * Registry implementation * Execution artifact schemas * Executor runtime * Traceability validation --- # 8. Testing and Verification Each phase must include: * Deterministic replay checks * Ordering validation tests * Snapshot equivalence tests * Cross-backend equivalence tests No phase advances without passing conformance. --- # 9. Operational Concerns --- ## 9.1 Deployment per Tier Tier 0: * Single-node embedded Tier 1: * Janus + Cassandra + ES * MinIO * Optional IPFS Tier 2: * Add Spark * Add Nessie * Governance workflows --- ## 9.2 Observability * Artifact write metrics * Graph projection metrics * Projection lag monitoring * Snapshot health indicators --- ## 9.3 Backup & Restore * MinIO replication or erasure coding * Cassandra repair strategy * Snapshot archival policy --- # 10. Open Questions and Risks * Canonical test dataset size * Cassandra compaction strategy * Projection drift detection * Snapshot storage growth * Executor trust hardening Each risk must have a mitigation strategy before Tier 2 rollout. --- # Design Outcome This implementation plan: * Preserves core invariants * Enables infrastructure scaling * Prevents semantic leakage * Structures phased, verifiable progress * Makes the OSS stack a compliant backend implementation