461 lines
7.6 KiB
Markdown
461 lines
7.6 KiB
Markdown
# Amduat Open-Source Backend
|
|
|
|
# Implementation Plan Proposal
|
|
|
|
**Status:** Architecture & Implementation Baseline v1
|
|
**Purpose:** Define phased implementation of the OSS backend stack as a spec-compliant implementation of Amduat store/query surfaces.
|
|
|
|
---
|
|
|
|
# 1. Goal and Scope
|
|
|
|
## 1.1 Purpose
|
|
|
|
This plan defines how to implement the open-source stack (MinIO, JanusGraph, Cassandra, Elasticsearch, Spark, Nessie, IPFS, Ollama) as a backend implementation of:
|
|
|
|
* `ASL/1-STORE`
|
|
* `TGK/STORE/1`
|
|
* Optional `TGK/PROV/1`
|
|
|
|
The plan ensures:
|
|
|
|
* Strict semantic conformance
|
|
* Deterministic reproducibility
|
|
* Backend substitutability
|
|
* Observational equivalence
|
|
|
|
## 1.2 Out of Scope
|
|
|
|
* Modifying core specifications
|
|
* Introducing new kernel ops
|
|
* Elevating projections to semantic authority
|
|
* Redefining identity or encoding rules
|
|
|
|
## 1.3 Target Tiers
|
|
|
|
This plan covers:
|
|
|
|
* Tier 0 — Embedded / Edge
|
|
* Tier 1 — Platform / Scale-Out
|
|
* Tier 2 — Enterprise / Audit-Grade
|
|
|
|
---
|
|
|
|
# 2. Semantic Invariants
|
|
|
|
All phases must preserve:
|
|
|
|
## 2.1 No Semantic Drift Rule
|
|
|
|
* No backend-specific interpretation of artifacts
|
|
* No implicit schema expansion
|
|
* No hidden type coercion
|
|
* No projection elevation without governance
|
|
* All optimizations must be observationally equivalent to spec-defined behavior
|
|
|
|
## 2.2 Non-Negotiable Properties
|
|
|
|
* Canonical encoding (`ENC/ASL1-CORE`)
|
|
* Deterministic identity (`ASL/1-CORE` + `HASH/ASL1`)
|
|
* Deterministic TGK projection (`TGK/1-CORE`)
|
|
* Store/query surface conformance
|
|
* Replay equivalence
|
|
|
|
---
|
|
|
|
# 3. Decision Checklist (Must Resolve Upfront)
|
|
|
|
These decisions must be explicitly documented before Phase 1 begins.
|
|
|
|
---
|
|
|
|
## 3.1 Conformance Tests
|
|
|
|
* Exact test suites for:
|
|
|
|
* `ASL/1-STORE`
|
|
* `TGK/STORE/1`
|
|
* Deterministic ordering checks
|
|
* Replay equivalence tests
|
|
* Acceptance criteria per tier
|
|
|
|
---
|
|
|
|
## 3.2 Backend Mapping
|
|
|
|
### ASL Mapping
|
|
|
|
* Artifact storage layout in MinIO
|
|
* Namespace strategy
|
|
* Object key derivation rules
|
|
* Versioning policy (if enabled)
|
|
|
|
### TGK Mapping
|
|
|
|
* Vertex/edge mapping strategy
|
|
* Deterministic ordering enforcement
|
|
* Cassandra schema design
|
|
* Index strategy in Elasticsearch (if used)
|
|
|
|
---
|
|
|
|
## 3.3 Consistency Model
|
|
|
|
Per tier:
|
|
|
|
* Write guarantees
|
|
* Read-after-write expectations
|
|
* Projection lag tolerance
|
|
* Eventual consistency boundaries
|
|
* Snapshot isolation model
|
|
|
|
---
|
|
|
|
## 3.4 Snapshot & Export Strategy
|
|
|
|
* Snapshot format (artifact set, graph export, columnar dataset)
|
|
* Snapshot identity model
|
|
* Export cadence (manual, periodic, event-triggered)
|
|
* Replay and equivalence validation rules
|
|
|
|
---
|
|
|
|
## 3.5 Governance & Projection Promotion
|
|
|
|
* Rules for elevating a projection to authoritative
|
|
* Audit requirements
|
|
* Approval workflow
|
|
|
|
---
|
|
|
|
## 3.6 Failure & Recovery
|
|
|
|
* Partial-write handling
|
|
* Corruption detection
|
|
* Divergence reconciliation
|
|
* Reprojection strategy
|
|
* Disaster recovery model
|
|
|
|
---
|
|
|
|
## 3.7 Pluggable Ops Registry
|
|
|
|
* Storage location
|
|
* Versioning strategy
|
|
* Discovery mechanism
|
|
* Trust policy
|
|
|
|
---
|
|
|
|
## 3.8 Execution Artifact Schemas
|
|
|
|
Define required fields for:
|
|
|
|
* ExecutionIntent
|
|
* ExecutionReceipt
|
|
* ExecutionOutput
|
|
* ExecutionObservation
|
|
|
|
Define canonical encoding and validation rules.
|
|
|
|
---
|
|
|
|
## 3.9 Security & Isolation
|
|
|
|
* Executor trust boundaries
|
|
* Secret management
|
|
* Artifact integrity validation
|
|
* Network isolation policies
|
|
|
|
---
|
|
|
|
## 3.10 Performance Targets
|
|
|
|
Define SLOs per tier:
|
|
|
|
* Ingest throughput
|
|
* Query latency
|
|
* Snapshot generation time
|
|
* Recovery time objectives
|
|
|
|
---
|
|
|
|
# 4. Architecture Mapping
|
|
|
|
---
|
|
|
|
## 4.1 ASL/1-STORE Implementation
|
|
|
|
### Responsibilities
|
|
|
|
* Canonical encoding validation
|
|
* Identity derivation
|
|
* Artifact immutability
|
|
* Retrieval guarantees
|
|
|
|
### Concrete Mapping
|
|
|
|
* MinIO object store
|
|
* Object key = Reference
|
|
* Metadata includes encoding profile version
|
|
* Optional replication policy
|
|
|
|
---
|
|
|
|
## 4.2 TGK/STORE/1 Implementation
|
|
|
|
### Responsibilities
|
|
|
|
* Deterministic graph projection
|
|
* Required query operations
|
|
* Error model conformance
|
|
* Deterministic ordering guarantees
|
|
|
|
### Concrete Mapping
|
|
|
|
* JanusGraph for query layer
|
|
* Cassandra for durable storage
|
|
* Explicit ordering enforcement in query results
|
|
* Elasticsearch as optional derived projection
|
|
|
|
---
|
|
|
|
## 4.3 Optional Projections
|
|
|
|
### Elasticsearch
|
|
|
|
* Full-text search
|
|
* Attribute filtering
|
|
|
|
### IPFS
|
|
|
|
* Artifact distribution and caching
|
|
|
|
### Lakehouse (MinIO + Nessie + Spark)
|
|
|
|
* Snapshot export
|
|
* Batch provenance analysis
|
|
* Audit workflows
|
|
|
|
All remain derived projections.
|
|
|
|
---
|
|
|
|
## 4.4 Provenance Strategy
|
|
|
|
`TGK/PROV/1` may be realized via:
|
|
|
|
* Native graph traversal
|
|
* Snapshot-based analytics
|
|
* Hybrid strategy
|
|
|
|
Must remain observationally equivalent.
|
|
|
|
---
|
|
|
|
# 5. Interfaces and Contracts
|
|
|
|
---
|
|
|
|
## 5.1 ASL Interface
|
|
|
|
* `put(Artifact) -> Reference`
|
|
* `get(Reference) -> Artifact`
|
|
|
|
Error model strictly per spec.
|
|
|
|
---
|
|
|
|
## 5.2 TGK Interface
|
|
|
|
* Required query operations per `TGK/STORE/1`
|
|
* Deterministic ordering rules
|
|
* Explicit error semantics
|
|
|
|
---
|
|
|
|
## 5.3 Serialization
|
|
|
|
* Canonical encoding at artifact boundary
|
|
* No alternative encoding permitted
|
|
* Versioned encoding profile tracking
|
|
|
|
---
|
|
|
|
# 6. Execution Model (Pluggable Ops)
|
|
|
|
---
|
|
|
|
## 6.1 Registry Structure
|
|
|
|
* External registry
|
|
* Versioned definitions
|
|
* Immutable op identifiers
|
|
* Policy-scoped availability
|
|
|
|
---
|
|
|
|
## 6.2 Executor Lifecycle
|
|
|
|
* Intent creation
|
|
* Dispatch
|
|
* Receipt capture
|
|
* Output capture
|
|
* Optional observation
|
|
|
|
---
|
|
|
|
## 6.3 Artifact Workflow
|
|
|
|
1. Write ExecutionIntent
|
|
2. Executor produces ExecutionReceipt
|
|
3. Executor produces ExecutionOutput
|
|
4. Optional ExecutionObservation
|
|
5. Link artifacts via EdgeArtifacts
|
|
|
|
All artifacts stored in ASL.
|
|
|
|
---
|
|
|
|
## 6.4 TGK Linking
|
|
|
|
* Intent → Receipt
|
|
* Receipt → Output
|
|
* Output → Observation
|
|
|
|
Graph projection ensures traceability.
|
|
|
|
---
|
|
|
|
# 7. Implementation Phases
|
|
|
|
---
|
|
|
|
## Phase 0 — Conformance Baseline
|
|
|
|
* Build ASL/TGK conformance harness
|
|
* Define canonical test datasets
|
|
* Establish replay checks
|
|
|
|
---
|
|
|
|
## Phase 1 — ASL Store
|
|
|
|
* MinIO integration
|
|
* Canonical encoding enforcement
|
|
* Hash-based identity derivation
|
|
* Conformance validation
|
|
|
|
---
|
|
|
|
## Phase 2 — TGK Projection
|
|
|
|
* Graph projection pipeline
|
|
* JanusGraph integration
|
|
* Deterministic ordering enforcement
|
|
* Conformance validation
|
|
|
|
---
|
|
|
|
## Phase 3 — Projections & Indexing
|
|
|
|
* Elasticsearch integration
|
|
* IPFS optional layer
|
|
* Monitoring of projection lag
|
|
|
|
---
|
|
|
|
## Phase 4 — Snapshot & Governance
|
|
|
|
* Snapshot export format
|
|
* Nessie integration
|
|
* Replay equivalence testing
|
|
* Governance workflows
|
|
|
|
---
|
|
|
|
## Phase 5 — Pluggable Ops & Executors
|
|
|
|
* Registry implementation
|
|
* Execution artifact schemas
|
|
* Executor runtime
|
|
* Traceability validation
|
|
|
|
---
|
|
|
|
# 8. Testing and Verification
|
|
|
|
Each phase must include:
|
|
|
|
* Deterministic replay checks
|
|
* Ordering validation tests
|
|
* Snapshot equivalence tests
|
|
* Cross-backend equivalence tests
|
|
|
|
No phase advances without passing conformance.
|
|
|
|
---
|
|
|
|
# 9. Operational Concerns
|
|
|
|
---
|
|
|
|
## 9.1 Deployment per Tier
|
|
|
|
Tier 0:
|
|
|
|
* Single-node embedded
|
|
|
|
Tier 1:
|
|
|
|
* Janus + Cassandra + ES
|
|
* MinIO
|
|
* Optional IPFS
|
|
|
|
Tier 2:
|
|
|
|
* Add Spark
|
|
* Add Nessie
|
|
* Governance workflows
|
|
|
|
---
|
|
|
|
## 9.2 Observability
|
|
|
|
* Artifact write metrics
|
|
* Graph projection metrics
|
|
* Projection lag monitoring
|
|
* Snapshot health indicators
|
|
|
|
---
|
|
|
|
## 9.3 Backup & Restore
|
|
|
|
* MinIO replication or erasure coding
|
|
* Cassandra repair strategy
|
|
* Snapshot archival policy
|
|
|
|
---
|
|
|
|
# 10. Open Questions and Risks
|
|
|
|
* Canonical test dataset size
|
|
* Cassandra compaction strategy
|
|
* Projection drift detection
|
|
* Snapshot storage growth
|
|
* Executor trust hardening
|
|
|
|
Each risk must have a mitigation strategy before Tier 2 rollout.
|
|
|
|
---
|
|
|
|
# Design Outcome
|
|
|
|
This implementation plan:
|
|
|
|
* Preserves core invariants
|
|
* Enables infrastructure scaling
|
|
* Prevents semantic leakage
|
|
* Structures phased, verifiable progress
|
|
* Makes the OSS stack a compliant backend implementation
|