niklas/amduat
1
0
Fork 0
Code Issues Pull requests Actions Packages Projects Releases Wiki Activity
amduat/docs/implementation-plan.md

7.6 KiB
Raw Blame History

Amduat Open-Source Backend

Implementation Plan Proposal

Status: Architecture & Implementation Baseline v1 Purpose: Define phased implementation of the OSS backend stack as a spec-compliant implementation of Amduat store/query surfaces.

1. Goal and Scope

1.1 Purpose

This plan defines how to implement the open-source stack (MinIO, JanusGraph, Cassandra, Elasticsearch, Spark, Nessie, IPFS, Ollama) as a backend implementation of:

  • ASL/1-STORE
  • TGK/STORE/1
  • Optional TGK/PROV/1

The plan ensures:

  • Strict semantic conformance
  • Deterministic reproducibility
  • Backend substitutability
  • Observational equivalence

1.2 Out of Scope

  • Modifying core specifications
  • Introducing new kernel ops
  • Elevating projections to semantic authority
  • Redefining identity or encoding rules

1.3 Target Tiers

This plan covers:

  • Tier 0 — Embedded / Edge
  • Tier 1 — Platform / Scale-Out
  • Tier 2 — Enterprise / Audit-Grade

2. Semantic Invariants

All phases must preserve:

2.1 No Semantic Drift Rule

  • No backend-specific interpretation of artifacts
  • No implicit schema expansion
  • No hidden type coercion
  • No projection elevation without governance
  • All optimizations must be observationally equivalent to spec-defined behavior

2.2 Non-Negotiable Properties

  • Canonical encoding (ENC/ASL1-CORE)
  • Deterministic identity (ASL/1-CORE + HASH/ASL1)
  • Deterministic TGK projection (TGK/1-CORE)
  • Store/query surface conformance
  • Replay equivalence

3. Decision Checklist (Must Resolve Upfront)

These decisions must be explicitly documented before Phase 1 begins.

3.1 Conformance Tests

  • Exact test suites for:

    • ASL/1-STORE
    • TGK/STORE/1

  • Deterministic ordering checks

  • Replay equivalence tests

  • Acceptance criteria per tier

3.2 Backend Mapping

ASL Mapping

  • Artifact storage layout in MinIO
  • Namespace strategy
  • Object key derivation rules
  • Versioning policy (if enabled)

TGK Mapping

  • Vertex/edge mapping strategy
  • Deterministic ordering enforcement
  • Cassandra schema design
  • Index strategy in Elasticsearch (if used)

3.3 Consistency Model

Per tier:

  • Write guarantees
  • Read-after-write expectations
  • Projection lag tolerance
  • Eventual consistency boundaries
  • Snapshot isolation model

3.4 Snapshot & Export Strategy

  • Snapshot format (artifact set, graph export, columnar dataset)
  • Snapshot identity model
  • Export cadence (manual, periodic, event-triggered)
  • Replay and equivalence validation rules

3.5 Governance & Projection Promotion

  • Rules for elevating a projection to authoritative
  • Audit requirements
  • Approval workflow

3.6 Failure & Recovery

  • Partial-write handling
  • Corruption detection
  • Divergence reconciliation
  • Reprojection strategy
  • Disaster recovery model

3.7 Pluggable Ops Registry

  • Storage location
  • Versioning strategy
  • Discovery mechanism
  • Trust policy

3.8 Execution Artifact Schemas

Define required fields for:

  • ExecutionIntent
  • ExecutionReceipt
  • ExecutionOutput
  • ExecutionObservation

Define canonical encoding and validation rules.

3.9 Security & Isolation

  • Executor trust boundaries
  • Secret management
  • Artifact integrity validation
  • Network isolation policies

3.10 Performance Targets

Define SLOs per tier:

  • Ingest throughput
  • Query latency
  • Snapshot generation time
  • Recovery time objectives

4. Architecture Mapping

4.1 ASL/1-STORE Implementation

Responsibilities

  • Canonical encoding validation
  • Identity derivation
  • Artifact immutability
  • Retrieval guarantees

Concrete Mapping

  • MinIO object store
  • Object key = Reference
  • Metadata includes encoding profile version
  • Optional replication policy

4.2 TGK/STORE/1 Implementation

Responsibilities

  • Deterministic graph projection
  • Required query operations
  • Error model conformance
  • Deterministic ordering guarantees

Concrete Mapping

  • JanusGraph for query layer
  • Cassandra for durable storage
  • Explicit ordering enforcement in query results
  • Elasticsearch as optional derived projection

4.3 Optional Projections

Elasticsearch

  • Full-text search
  • Attribute filtering

IPFS

  • Artifact distribution and caching

Lakehouse (MinIO + Nessie + Spark)

  • Snapshot export
  • Batch provenance analysis
  • Audit workflows

All remain derived projections.

4.4 Provenance Strategy

TGK/PROV/1 may be realized via:

  • Native graph traversal
  • Snapshot-based analytics
  • Hybrid strategy

Must remain observationally equivalent.

5. Interfaces and Contracts

5.1 ASL Interface

  • put(Artifact) -> Reference
  • get(Reference) -> Artifact

Error model strictly per spec.

5.2 TGK Interface

  • Required query operations per TGK/STORE/1
  • Deterministic ordering rules
  • Explicit error semantics

5.3 Serialization

  • Canonical encoding at artifact boundary
  • No alternative encoding permitted
  • Versioned encoding profile tracking

6. Execution Model (Pluggable Ops)

6.1 Registry Structure

  • External registry
  • Versioned definitions
  • Immutable op identifiers
  • Policy-scoped availability

6.2 Executor Lifecycle

  • Intent creation
  • Dispatch
  • Receipt capture
  • Output capture
  • Optional observation

6.3 Artifact Workflow

  1. Write ExecutionIntent
  2. Executor produces ExecutionReceipt
  3. Executor produces ExecutionOutput
  4. Optional ExecutionObservation
  5. Link artifacts via EdgeArtifacts

All artifacts stored in ASL.

6.4 TGK Linking

  • Intent → Receipt
  • Receipt → Output
  • Output → Observation

Graph projection ensures traceability.

7. Implementation Phases

Phase 0 — Conformance Baseline

  • Build ASL/TGK conformance harness
  • Define canonical test datasets
  • Establish replay checks

Phase 1 — ASL Store

  • MinIO integration
  • Canonical encoding enforcement
  • Hash-based identity derivation
  • Conformance validation

Phase 2 — TGK Projection

  • Graph projection pipeline
  • JanusGraph integration
  • Deterministic ordering enforcement
  • Conformance validation

Phase 3 — Projections & Indexing

  • Elasticsearch integration
  • IPFS optional layer
  • Monitoring of projection lag

Phase 4 — Snapshot & Governance

  • Snapshot export format
  • Nessie integration
  • Replay equivalence testing
  • Governance workflows

Phase 5 — Pluggable Ops & Executors

  • Registry implementation
  • Execution artifact schemas
  • Executor runtime
  • Traceability validation

8. Testing and Verification

Each phase must include:

  • Deterministic replay checks
  • Ordering validation tests
  • Snapshot equivalence tests
  • Cross-backend equivalence tests

No phase advances without passing conformance.

9. Operational Concerns

9.1 Deployment per Tier

Tier 0:

  • Single-node embedded

Tier 1:

  • Janus + Cassandra + ES
  • MinIO
  • Optional IPFS

Tier 2:

  • Add Spark
  • Add Nessie
  • Governance workflows

9.2 Observability

  • Artifact write metrics
  • Graph projection metrics
  • Projection lag monitoring
  • Snapshot health indicators

9.3 Backup & Restore

  • MinIO replication or erasure coding
  • Cassandra repair strategy
  • Snapshot archival policy

10. Open Questions and Risks

  • Canonical test dataset size
  • Cassandra compaction strategy
  • Projection drift detection
  • Snapshot storage growth
  • Executor trust hardening

Each risk must have a mitigation strategy before Tier 2 rollout.

Design Outcome

This implementation plan:

  • Preserves core invariants
  • Enables infrastructure scaling
  • Prevents semantic leakage
  • Structures phased, verifiable progress
  • Makes the OSS stack a compliant backend implementation