352 lines
8 KiB
Markdown
352 lines
8 KiB
Markdown
# Amduat Open-Source Backend Proposal
|
||
|
||
## Contract-First Backend Integration Architecture
|
||
|
||
**Status:** Proposal
|
||
**Scope:** Define how the OSS stack (MinIO, JanusGraph, Cassandra, Elasticsearch, Spark, Nessie, IPFS, Ollama) can be deployed as a backend implementation of Amduat while preserving semantic invariants.
|
||
|
||
---
|
||
|
||
# Related Documents
|
||
|
||
* `docs/implementation-plan.md`
|
||
* `docs/phase0-1-execution-checklist.md`
|
||
* `docs/decision-checklist.md`
|
||
* `docs/conformance-harness-spec.md`
|
||
* `docs/adapter-error-taxonomy.md`
|
||
* `docs/tgk-ordering-enforcement.md`
|
||
* `docs/indexing-requirements-and-projections.md`
|
||
* `docs/es-indexing-plan.md`
|
||
* `docs/projection-checkpoint-schema.md`
|
||
* `docs/oss-backend-adapter-boundary.md`
|
||
|
||
---
|
||
|
||
# 1. Executive Summary
|
||
|
||
This document proposes integrating a scalable open-source infrastructure stack as a backend implementation of Amduat.
|
||
|
||
The proposal preserves Amduat’s defining principle:
|
||
|
||
> The semantic core is invariant.
|
||
> Backends are interchangeable implementations of store/query surfaces.
|
||
|
||
The OSS stack provides scalability, distribution, analytics, and optional acceleration layers without altering the deterministic semantics defined by the Amduat core specifications.
|
||
|
||
---
|
||
|
||
# 2. Architectural Position
|
||
|
||
## 2.1 Semantic Core
|
||
|
||
Core semantics are defined by:
|
||
|
||
* `ASL/1-CORE`
|
||
* `PEL/1-CORE`
|
||
* `TGK/1-CORE`
|
||
|
||
Canonical encodings are provided by:
|
||
|
||
* `ENC/ASL1-CORE` (and related encoding profiles)
|
||
|
||
Identity derivation is defined by:
|
||
|
||
* `ASL/1-CORE`
|
||
* `HASH/ASL1`
|
||
|
||
These define:
|
||
|
||
* Artifact identity and immutability
|
||
* Core execution semantics
|
||
* Core graph semantics
|
||
* Deterministic projection rules
|
||
|
||
These are deployment-invariant.
|
||
|
||
## 2.2 Store / Query Surfaces
|
||
|
||
Operational interfaces are defined by:
|
||
|
||
* `ASL/1-STORE`
|
||
* `TGK/STORE/1`
|
||
|
||
Optional provenance semantics:
|
||
|
||
* `TGK/PROV/1`
|
||
|
||
Storage and indexing layers are pluggable implementations of these surfaces.
|
||
They do not redefine core semantics.
|
||
|
||
---
|
||
|
||
# 3. No Semantic Drift Rule
|
||
|
||
To preserve Amduat’s portability and determinism, all backend implementations must adhere to the following:
|
||
|
||
1. **No Backend-Specific Semantics**
|
||
|
||
* No interpretation of artifacts beyond canonical encoding.
|
||
* No implicit schema expansion.
|
||
* No hidden type coercion.
|
||
|
||
2. **No Semantic Elevation of Projections**
|
||
|
||
* Search indexes (e.g., Elasticsearch)
|
||
* Distribution caches (e.g., IPFS)
|
||
* Analytics views (e.g., Spark)
|
||
* AI-generated outputs
|
||
|
||
…are derived projections unless explicitly promoted by governance.
|
||
|
||
3. **Deterministic Reproducibility**
|
||
|
||
* The logical TGK graph must be reproducible from ASL state.
|
||
* Snapshot exports must reproduce identical graph projections.
|
||
|
||
4. **Surface Conformance Only**
|
||
|
||
* The OSS stack may optimize storage and indexing.
|
||
* It may not redefine core behavior.
|
||
|
||
5. **Observational Equivalence**
|
||
|
||
* Any backend-specific optimization must be observationally equivalent to the core semantics and query results defined by the specifications.
|
||
|
||
This rule guarantees backend substitution without semantic drift.
|
||
|
||
---
|
||
|
||
# 4. ASL/1-STORE Implementation (Artifact Layer)
|
||
|
||
## 4.1 Contract
|
||
|
||
Must implement:
|
||
|
||
* `put(Artifact) -> Reference`
|
||
* `get(Reference) -> Artifact | NotFound`
|
||
|
||
Identity derivation follows:
|
||
|
||
1. Canonical encoding (`ENC/ASL1-CORE`)
|
||
2. Identity derivation rules (`ASL/1-CORE`)
|
||
3. Hash function (`HASH/ASL1`)
|
||
|
||
In abstract form:
|
||
|
||
```
|
||
Reference = HASH( CANONICAL_ENCODE(Artifact) )
|
||
```
|
||
|
||
Reference derivation is deterministic and caller-independent.
|
||
|
||
## 4.2 Immutability
|
||
|
||
* Artifacts are immutable.
|
||
* Rewriting the same Artifact yields the same Reference.
|
||
* Mutation semantics do not exist at this layer.
|
||
|
||
## 4.3 Authority Rule
|
||
|
||
The authoritative store in any deployment is the backend implementation that conforms to `ASL/1-STORE`.
|
||
|
||
`ASL/1-STORE` is a specification; authority resides in whichever concrete backend implements it.
|
||
|
||
Caches and projections derive from this authoritative ASL implementation unless explicitly promoted by governance.
|
||
|
||
## 4.4 Proposed Backend
|
||
|
||
Primary implementation:
|
||
|
||
* MinIO-backed object storage
|
||
|
||
Optional:
|
||
|
||
* IPFS as distribution/cache layer
|
||
|
||
---
|
||
|
||
# 5. TGK/STORE/1 Implementation (Graph Query Surface)
|
||
|
||
## 5.1 Projection Model
|
||
|
||
The TGK graph is a logical projection derived from:
|
||
|
||
* Artifacts stored via ASL
|
||
* Core graph semantics defined in `TGK/1-CORE`
|
||
|
||
It is not an independently authoritative database.
|
||
|
||
Graph state must be reproducible from ASL state.
|
||
|
||
## 5.2 Required Operations
|
||
|
||
Backends must implement the required query operations and error model defined in `TGK/STORE/1`.
|
||
|
||
This proposal references the specification directly to avoid semantic drift through re-enumeration.
|
||
|
||
## 5.3 Read-Only Nature
|
||
|
||
* `GraphStoreInstance` is logically read-only.
|
||
* Edges arise from writing EdgeArtifacts to ASL.
|
||
* Deletion, compaction, or tombstone policies are backend or profile-level implementation concerns and are not part of `TGK/STORE/1`.
|
||
|
||
## 5.4 Proposed Backend
|
||
|
||
* JanusGraph as query engine
|
||
* Cassandra as storage backend
|
||
* Optional Elasticsearch as search projection
|
||
|
||
JanusGraph functions as an implementation detail of the `TGK/STORE/1` surface.
|
||
|
||
---
|
||
|
||
# 6. TGK/PROV/1 (Optional Provenance Layer)
|
||
|
||
`TGK/PROV/1` defines pure functions over a `ProvenanceGraph`.
|
||
|
||
* It does not require a particular store.
|
||
* It is evaluated against `TGK/1-CORE` graph semantics.
|
||
* Implementations may use `TGK/STORE/1` queries to realize provenance functions.
|
||
|
||
Provenance is layered above store semantics and does not alter core behavior.
|
||
|
||
Implementation options:
|
||
|
||
* Native traversal realization
|
||
* Snapshot-based traversal in analytics engines
|
||
* Hybrid realization
|
||
|
||
---
|
||
|
||
# 7. Projection & Acceleration Layers
|
||
|
||
Unless explicitly promoted by governance:
|
||
|
||
* Elasticsearch
|
||
* IPFS
|
||
* Lakehouse datasets (MinIO + Nessie)
|
||
* Spark analytics outputs
|
||
* AI-derived artifacts
|
||
|
||
…are derived projections.
|
||
|
||
Authoritative truth resides in:
|
||
|
||
* The deployed `ASL/1-STORE` implementation
|
||
* Deterministic TGK projection derived from ASL state
|
||
|
||
Projections may be eventually consistent.
|
||
Audit-grade claims must reference reproducible ASL state or snapshot identifiers.
|
||
|
||
---
|
||
|
||
# 8. Capability Tiers
|
||
|
||
Capability tiers define operational envelope, not semantic differences.
|
||
|
||
---
|
||
|
||
## Tier 0 — Embedded / Edge
|
||
|
||
Implements:
|
||
|
||
* `ASL/1-STORE`
|
||
* `TGK/STORE/1`
|
||
* Optional minimal `TGK/PROV/1`
|
||
|
||
Characteristics:
|
||
|
||
* Single-node
|
||
* Deterministic local projection
|
||
* Strong local consistency
|
||
* Optional sync/export
|
||
|
||
Use cases:
|
||
|
||
* Personal knowledge systems
|
||
* Offline assistant systems
|
||
|
||
---
|
||
|
||
## Tier 1 — Platform / Scale-Out
|
||
|
||
Implements:
|
||
|
||
* `ASL/1-STORE`
|
||
* `TGK/STORE/1`
|
||
* Optional `TGK/PROV/1`
|
||
|
||
Typical infrastructure:
|
||
|
||
* Distributed object store (e.g., MinIO)
|
||
* Distributed graph backend (e.g., JanusGraph over Cassandra)
|
||
* Optional derived search index
|
||
|
||
Characteristics:
|
||
|
||
* Multi-node
|
||
* Horizontal scaling
|
||
* Operational consistency
|
||
* Deterministic core preserved
|
||
|
||
Use cases:
|
||
|
||
* Community platforms
|
||
* Shared knowledge systems
|
||
|
||
---
|
||
|
||
## Tier 2 — Enterprise / Audit-Grade
|
||
|
||
Includes Tier 1 plus:
|
||
|
||
* Snapshot export capability
|
||
* Lakehouse projection
|
||
* Versioned snapshots (e.g., Nessie)
|
||
* Governance workflows
|
||
* Promotion pipelines
|
||
|
||
Characteristics:
|
||
|
||
* Snapshot-based reproducibility
|
||
* Audit traceability
|
||
* Multi-environment promotion
|
||
|
||
Use cases:
|
||
|
||
* Enterprise backbone systems
|
||
* Regulated environments
|
||
|
||
---
|
||
|
||
# 9. Conformance Requirements
|
||
|
||
Each backend must:
|
||
|
||
* Pass `ASL/1-STORE` conformance tests
|
||
* Pass `TGK/STORE/1` conformance and deterministic ordering tests
|
||
* Demonstrate projection reproducibility from ASL state
|
||
* Preserve encoding stability across versions
|
||
|
||
Tier 2 must additionally:
|
||
|
||
* Validate snapshot equivalence
|
||
* Demonstrate replay determinism from ASL state
|
||
|
||
Conformance is versioned alongside spec revisions.
|
||
|
||
---
|
||
|
||
# 10. Design Outcome
|
||
|
||
This architecture ensures:
|
||
|
||
* One invariant semantic kernel
|
||
* Backend substitution without semantic drift
|
||
* Scaling through infrastructure choice
|
||
* Deterministic reproducibility
|
||
* Clear separation between authority and projection layers
|
||
* Portability from embedded edge systems to enterprise-scale deployments
|
||
|
||
The OSS stack is therefore a backend implementation of Amduat — not a semantic authority.
|