352 lines
8 KiB
Markdown
352 lines
8 KiB
Markdown
|
# Amduat Open-Source Backend Proposal
|
|||
|
|
|
|||
|
|
## Contract-First Backend Integration Architecture
|
|||
|
|
|
|||
|
|
**Status:** Proposal
|
|||
|
|
**Scope:** Define how the OSS stack (MinIO, JanusGraph, Cassandra, Elasticsearch, Spark, Nessie, IPFS, Ollama) can be deployed as a backend implementation of Amduat while preserving semantic invariants.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# Related Documents
|
|||
|
|
|
|||
|
|
* `docs/implementation-plan.md`
|
|||
|
|
* `docs/phase0-1-execution-checklist.md`
|
|||
|
|
* `docs/decision-checklist.md`
|
|||
|
|
* `docs/conformance-harness-spec.md`
|
|||
|
|
* `docs/adapter-error-taxonomy.md`
|
|||
|
|
* `docs/tgk-ordering-enforcement.md`
|
|||
|
|
* `docs/indexing-requirements-and-projections.md`
|
|||
|
|
* `docs/es-indexing-plan.md`
|
|||
|
|
* `docs/projection-checkpoint-schema.md`
|
|||
|
|
* `docs/oss-backend-adapter-boundary.md`
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# 1. Executive Summary
|
|||
|
|
|
|||
|
|
This document proposes integrating a scalable open-source infrastructure stack as a backend implementation of Amduat.
|
|||
|
|
|
|||
|
|
The proposal preserves Amduat’s defining principle:
|
|||
|
|
|
|||
|
|
> The semantic core is invariant.
|
|||
|
|
> Backends are interchangeable implementations of store/query surfaces.
|
|||
|
|
|
|||
|
|
The OSS stack provides scalability, distribution, analytics, and optional acceleration layers without altering the deterministic semantics defined by the Amduat core specifications.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# 2. Architectural Position
|
|||
|
|
|
|||
|
|
## 2.1 Semantic Core
|
|||
|
|
|
|||
|
|
Core semantics are defined by:
|
|||
|
|
|
|||
|
|
* `ASL/1-CORE`
|
|||
|
|
* `PEL/1-CORE`
|
|||
|
|
* `TGK/1-CORE`
|
|||
|
|
|
|||
|
|
Canonical encodings are provided by:
|
|||
|
|
|
|||
|
|
* `ENC/ASL1-CORE` (and related encoding profiles)
|
|||
|
|
|
|||
|
|
Identity derivation is defined by:
|
|||
|
|
|
|||
|
|
* `ASL/1-CORE`
|
|||
|
|
* `HASH/ASL1`
|
|||
|
|
|
|||
|
|
These define:
|
|||
|
|
|
|||
|
|
* Artifact identity and immutability
|
|||
|
|
* Core execution semantics
|
|||
|
|
* Core graph semantics
|
|||
|
|
* Deterministic projection rules
|
|||
|
|
|
|||
|
|
These are deployment-invariant.
|
|||
|
|
|
|||
|
|
## 2.2 Store / Query Surfaces
|
|||
|
|
|
|||
|
|
Operational interfaces are defined by:
|
|||
|
|
|
|||
|
|
* `ASL/1-STORE`
|
|||
|
|
* `TGK/STORE/1`
|
|||
|
|
|
|||
|
|
Optional provenance semantics:
|
|||
|
|
|
|||
|
|
* `TGK/PROV/1`
|
|||
|
|
|
|||
|
|
Storage and indexing layers are pluggable implementations of these surfaces.
|
|||
|
|
They do not redefine core semantics.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# 3. No Semantic Drift Rule
|
|||
|
|
|
|||
|
|
To preserve Amduat’s portability and determinism, all backend implementations must adhere to the following:
|
|||
|
|
|
|||
|
|
1. **No Backend-Specific Semantics**
|
|||
|
|
|
|||
|
|
* No interpretation of artifacts beyond canonical encoding.
|
|||
|
|
* No implicit schema expansion.
|
|||
|
|
* No hidden type coercion.
|
|||
|
|
|
|||
|
|
2. **No Semantic Elevation of Projections**
|
|||
|
|
|
|||
|
|
* Search indexes (e.g., Elasticsearch)
|
|||
|
|
* Distribution caches (e.g., IPFS)
|
|||
|
|
* Analytics views (e.g., Spark)
|
|||
|
|
* AI-generated outputs
|
|||
|
|
|
|||
|
|
…are derived projections unless explicitly promoted by governance.
|
|||
|
|
|
|||
|
|
3. **Deterministic Reproducibility**
|
|||
|
|
|
|||
|
|
* The logical TGK graph must be reproducible from ASL state.
|
|||
|
|
* Snapshot exports must reproduce identical graph projections.
|
|||
|
|
|
|||
|
|
4. **Surface Conformance Only**
|
|||
|
|
|
|||
|
|
* The OSS stack may optimize storage and indexing.
|
|||
|
|
* It may not redefine core behavior.
|
|||
|
|
|
|||
|
|
5. **Observational Equivalence**
|
|||
|
|
|
|||
|
|
* Any backend-specific optimization must be observationally equivalent to the core semantics and query results defined by the specifications.
|
|||
|
|
|
|||
|
|
This rule guarantees backend substitution without semantic drift.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# 4. ASL/1-STORE Implementation (Artifact Layer)
|
|||
|
|
|
|||
|
|
## 4.1 Contract
|
|||
|
|
|
|||
|
|
Must implement:
|
|||
|
|
|
|||
|
|
* `put(Artifact) -> Reference`
|
|||
|
|
* `get(Reference) -> Artifact | NotFound`
|
|||
|
|
|
|||
|
|
Identity derivation follows:
|
|||
|
|
|
|||
|
|
1. Canonical encoding (`ENC/ASL1-CORE`)
|
|||
|
|
2. Identity derivation rules (`ASL/1-CORE`)
|
|||
|
|
3. Hash function (`HASH/ASL1`)
|
|||
|
|
|
|||
|
|
In abstract form:
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
Reference = HASH( CANONICAL_ENCODE(Artifact) )
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Reference derivation is deterministic and caller-independent.
|
|||
|
|
|
|||
|
|
## 4.2 Immutability
|
|||
|
|
|
|||
|
|
* Artifacts are immutable.
|
|||
|
|
* Rewriting the same Artifact yields the same Reference.
|
|||
|
|
* Mutation semantics do not exist at this layer.
|
|||
|
|
|
|||
|
|
## 4.3 Authority Rule
|
|||
|
|
|
|||
|
|
The authoritative store in any deployment is the backend implementation that conforms to `ASL/1-STORE`.
|
|||
|
|
|
|||
|
|
`ASL/1-STORE` is a specification; authority resides in whichever concrete backend implements it.
|
|||
|
|
|
|||
|
|
Caches and projections derive from this authoritative ASL implementation unless explicitly promoted by governance.
|
|||
|
|
|
|||
|
|
## 4.4 Proposed Backend
|
|||
|
|
|
|||
|
|
Primary implementation:
|
|||
|
|
|
|||
|
|
* MinIO-backed object storage
|
|||
|
|
|
|||
|
|
Optional:
|
|||
|
|
|
|||
|
|
* IPFS as distribution/cache layer
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# 5. TGK/STORE/1 Implementation (Graph Query Surface)
|
|||
|
|
|
|||
|
|
## 5.1 Projection Model
|
|||
|
|
|
|||
|
|
The TGK graph is a logical projection derived from:
|
|||
|
|
|
|||
|
|
* Artifacts stored via ASL
|
|||
|
|
* Core graph semantics defined in `TGK/1-CORE`
|
|||
|
|
|
|||
|
|
It is not an independently authoritative database.
|
|||
|
|
|
|||
|
|
Graph state must be reproducible from ASL state.
|
|||
|
|
|
|||
|
|
## 5.2 Required Operations
|
|||
|
|
|
|||
|
|
Backends must implement the required query operations and error model defined in `TGK/STORE/1`.
|
|||
|
|
|
|||
|
|
This proposal references the specification directly to avoid semantic drift through re-enumeration.
|
|||
|
|
|
|||
|
|
## 5.3 Read-Only Nature
|
|||
|
|
|
|||
|
|
* `GraphStoreInstance` is logically read-only.
|
|||
|
|
* Edges arise from writing EdgeArtifacts to ASL.
|
|||
|
|
* Deletion, compaction, or tombstone policies are backend or profile-level implementation concerns and are not part of `TGK/STORE/1`.
|
|||
|
|
|
|||
|
|
## 5.4 Proposed Backend
|
|||
|
|
|
|||
|
|
* JanusGraph as query engine
|
|||
|
|
* Cassandra as storage backend
|
|||
|
|
* Optional Elasticsearch as search projection
|
|||
|
|
|
|||
|
|
JanusGraph functions as an implementation detail of the `TGK/STORE/1` surface.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# 6. TGK/PROV/1 (Optional Provenance Layer)
|
|||
|
|
|
|||
|
|
`TGK/PROV/1` defines pure functions over a `ProvenanceGraph`.
|
|||
|
|
|
|||
|
|
* It does not require a particular store.
|
|||
|
|
* It is evaluated against `TGK/1-CORE` graph semantics.
|
|||
|
|
* Implementations may use `TGK/STORE/1` queries to realize provenance functions.
|
|||
|
|
|
|||
|
|
Provenance is layered above store semantics and does not alter core behavior.
|
|||
|
|
|
|||
|
|
Implementation options:
|
|||
|
|
|
|||
|
|
* Native traversal realization
|
|||
|
|
* Snapshot-based traversal in analytics engines
|
|||
|
|
* Hybrid realization
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# 7. Projection & Acceleration Layers
|
|||
|
|
|
|||
|
|
Unless explicitly promoted by governance:
|
|||
|
|
|
|||
|
|
* Elasticsearch
|
|||
|
|
* IPFS
|
|||
|
|
* Lakehouse datasets (MinIO + Nessie)
|
|||
|
|
* Spark analytics outputs
|
|||
|
|
* AI-derived artifacts
|
|||
|
|
|
|||
|
|
…are derived projections.
|
|||
|
|
|
|||
|
|
Authoritative truth resides in:
|
|||
|
|
|
|||
|
|
* The deployed `ASL/1-STORE` implementation
|
|||
|
|
* Deterministic TGK projection derived from ASL state
|
|||
|
|
|
|||
|
|
Projections may be eventually consistent.
|
|||
|
|
Audit-grade claims must reference reproducible ASL state or snapshot identifiers.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# 8. Capability Tiers
|
|||
|
|
|
|||
|
|
Capability tiers define operational envelope, not semantic differences.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Tier 0 — Embedded / Edge
|
|||
|
|
|
|||
|
|
Implements:
|
|||
|
|
|
|||
|
|
* `ASL/1-STORE`
|
|||
|
|
* `TGK/STORE/1`
|
|||
|
|
* Optional minimal `TGK/PROV/1`
|
|||
|
|
|
|||
|
|
Characteristics:
|
|||
|
|
|
|||
|
|
* Single-node
|
|||
|
|
* Deterministic local projection
|
|||
|
|
* Strong local consistency
|
|||
|
|
* Optional sync/export
|
|||
|
|
|
|||
|
|
Use cases:
|
|||
|
|
|
|||
|
|
* Personal knowledge systems
|
|||
|
|
* Offline assistant systems
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Tier 1 — Platform / Scale-Out
|
|||
|
|
|
|||
|
|
Implements:
|
|||
|
|
|
|||
|
|
* `ASL/1-STORE`
|
|||
|
|
* `TGK/STORE/1`
|
|||
|
|
* Optional `TGK/PROV/1`
|
|||
|
|
|
|||
|
|
Typical infrastructure:
|
|||
|
|
|
|||
|
|
* Distributed object store (e.g., MinIO)
|
|||
|
|
* Distributed graph backend (e.g., JanusGraph over Cassandra)
|
|||
|
|
* Optional derived search index
|
|||
|
|
|
|||
|
|
Characteristics:
|
|||
|
|
|
|||
|
|
* Multi-node
|
|||
|
|
* Horizontal scaling
|
|||
|
|
* Operational consistency
|
|||
|
|
* Deterministic core preserved
|
|||
|
|
|
|||
|
|
Use cases:
|
|||
|
|
|
|||
|
|
* Community platforms
|
|||
|
|
* Shared knowledge systems
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Tier 2 — Enterprise / Audit-Grade
|
|||
|
|
|
|||
|
|
Includes Tier 1 plus:
|
|||
|
|
|
|||
|
|
* Snapshot export capability
|
|||
|
|
* Lakehouse projection
|
|||
|
|
* Versioned snapshots (e.g., Nessie)
|
|||
|
|
* Governance workflows
|
|||
|
|
* Promotion pipelines
|
|||
|
|
|
|||
|
|
Characteristics:
|
|||
|
|
|
|||
|
|
* Snapshot-based reproducibility
|
|||
|
|
* Audit traceability
|
|||
|
|
* Multi-environment promotion
|
|||
|
|
|
|||
|
|
Use cases:
|
|||
|
|
|
|||
|
|
* Enterprise backbone systems
|
|||
|
|
* Regulated environments
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# 9. Conformance Requirements
|
|||
|
|
|
|||
|
|
Each backend must:
|
|||
|
|
|
|||
|
|
* Pass `ASL/1-STORE` conformance tests
|
|||
|
|
* Pass `TGK/STORE/1` conformance and deterministic ordering tests
|
|||
|
|
* Demonstrate projection reproducibility from ASL state
|
|||
|
|
* Preserve encoding stability across versions
|
|||
|
|
|
|||
|
|
Tier 2 must additionally:
|
|||
|
|
|
|||
|
|
* Validate snapshot equivalence
|
|||
|
|
* Demonstrate replay determinism from ASL state
|
|||
|
|
|
|||
|
|
Conformance is versioned alongside spec revisions.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# 10. Design Outcome
|
|||
|
|
|
|||
|
|
This architecture ensures:
|
|||
|
|
|
|||
|
|
* One invariant semantic kernel
|
|||
|
|
* Backend substitution without semantic drift
|
|||
|
|
* Scaling through infrastructure choice
|
|||
|
|
* Deterministic reproducibility
|
|||
|
|
* Clear separation between authority and projection layers
|
|||
|
|
* Portability from embedded edge systems to enterprise-scale deployments
|
|||
|
|
|
|||
|
|
The OSS stack is therefore a backend implementation of Amduat — not a semantic authority.
|