niklas/amduat
1
0
Fork 0
Code Issues Pull requests Actions Packages Projects Releases Wiki Activity
amduat/docs/amduat-oss-backend-proposal.md

8 KiB
Raw Permalink Blame History

Amduat Open-Source Backend Proposal

Contract-First Backend Integration Architecture

Status: Proposal Scope: Define how the OSS stack (MinIO, JanusGraph, Cassandra, Elasticsearch, Spark, Nessie, IPFS, Ollama) can be deployed as a backend implementation of Amduat while preserving semantic invariants.

Related Documents

  • docs/implementation-plan.md
  • docs/phase0-1-execution-checklist.md
  • docs/decision-checklist.md
  • docs/conformance-harness-spec.md
  • docs/adapter-error-taxonomy.md
  • docs/tgk-ordering-enforcement.md
  • docs/indexing-requirements-and-projections.md
  • docs/es-indexing-plan.md
  • docs/projection-checkpoint-schema.md
  • docs/oss-backend-adapter-boundary.md

1. Executive Summary

This document proposes integrating a scalable open-source infrastructure stack as a backend implementation of Amduat.

The proposal preserves Amduat’s defining principle:

The semantic core is invariant. Backends are interchangeable implementations of store/query surfaces.

The OSS stack provides scalability, distribution, analytics, and optional acceleration layers without altering the deterministic semantics defined by the Amduat core specifications.

2. Architectural Position

2.1 Semantic Core

Core semantics are defined by:

  • ASL/1-CORE
  • PEL/1-CORE
  • TGK/1-CORE

Canonical encodings are provided by:

  • ENC/ASL1-CORE (and related encoding profiles)

Identity derivation is defined by:

  • ASL/1-CORE
  • HASH/ASL1

These define:

  • Artifact identity and immutability
  • Core execution semantics
  • Core graph semantics
  • Deterministic projection rules

These are deployment-invariant.

2.2 Store / Query Surfaces

Operational interfaces are defined by:

  • ASL/1-STORE
  • TGK/STORE/1

Optional provenance semantics:

  • TGK/PROV/1

Storage and indexing layers are pluggable implementations of these surfaces. They do not redefine core semantics.

3. No Semantic Drift Rule

To preserve Amduat’s portability and determinism, all backend implementations must adhere to the following:

  1. No Backend-Specific Semantics

    • No interpretation of artifacts beyond canonical encoding.
    • No implicit schema expansion.
    • No hidden type coercion.

  2. No Semantic Elevation of Projections

    • Search indexes (e.g., Elasticsearch)
    • Distribution caches (e.g., IPFS)
    • Analytics views (e.g., Spark)
    • AI-generated outputs

    …are derived projections unless explicitly promoted by governance.

  3. Deterministic Reproducibility

    • The logical TGK graph must be reproducible from ASL state.
    • Snapshot exports must reproduce identical graph projections.

  4. Surface Conformance Only

    • The OSS stack may optimize storage and indexing.
    • It may not redefine core behavior.

  5. Observational Equivalence

    • Any backend-specific optimization must be observationally equivalent to the core semantics and query results defined by the specifications.

This rule guarantees backend substitution without semantic drift.

4. ASL/1-STORE Implementation (Artifact Layer)

4.1 Contract

Must implement:

  • put(Artifact) -> Reference
  • get(Reference) -> Artifact | NotFound

Identity derivation follows:

  1. Canonical encoding (ENC/ASL1-CORE)
  2. Identity derivation rules (ASL/1-CORE)
  3. Hash function (HASH/ASL1)

In abstract form:

Reference = HASH( CANONICAL_ENCODE(Artifact) )

Reference derivation is deterministic and caller-independent.

4.2 Immutability

  • Artifacts are immutable.
  • Rewriting the same Artifact yields the same Reference.
  • Mutation semantics do not exist at this layer.

4.3 Authority Rule

The authoritative store in any deployment is the backend implementation that conforms to ASL/1-STORE.

ASL/1-STORE is a specification; authority resides in whichever concrete backend implements it.

Caches and projections derive from this authoritative ASL implementation unless explicitly promoted by governance.

4.4 Proposed Backend

Primary implementation:

  • MinIO-backed object storage

Optional:

  • IPFS as distribution/cache layer

5. TGK/STORE/1 Implementation (Graph Query Surface)

5.1 Projection Model

The TGK graph is a logical projection derived from:

  • Artifacts stored via ASL
  • Core graph semantics defined in TGK/1-CORE

It is not an independently authoritative database.

Graph state must be reproducible from ASL state.

5.2 Required Operations

Backends must implement the required query operations and error model defined in TGK/STORE/1.

This proposal references the specification directly to avoid semantic drift through re-enumeration.

5.3 Read-Only Nature

  • GraphStoreInstance is logically read-only.
  • Edges arise from writing EdgeArtifacts to ASL.
  • Deletion, compaction, or tombstone policies are backend or profile-level implementation concerns and are not part of TGK/STORE/1.

5.4 Proposed Backend

  • JanusGraph as query engine
  • Cassandra as storage backend
  • Optional Elasticsearch as search projection

JanusGraph functions as an implementation detail of the TGK/STORE/1 surface.

6. TGK/PROV/1 (Optional Provenance Layer)

TGK/PROV/1 defines pure functions over a ProvenanceGraph.

  • It does not require a particular store.
  • It is evaluated against TGK/1-CORE graph semantics.
  • Implementations may use TGK/STORE/1 queries to realize provenance functions.

Provenance is layered above store semantics and does not alter core behavior.

Implementation options:

  • Native traversal realization
  • Snapshot-based traversal in analytics engines
  • Hybrid realization

7. Projection & Acceleration Layers

Unless explicitly promoted by governance:

  • Elasticsearch
  • IPFS
  • Lakehouse datasets (MinIO + Nessie)
  • Spark analytics outputs
  • AI-derived artifacts

…are derived projections.

Authoritative truth resides in:

  • The deployed ASL/1-STORE implementation
  • Deterministic TGK projection derived from ASL state

Projections may be eventually consistent. Audit-grade claims must reference reproducible ASL state or snapshot identifiers.

8. Capability Tiers

Capability tiers define operational envelope, not semantic differences.

Tier 0 — Embedded / Edge

Implements:

  • ASL/1-STORE
  • TGK/STORE/1
  • Optional minimal TGK/PROV/1

Characteristics:

  • Single-node
  • Deterministic local projection
  • Strong local consistency
  • Optional sync/export

Use cases:

  • Personal knowledge systems
  • Offline assistant systems

Tier 1 — Platform / Scale-Out

Implements:

  • ASL/1-STORE
  • TGK/STORE/1
  • Optional TGK/PROV/1

Typical infrastructure:

  • Distributed object store (e.g., MinIO)
  • Distributed graph backend (e.g., JanusGraph over Cassandra)
  • Optional derived search index

Characteristics:

  • Multi-node
  • Horizontal scaling
  • Operational consistency
  • Deterministic core preserved

Use cases:

  • Community platforms
  • Shared knowledge systems

Tier 2 — Enterprise / Audit-Grade

Includes Tier 1 plus:

  • Snapshot export capability
  • Lakehouse projection
  • Versioned snapshots (e.g., Nessie)
  • Governance workflows
  • Promotion pipelines

Characteristics:

  • Snapshot-based reproducibility
  • Audit traceability
  • Multi-environment promotion

Use cases:

  • Enterprise backbone systems
  • Regulated environments

9. Conformance Requirements

Each backend must:

  • Pass ASL/1-STORE conformance tests
  • Pass TGK/STORE/1 conformance and deterministic ordering tests
  • Demonstrate projection reproducibility from ASL state
  • Preserve encoding stability across versions

Tier 2 must additionally:

  • Validate snapshot equivalence
  • Demonstrate replay determinism from ASL state

Conformance is versioned alongside spec revisions.

10. Design Outcome

This architecture ensures:

  • One invariant semantic kernel
  • Backend substitution without semantic drift
  • Scaling through infrastructure choice
  • Deterministic reproducibility
  • Clear separation between authority and projection layers
  • Portability from embedded edge systems to enterprise-scale deployments

The OSS stack is therefore a backend implementation of Amduat — not a semantic authority.