amduat-api/notes/on-disk layout.md

Good, this is the right moment to pin this down, because on-disk shape is where abstraction leaks if we’re sloppy.

I’ll answer in layers:

1. What ASL actually requires on disk (minimal contract)
2. A concrete, sane on-disk layout that fits what you already have
3. How shards, blocks, artifacts, logs, and snapshots map
4. How this scales from “single laptop” to “Common”
5. Why this does not over-spec ASL-CORE

I’ll stay very explicit about what is normative vs conventional.

---

1. What ASL actually requires on disk (the hard minimum)

ASL itself does not care about directories.

ASL-STORE / ASL-STORE-INDEX require only that the store can provide:

1. Immutable blocks
2. An append-only log
3. Sealed snapshots
4. Deterministic replay
5. CAS addressing

So the only hard requirements are:

• Blocks are immutable after seal
• Index entries point to (block_id, offset, length)
• Logs are append-only and ordered
• Snapshots pin a prefix of state

Everything else is an implementation choice.

That means: 👉 your current “store root with shards” is already compatible.

---

2. Recommended on-disk layout (practical, explicit)

Here is a layout that is:

• compatible with your current shard idea
• works on POSIX
• maps cleanly to ZFS later
• easy to reason about during rescue / forensics

Store root (per domain)

/asl/
└── domains/
    └── <domain-id>/
        ├── meta/
        ├── blocks/
        ├── index/
        ├── log/
        ├── snapshots/
        └── tmp/

Everything below is domain-local.

---

3. Blocks (the real storage substrate)

3.1 Block storage (immutable)

blocks/
├── open/
│   └── blk_<uuid>.tmp
└── sealed/
    ├── 00/
    │   └── <blockid>.blk
    ├── 01/
    │   └── <blockid>.blk
    └── ff/
        └── <blockid>.blk

• blockid = CAS hash
• Sharded by prefix (first byte or two)
• Open blocks are never visible
• Sealed blocks are immutable

This directly matches your block + offset mental model.

Important: artifacts do not live as files They live inside blocks.

---

4. Index (where artifacts become visible)

Your shard-based approach fits perfectly here.

index/
├── shard-000/
│   ├── segment-0001.idx
│   ├── segment-0002.idx
│   └── bloom.bin
├── shard-001/
│   └── ...
└── shard-fff/

Each segment-XXXX.idx is:

• append-only while open
• immutable once sealed
• referenced by the log

Index records point to:

ArtifactKey → (BlockID, offset, length)

This is exactly ASL-STORE-INDEX.

---

5. Append-only log (the spine of truth)

log/
├── log-0000000000000000.asl
├── log-0000000000001000.asl
└── current

Log records include:

• index additions
• tombstones
• segment seals
• DAM updates
• witness rotation artifacts

Rules:

• Logs are strictly ordered
• Never rewritten
• Replayable from snapshot

---

6. Snapshots (checkpoints, not magic)

snapshots/
├── snapshot-00000123/
│   ├── manifest.yaml
│   ├── pinned-segments.txt
│   └── pinned-blocks.txt
└── snapshot-00000124/

A snapshot is:

• a commit point
• a pin-set (segments + blocks)
• a replay anchor

ZFS snapshots fit perfectly here but are not required.

---

7. Meta (authority, policy, identity)

meta/
├── dam/
│   └── dam-current.art
├── policy/
│   └── policy-current.art
├── keys/
│   ├── pub/
│   └── priv/
└── host/
    └── host-id

Important rule:

Meta artifacts are still artifacts

They must:

• be sealed
• be indexed
• appear in the log

The files here are cached materializations.

---

8. Tmp (explicitly unsafe)

tmp/
├── ingest/
├── scratch/
└── pty/

This is where:

• asl-capture writes streams
• open blocks live
• crashes are allowed

Nothing here is trusted.

---

9. How shards fit (your current design)

You said:

“I have a store root with shards with artifacts in them.”

That maps cleanly if you reinterpret it as:

• shards own index segments
• artifacts are logical entries, not files
• shard locality = lookup acceleration

So instead of:

shard/
└── artifact-file

You have:

shard/
└── index segments → blocks

This is the key mental shift you already sensed.

---

10. Common vs Personal vs Group (on disk)

They look identical on disk.

Differences live in:

• DAM
• policy
• witness artifacts

Common

• quorum witness
• stricter validation
• often read-only on most hosts

Personal

• self-authority
• multiple active writers allowed
• same block/index/log structure

Group

• hybrid policies

No special filesystem layout needed.

---

11. Why this is the right abstraction boundary

You said:

“I used to look at the data as artifacts. Now it seems I’m better off looking at data as blocks with artifacts in them.”

That’s exactly right.

Artifacts are semantic Blocks are physical

This layout lets you:

• reason in artifacts
• store in blocks
• snapshot blocks
• index artifacts
• federate semantics

without mixing concerns.

---

12. One sentence to lock it in

Artifacts are indexed meanings; blocks are frozen bytes; snapshots pin the relationship.

---

13. What you can safely implement now

Even before ASL-CORE / PEL are finished, you can implement:

• block store
• block sealing
• index segments
• append-only log
• snapshot pinning
• DAM storage and validation

All of that is infrastructure, not policy.

---

If you want next, I strongly suggest one of these:

• walk through one artifact write end-to-end on disk
• show how asl-capture writes into blocks
• show how a personal domain syncs across two machines
• map this layout directly onto ZFS datasets

You’re no longer at risk of mixing projects — you’re converging.