225 lines
5.8 KiB
Markdown
225 lines
5.8 KiB
Markdown
NOTE: Superseded by `tier1/tgk-1.md` (TGK/1). Kept for historical context.
|
||
|
||
Absolutely — here’s a **draft for ENC-ASL-TGK-INDEX**, carefully merging ASL artifact indexes and TGK edge indexes while respecting the **separation of concerns** and **snapshot determinism**.
|
||
|
||
This design keeps **ENC-ASL-CORE** and **ENC-TGK-CORE** authoritative, and only merges **index references and acceleration structures**.
|
||
|
||
---
|
||
|
||
# ENC-ASL-TGK-INDEX
|
||
|
||
### Merged On-Disk Index for ASL Artifacts and TGK Edges
|
||
|
||
---
|
||
|
||
## 1. Purpose
|
||
|
||
ENC-ASL-TGK-INDEX defines a **unified on-disk index** that:
|
||
|
||
* References **ASL artifacts** (ENC-ASL-CORE)
|
||
* References **TGK edges** (ENC-TGK-CORE)
|
||
* Supports **routing keys, filters, sharding, SIMD acceleration** per ASL-INDEX-ACCEL
|
||
* Preserves **snapshot safety, log-sequence ordering, and immutability**
|
||
|
||
> Semantic data lives in the respective CORE layers; this index layer **only stores references**.
|
||
|
||
---
|
||
|
||
## 2. Layering Principle
|
||
|
||
| Layer | Responsibility |
|
||
| --------------------- | -------------------------------------------- |
|
||
| ENC-ASL-CORE | Artifact structure and type tags |
|
||
| ENC-TGK-CORE | Edge structure (`from[] → to[]`) |
|
||
| TGK-INDEX / ASL-INDEX | Canonical & routing keys, index semantics |
|
||
| ENC-ASL-TGK-INDEX | On-disk references and acceleration metadata |
|
||
|
||
**Invariant:** This index never re-encodes artifacts or edges.
|
||
|
||
---
|
||
|
||
## 3. Segment Layout
|
||
|
||
Segments are **append-only** and **snapshot-bound**:
|
||
|
||
```
|
||
+-----------------------------+
|
||
| Segment Header |
|
||
+-----------------------------+
|
||
| Routing Filters |
|
||
+-----------------------------+
|
||
| ASL Artifact Index Records |
|
||
+-----------------------------+
|
||
| TGK Edge Index Records |
|
||
+-----------------------------+
|
||
| Optional Acceleration Data |
|
||
+-----------------------------+
|
||
| Segment Footer |
|
||
+-----------------------------+
|
||
```
|
||
|
||
* Segment atomicity enforced
|
||
* Footer checksum guarantees integrity
|
||
|
||
---
|
||
|
||
## 4. Segment Header
|
||
|
||
```c
|
||
struct asl_tgk_index_segment_header {
|
||
uint32_t magic; // 'ATXI'
|
||
uint16_t version;
|
||
uint16_t flags;
|
||
uint64_t segment_id;
|
||
uint64_t logseq_min;
|
||
uint64_t logseq_max;
|
||
uint64_t asl_record_count;
|
||
uint64_t tgk_record_count;
|
||
uint64_t record_area_offset;
|
||
uint64_t footer_offset;
|
||
};
|
||
```
|
||
|
||
* `logseq_*` enforce snapshot visibility
|
||
* Separate counts for ASL and TGK entries
|
||
|
||
---
|
||
|
||
## 5. Routing Filters
|
||
|
||
Filters may be **segmented by type**:
|
||
|
||
* **ASL filters**: artifact hash + type tag
|
||
* **TGK filters**: canonical edge ID + edge type key + optional role
|
||
|
||
```c
|
||
struct asl_tgk_filter_header {
|
||
uint16_t filter_type; // e.g., BLOOM, XOR
|
||
uint16_t version;
|
||
uint32_t flags;
|
||
uint64_t size_bytes; // length of filter payload
|
||
};
|
||
```
|
||
|
||
* Filters are advisory; false positives allowed, false negatives forbidden
|
||
* Must be deterministic per snapshot
|
||
|
||
---
|
||
|
||
## 6. ASL Artifact Index Record
|
||
|
||
```c
|
||
struct asl_index_record {
|
||
uint64_t logseq;
|
||
uint64_t artifact_id; // ENC-ASL-CORE reference
|
||
uint32_t type_tag; // optional
|
||
uint8_t has_type_tag; // 0 or 1
|
||
uint16_t flags; // tombstone, reserved
|
||
};
|
||
```
|
||
|
||
* `artifact_id` = canonical identity
|
||
* No artifact payload here
|
||
|
||
---
|
||
|
||
## 7. TGK Edge Index Record
|
||
|
||
```c
|
||
struct tgk_index_record {
|
||
uint64_t logseq;
|
||
uint64_t tgk_edge_id; // ENC-TGK-CORE reference
|
||
uint32_t edge_type_key; // optional
|
||
uint8_t has_edge_type;
|
||
uint8_t role; // optional from/to/both
|
||
uint16_t flags; // tombstone, reserved
|
||
};
|
||
```
|
||
|
||
* `tgk_edge_id` = canonical TGK-CORE edge ID
|
||
* No node lists stored in index
|
||
|
||
---
|
||
|
||
## 8. Optional Node-Projection Records
|
||
|
||
For acceleration:
|
||
|
||
```c
|
||
struct node_edge_ref {
|
||
uint64_t logseq;
|
||
uint64_t node_id; // from/to node
|
||
uint64_t tgk_edge_id;
|
||
uint8_t position; // from or to
|
||
};
|
||
```
|
||
|
||
* Fully derivable from TGK-CORE edges
|
||
* Optional; purely for lookup speed
|
||
|
||
---
|
||
|
||
## 9. Sharding and SIMD
|
||
|
||
* Shard assignment is **routing key based** (ASL artifact or TGK edge)
|
||
* SIMD arrays may store precomputed routing keys for fast filter evaluation
|
||
* Must follow ASL-INDEX-ACCEL invariants: deterministic, immutable, snapshot-safe
|
||
|
||
---
|
||
|
||
## 10. Snapshot Interaction
|
||
|
||
At snapshot `S`:
|
||
|
||
* Segment visible if `logseq_min ≤ S`
|
||
* ASL or TGK record visible if `logseq ≤ S`
|
||
* Tombstones shadow earlier records
|
||
* Filters may be used as advisory before canonical verification
|
||
|
||
---
|
||
|
||
## 11. Segment Footer
|
||
|
||
```c
|
||
struct asl_tgk_index_segment_footer {
|
||
uint64_t checksum; // covers header, filters, records
|
||
uint64_t asl_record_bytes;
|
||
uint64_t tgk_record_bytes;
|
||
uint64_t filter_bytes;
|
||
};
|
||
```
|
||
|
||
* Ensures atomicity and completeness
|
||
|
||
---
|
||
|
||
## 12. Normative Invariants
|
||
|
||
1. **ASL artifact identity = ENC-ASL-CORE artifact ID**
|
||
2. **TGK edge identity = ENC-TGK-CORE edge ID**
|
||
3. Edge type tag and artifact type tag **do not affect canonical identity**
|
||
4. Filters are advisory only; no false negatives
|
||
5. Sharding is observationally invisible
|
||
6. Index records are immutable once written
|
||
7. Snapshot visibility strictly follows `logseq`
|
||
8. Determinism guaranteed per snapshot
|
||
|
||
---
|
||
|
||
## 13. Summary
|
||
|
||
ENC-ASL-TGK-INDEX merges ASL artifacts and TGK edges into a **single, snapshot-safe, acceleration-friendly index layer**:
|
||
|
||
* Keeps core semantics authoritative
|
||
* Enables high-performance lookups using routing, sharding, SIMD, and filters
|
||
* Preserves immutability and determinism
|
||
* Fully compatible with ASL-INDEX-ACCEL principles
|
||
|
||
This design supports billions of references while avoiding semantic collisions between ASL and TGK layers.
|
||
|
||
---
|
||
|
||
If you want, the next step could be **drafting a unified query execution model** over this merged index, connecting **artifact lookups** and **TGK graph traversals** in a snapshot-safe, deterministic way.
|
||
|
||
Do you want me to do that next?
|