203 lines
4.9 KiB
Markdown
203 lines
4.9 KiB
Markdown
Absolutely — let’s lay down **ENC-ASL-LOG v0.1**, a minimal deterministic encoding for the append-only ASL log. I’ll aim for something that is **platform-independent, reproducible, and maps cleanly to your index and artifact workflow**.
|
||
|
||
---
|
||
|
||
# ENC-ASL-LOG v0.1
|
||
|
||
## 1. Purpose
|
||
|
||
This document specifies the **exact byte-level encoding** of the ASL append-only log.
|
||
|
||
It is intended for:
|
||
|
||
* C libraries
|
||
* CLI tools
|
||
* Memory-mapped readers
|
||
* Interchange between hosts
|
||
|
||
It defines **encoding**, **record structure**, and **persistence semantics**, but **not log semantics** — see ASL-STORE and ASL-CORE for ordering, snapshot, and visibility rules.
|
||
|
||
---
|
||
|
||
## 2. Encoding Principles
|
||
|
||
1. **Little-endian integers** (multi-byte)
|
||
2. **Packed structures**: no compiler padding
|
||
3. **Forward-compatible versioning** via `header.version`
|
||
4. **Checksums** for corruption detection
|
||
5. **Deterministic serialization**: same log content → same byte sequence
|
||
|
||
---
|
||
|
||
## 3. Log File Layout
|
||
|
||
```
|
||
+----------------+
|
||
| LogHeader |
|
||
+----------------+
|
||
| LogRecord[ ] |
|
||
+----------------+
|
||
| LogFooter |
|
||
+----------------+
|
||
```
|
||
|
||
* **LogHeader**: fixed-size, mandatory, begins file
|
||
* **LogRecord[]**: append-only entries, variable number
|
||
* **LogFooter**: optional, contains global checksum
|
||
|
||
---
|
||
|
||
## 4. LogHeader
|
||
|
||
```c
|
||
#pragma pack(push,1)
|
||
typedef struct {
|
||
uint64_t magic; // Unique magic for ASL log
|
||
uint16_t version; // Encoding version
|
||
uint16_t flags; // Reserved
|
||
uint32_t header_size; // Total header bytes including this struct
|
||
uint64_t first_snapshot; // First snapshot referenced
|
||
uint64_t last_snapshot; // Last snapshot referenced
|
||
} LogHeader;
|
||
#pragma pack(pop)
|
||
```
|
||
|
||
Notes:
|
||
|
||
* `magic` ensures correct file type
|
||
* `version` allows forward compatibility
|
||
* `first_snapshot` and `last_snapshot` help range validation
|
||
|
||
---
|
||
|
||
## 5. LogRecord
|
||
|
||
```c
|
||
#pragma pack(push,1)
|
||
typedef enum {
|
||
LOG_RECORD_ADD_INDEX_SEGMENT = 1,
|
||
LOG_RECORD_SEAL_SEGMENT = 2,
|
||
LOG_RECORD_TOMBSTONE = 3,
|
||
LOG_RECORD_CUSTOM = 0x1000
|
||
} LogRecordType;
|
||
|
||
typedef struct {
|
||
uint64_t record_id; // Unique log entry ID
|
||
uint64_t timestamp_ns; // Monotonic time of record creation
|
||
uint64_t snapshot_id; // Snapshot for which entry applies
|
||
uint32_t type; // LogRecordType
|
||
uint32_t payload_size; // Size of payload bytes following this header
|
||
} LogRecordHeader;
|
||
#pragma pack(pop)
|
||
```
|
||
|
||
* Immediately after `LogRecordHeader`, `payload_size` bytes of **payload** follow.
|
||
* Payload encodes type-specific information.
|
||
|
||
### 5.1 Payload Examples
|
||
|
||
**Add Index Segment**
|
||
|
||
```
|
||
struct {
|
||
uint64_t segment_file_id; // Corresponding ENC-ASL-CORE-INDEX segment
|
||
uint64_t entry_count; // Number of index entries
|
||
uint8_t reserved[16]; // Future fields
|
||
};
|
||
```
|
||
|
||
**Seal Segment**
|
||
|
||
```
|
||
struct {
|
||
uint64_t segment_file_id;
|
||
uint64_t seal_snapshot_id;
|
||
uint64_t seal_time_ns;
|
||
};
|
||
```
|
||
|
||
**Tombstone**
|
||
|
||
```
|
||
struct {
|
||
uint64_t artifact_hash_hi;
|
||
uint64_t artifact_hash_mid;
|
||
uint64_t artifact_hash_lo;
|
||
uint32_t artifact_hash_tail;
|
||
uint32_t flags; // reserved
|
||
};
|
||
```
|
||
|
||
**Custom**
|
||
|
||
```
|
||
Arbitrary byte sequence, self-describing.
|
||
```
|
||
|
||
---
|
||
|
||
## 6. LogFooter
|
||
|
||
```c
|
||
#pragma pack(push,1)
|
||
typedef struct {
|
||
uint64_t crc64; // CRC over entire log (excluding footer)
|
||
uint64_t record_count;// Total records in log
|
||
} LogFooter;
|
||
#pragma pack(pop)
|
||
```
|
||
|
||
---
|
||
|
||
## 7. Record Ordering & Durability
|
||
|
||
* Entries are **strictly append-only**.
|
||
* `record_id` must be strictly increasing.
|
||
* `snapshot_id` reflects the visibility snapshot.
|
||
* Readers **replay records in order** to derive CURRENT state.
|
||
|
||
---
|
||
|
||
## 8. Checksums & Integrity
|
||
|
||
* All multi-byte fields are little-endian.
|
||
* `LogFooter.crc64` is optional but recommended.
|
||
* Per-record integrity may optionally include a **record CRC** as first 8 bytes of payload.
|
||
|
||
---
|
||
|
||
## 9. Versioning
|
||
|
||
* `LogHeader.version` = 0x01
|
||
* New fields must not break existing layout.
|
||
* Unsupported versions must be rejected by readers.
|
||
|
||
---
|
||
|
||
## 10. Relationship to Other Layers
|
||
|
||
| Layer | Responsibility |
|
||
| ------------------ | ----------------------------------------------- |
|
||
| ASL-STORE | Lifecycle, visibility, snapshot enforcement |
|
||
| ASL-CORE-INDEX | Index semantics (ArtifactKey → BlockID mapping) |
|
||
| ENC-ASL-CORE-INDEX | Exact byte layout for index segments |
|
||
| ENC-ASL-LOG | Exact byte layout for append-only log |
|
||
|
||
---
|
||
|
||
✅ This gives you a **minimal, deterministic log encoding**.
|
||
|
||
You could now use it to:
|
||
|
||
1. Append a new log record for an index segment
|
||
2. Seal segments
|
||
3. Mark tombstones
|
||
4. Replay in order to reconstruct CURRENT
|
||
|
||
---
|
||
|
||
If you want, I can **draft a minimal C writer/reader API** next that directly reads/writes this log, including helpers for `add_index_segment()`, `seal_segment()`, and `tombstone()` with payload serialization.
|
||
|
||
Do you want me to do that next?
|
||
|