Index Backend Repro (Isolated)

Summary

With a fresh store root, amduatd on --store-backend index accepts the first /v2/graph/nodes write, then fails on the second node write with {"error":"store error"}. The same call sequence succeeds on --store-backend fs.

Observed daemon log on index:

ERROR: edge append failed for space/app1/daemon/edges (err=2)

err=2 maps to AMDUAT_ASL_COLLECTION_ERR_IO.

Minimal Repro

1) Start index backend

STORE_BACKEND=index INDEX_BACKEND_PROBE=0 \
STORE_ROOT=/tmp/amduat-asl-index-iso \
SOCK=/tmp/amduatd-index-iso.sock SPACE=app1 \
./scripts/dev_start_daemon.sh

2) Execute direct API calls

curl --globoff --silent --show-error --unix-socket /tmp/amduatd-index-iso.sock \
  -H 'Content-Type: application/json' -H 'X-Amduat-Space: app1' \
  -X POST --data-binary '{"name":"doca1"}' \
  http://localhost/v2/graph/nodes

curl --globoff --silent --show-error --unix-socket /tmp/amduatd-index-iso.sock \
  -H 'Content-Type: application/json' -H 'X-Amduat-Space: app1' \
  -X POST --data-binary '{"name":"topica1"}' \
  http://localhost/v2/graph/nodes

curl --globoff --silent --show-error --unix-socket /tmp/amduatd-index-iso.sock \
  -H 'Content-Type: application/json' -H 'X-Amduat-Space: app1' \
  -X POST --data-binary '{"subject":"doca1","predicate":"ms.within_domain","object":"topica1","provenance":{"source_uri":"urn:t","extractor":"x","observed_at":1,"ingested_at":2,"trace_id":"t1"}}' \
  http://localhost/v2/graph/edges

Expected: both nodes + edge succeed. Actual on index: second node returns {"error":"store error"}, edge returns {"error":"object not found"}.

3) Control check (fs backend)

Run the same payloads against fs:

STORE_BACKEND=fs STORE_ROOT=/tmp/amduat-asl-fs-iso \
SOCK=/tmp/amduatd-fs-iso.sock SPACE=app1 \
./scripts/dev_start_daemon.sh

All three calls succeed on fs.

Likely Failure Path

From source:

vendor/amduat-api/src/amduatd_concepts.c
- edge append path calls amduat_asl_collection_append(...)
vendor/amduat-api/vendor/amduat/src/core/asl_collection.c
- AMDUAT_ASL_COLLECTION_ERR_IO comes from amduat_asl_log_append(...) failure
vendor/amduat-api/vendor/amduat/src/core/asl_log_store.c
- failure likely in pointer/log append path (pointer_get, store_get, or pointer_cas)

Targeted Upstream Patch Path

Add temporary diagnostic logging in amduat_asl_log_append(...) for:
- pointer_name
- ptr_err from amduat_asl_pointer_get
- store_err from amduat_asl_store_get / amduat_asl_store_put
- cas_err and retry exhaustion path
Re-run the minimal repro above and capture the first non-OK internal return.
Patch the specific failing branch in asl_log_store.c (or lower index store path) rather than adding retries in app code.
Restore/commit missing graph index append scripts expected by CTest:
- vendor/amduat-api/scripts/test_graph_index_append.sh
- vendor/amduat-api/scripts/test_graph_index_append_stress.sh

3 KiB Raw Blame History