feat: git locks doctor, a read-only invariant check of the store (0.5.0)

CHANGELOG.md

@@ -4,6 +4,12 @@ All notable changes to this project are recorded here. The format follows Keep a
 
 ## [Unreleased]
 
+## [0.5.0] - 2026-09-16
+
+### Added
+
+- `git locks doctor`: a read-only invariant check of the store (#19). One `finding` line per broken invariant as it is found, then one `doctor` line with the store, the reading basis (refs and records in the one snapshot it read, and the clock), the checks run, the count and the verdict. The invariants: every job record decodes and names its own job; every path a record lists has a path ref pointing at that record; every path ref points at a record some job ref points at, and that record lists the path; every child's parent exists, is live and has the same holder, and no parent chain cycles; every semaphore has meta and gen, its records decode, and its live slots fit its capacity. Exit 0 healthy, 1 with findings, 2 when the store cannot be read, which is never reported healthy. Paths are hashed in one `hash-object` process, so the process count does not grow with the store. Nothing is repaired: diagnosis is the whole command.
+
 ## [0.4.0] - 2026-09-16
 
 ### Changed

README.md

@@ -353,6 +353,8 @@ An outside review of 0.2.1 found the guarantees running ahead of the implementat
 
 **What a failed read is.** An error, never a free path. If `for-each-ref` or `cat-file` fails, or an object does not parse, the command exits 2 with `{"event":"error","reason":"store-read"}` and reports nothing as free or held.
 
+**What the invariants are, and how to see them hold.** `git locks doctor` reads one snapshot and checks it, writing nothing: every job record decodes and names its own job; every path a record lists has a path ref pointing at that record; every path ref points at a record some job ref points at, and that record lists the path; every child's parent exists, is live and has the same holder, and no parent chain cycles; every semaphore has its meta and gen refs, its records decode, and its live slots fit its capacity. Each broken invariant is one `finding` line as it is found, and the last line states the basis it was checked against, the refs and records of that one snapshot and the clock, so a clean report says what was clean. An unreadable store is an error, never healthy. Repair is not a mode of this command; when a finding needs a hand, the fix is a `release`, a `sweep`, or an explicit `update-ref` on the store by someone who has read the finding.
+
 **What the tests are.** A contract with bounded conformance evidence, not a proof. The race tests show one winner among twenty racers and three among twenty on capacity three, in those runs. The interleaving that let a child survive its parent's release is forced deterministically with `GIT_LOCKS_PAUSE_BEFORE_COMMIT`, a test-only gate that makes a transaction wait for a file before committing, and the invariant is asserted on the resulting store.
 
 ## How it was built, including the missteps worth keeping
@@ -398,6 +400,7 @@ Output is JSON Lines on every command; there is no text mode.
 | `with --job <id> --holder <name> [--ttl <s>] [--wait <s>] [--parent <id>] <path>... -- <cmd>...` | claim, run the command, release; `--wait` retries once a second until the paths are free or the wait runs out | the command's own stdout; git-locks' `claimed`, `released` and refusals go to **stderr** | the command's exit status; 1 if never acquired; 130/143 on INT/TERM after releasing |
 | `version` | tool name and version | one object | 0 |
 | `schema` | the JSON Schema every line above conforms to | the schema document | 0 |
+| `doctor` | read-only invariant check of the store; nothing is repaired | one `finding` object per broken invariant as it is found, then one `doctor` object with the basis (refs, records, clock), the checks run and the verdict | 0 healthy, 1 with findings, 2 if the store cannot be read |
 | `sem create <name> --capacity <n>` | a semaphore with n slots | one `created` object | 0, 1 if it exists |
 | `sem acquire <name> --job <id> --holder <name> [--ttl <s>] [--wait <s>]` | take a slot; re-acquiring refreshes the job's own slot; `--wait` retries once a second | one `acquired` object with `live` and `capacity` | 0, 1 when full |
 | `sem release <name> --job <id>` | give the slot back | one `released` or `nothing` object | 0 |

lib/000-prelude.sh

@@ -60,7 +60,7 @@ DEFAULT_TTL=14400
 SCHEMA='git-locks/1'
 SEM_SCHEMA='git-locks-sem/1'
 SLOT_SCHEMA='git-locks-slot/1'
-VERSION='0.4.0'
+VERSION='0.5.0'
 RETRIES=200   # a plan refused for a stale expectation is re-read and re-planned this many times
 NOW_CACHED='' # the clock, read once per invocation by now()
 
@@ -79,6 +79,7 @@ usage: git locks claim   --job <id> --holder <name> [--ttl <seconds>] [--parent
        git locks with    --job <id> --holder <name> [--ttl <seconds>] [--wait <seconds>] [--sem <name>] [<path>...] -- <command>...
        git locks sem     create <name> --capacity <n> | acquire <name> --job <id> --holder <name> [--ttl <s>] [--wait <s>]
                                   | release <name> --job <id> [--record <oid> | --acquisition <id>] | show <name> | list | delete <name>
+       git locks doctor
        git locks version
        git locks help | schema
 
@@ -104,6 +105,9 @@ with     claim, run the command, release the acquisition it made (also on failur
 sem      capacity, not exclusivity: up to <n> jobs hold a named semaphore at once; a slot expires like a
          lock; acquire is one transaction with a compare-and-swap on the semaphore's generation, so racers
          beyond capacity fail and exactly <n> win
+doctor   read-only invariant check of the store: one finding line per problem, then a doctor line with the
+         basis (refs and records read, the clock) and the verdict; exit 1 on findings, 2 when the store
+         cannot be read (an unreadable store is never healthy). Diagnosis only: nothing is repaired
 schema   print the JSON Schema every output line conforms to
 
 output:  JSON Lines, always: one object per result on stdout, written as each result is known;
@@ -150,6 +154,7 @@ sub_usage_text() {
     ttl) printf 'usage: git locks ttl --job <id>\n' ;;
     extend) printf 'usage: git locks extend --job <id> --ttl <seconds>\n' ;;
     with) printf 'usage: git locks with --job <id> --holder <name> [--ttl <seconds>] [--wait <seconds>] [--sem <name>] [<path>...] -- <command>...\n' ;;
+    doctor) printf 'usage: git locks doctor\n' ;;
     sem) printf 'usage: git locks sem create <name> --capacity <n> | acquire <name> --job <id> --holder <name> [--ttl <s>] [--wait <s>] | release <name> --job <id> [--record <oid> | --acquisition <id>] | show <name> | list | delete <name>\n' ;;
     *) usage_text ;;
   esac

lib/010-json.sh

@@ -36,18 +36,18 @@ json_paths() { # VAR: set VAR to a JSON array of the lines on stdin
   printf -v "$1" '[%s]' "${items[*]}"
 }
 
-json_paths_v() { # VAR TEXT: set VAR to a JSON array of TEXT's non-empty lines; no fork
-  local text="$2" line items=() one IFS
-  while [[ -n "${text}" ]]; do
-    line="${text%%$'\n'*}"
-    if [[ "${line}" == "${text}" ]]; then text=''; else text="${text#*$'\n'}"; fi
-    if [[ -n "${line}" ]]; then
-      json_str one "${line}"
-      items+=("${one}")
+json_paths_v() { # VAR TEXT: set VAR to a JSON array of TEXT's non-empty lines; no fork (locals underscored so none can shadow VAR)
+  local _jp_text="$2" _jp_line _jp_items=() _jp_one IFS
+  while [[ -n "${_jp_text}" ]]; do
+    _jp_line="${_jp_text%%$'\n'*}"
+    if [[ "${_jp_line}" == "${_jp_text}" ]]; then _jp_text=''; else _jp_text="${_jp_text#*$'\n'}"; fi
+    if [[ -n "${_jp_line}" ]]; then
+      json_str _jp_one "${_jp_line}"
+      _jp_items+=("${_jp_one}")
     fi
   done
   IFS=','
-  printf -v "$1" '[%s]' "${items[*]}"
+  printf -v "$1" '[%s]' "${_jp_items[*]}"
 }
 
 json_jobs() { # VAR job... -> JSON array of job ids

lib/175-doctor.sh

@@ -0,0 +1,289 @@
+# ---------------------------------------------------------------- doctor
+#
+# A read-only invariant check over one snapshot. Every finding is one line
+# as it is found; the last line states the basis (how many refs and records
+# were read, at what clock) and the verdict. A store that cannot be read is
+# a store-read error, exit 2, never "healthy". Diagnosis only: nothing here
+# writes, and repair, if it ever exists, is a separate explicit command.
+
+DOC_FINDINGS=0
+DOC_CHECKS='"record-decodes","job-ref-name","path-ref-missing","path-ref-elsewhere","path-ref-orphan","path-ref-stray","parent-missing","parent-expired","parent-holder","family-cycle","sem-meta","sem-gen","sem-record","sem-capacity","unknown-ref"'
+
+finding() { # check subject detail -> one finding line on stdout
+  local _j1 _j2 _j3
+  json_str _j1 "$1"
+  json_str _j2 "$2"
+  json_str _j3 "$3"
+  printf '{"event":"finding","check":%s,"subject":%s,"detail":%s}\n' "${_j1}" "${_j2}" "${_j3}"
+  DOC_FINDINGS=$((DOC_FINDINGS + 1))
+}
+
+is_int() { [[ "$1" =~ ^[0-9]+$ ]]; }
+
+doctor_hash_paths() { # path... -> PATH_HASH for every path, in one git process: each path becomes a file, hash-object hashes them all
+  local dir p i=0 files=() todo=() out h rc
+  for p in "$@"; do
+    [[ -n "${PATH_HASH[${p}]+x}" ]] && continue
+    in_list "${p}" "${todo[@]}" && continue
+    todo+=("${p}")
+  done
+  ((${#todo[@]} > 0)) || return 0
+  dir="$(mktemp -d "${TMPDIR:-/tmp}/git-locks-doctor.XXXXXX")" || fail 'cannot create a temporary directory for hashing' 2
+  for p in "${todo[@]}"; do
+    printf '%s' "${p}" >"${dir}/${i}"
+    files+=("${dir}/${i}")
+    i=$((i + 1))
+  done
+  out="$(g hash-object --no-filters "${files[@]}" 2>&1)" # --no-filters: the same bytes path_ref hashes from stdin
+  rc=$?
+  rm -rf "${dir}"
+  ((rc == 0)) || store_error "hash-object exited ${rc}: ${out}"
+  i=0
+  while IFS= read -r h; do
+    [[ -z "${h}" ]] && continue
+    valid_oid "${h}" || store_error "hash-object line does not parse: ${h}"
+    PATH_HASH["${todo[${i}]}"]="${h}"
+    i=$((i + 1))
+  done <<<"${out}"
+  ((i == ${#todo[@]})) || store_error "hash-object returned ${i} hashes for ${#todo[@]} paths"
+}
+
+doctor_lock_record() { # subject oid -> 0 when the record decodes as a lock record, else findings and 1
+  local schema job holder claimed expires acq paths bad=0
+  field_v schema "$2" schema
+  if [[ "${schema}" != "${SCHEMA}" ]]; then
+    finding record-decodes "$1" "record ${2} has schema '${schema}', not ${SCHEMA}"
+    return 1
+  fi
+  field_v job "$2" job
+  valid_job "${job}" || {
+    finding record-decodes "$1" "record ${2} has no valid job id"
+    bad=1
+  }
+  field_v holder "$2" holder
+  [[ -n "${holder}" ]] || {
+    finding record-decodes "$1" "record ${2} has no holder"
+    bad=1
+  }
+  field_v claimed "$2" claimed
+  is_int "${claimed}" || {
+    finding record-decodes "$1" "record ${2} has no numeric claimed"
+    bad=1
+  }
+  field_v expires "$2" expires
+  is_int "${expires}" || {
+    finding record-decodes "$1" "record ${2} has no numeric expires"
+    bad=1
+  }
+  field_v acq "$2" acquisition
+  [[ -n "${acq}" ]] || {
+    finding record-decodes "$1" "record ${2} has no acquisition id"
+    bad=1
+  }
+  record_paths_v paths "$2"
+  [[ -n "${paths}" ]] || {
+    finding record-decodes "$1" "record ${2} lists no paths"
+    bad=1
+  }
+  return "${bad}"
+}
+
+doctor_slot_record() { # subject oid name job -> 0 when the record decodes as a slot of that semaphore for that job
+  local schema v bad=0
+  field_v schema "$2" schema
+  if [[ "${schema}" != "${SLOT_SCHEMA}" ]]; then
+    finding sem-record "$1" "slot record ${2} has schema '${schema}', not ${SLOT_SCHEMA}"
+    return 1
+  fi
+  field_v v "$2" semaphore
+  [[ "${v}" == "$3" ]] || {
+    finding sem-record "$1" "slot record ${2} names semaphore '${v}'"
+    bad=1
+  }
+  field_v v "$2" job
+  [[ "${v}" == "$4" ]] || {
+    finding sem-record "$1" "slot record ${2} names job '${v}'"
+    bad=1
+  }
+  field_v v "$2" holder
+  [[ -n "${v}" ]] || {
+    finding sem-record "$1" "slot record ${2} has no holder"
+    bad=1
+  }
+  field_v v "$2" claimed
+  is_int "${v}" || {
+    finding sem-record "$1" "slot record ${2} has no numeric claimed"
+    bad=1
+  }
+  field_v v "$2" expires
+  is_int "${v}" || {
+    finding sem-record "$1" "slot record ${2} has no numeric expires"
+    bad=1
+  }
+  return "${bad}"
+}
+
+cmd_doctor() {
+  (($# == 0)) || usage
+  ensure_snapshot
+  local rows ref oid job name rest at refs_n=0 recs_n="${#BLOB[@]}"
+  local -A JOB_OID=() OID_JOBS=() PATHREF_OID=() EXPECTED_PATHREF=() JOB_OK=()
+  local -A SEM_META=() SEM_GEN=() SEM_SLOT_OIDS=() SEM_SLOT_JOBS=() SEM_NAMES=()
+  local jobs=() sems=() pathrefs=() all_paths=() p paths
+  now_v at
+  rows="$(refs_under "${NS}/")" # sorted, so findings come in a stable order
+  while IFS=' ' read -r ref oid; do
+    [[ -z "${ref}" ]] && continue
+    refs_n=$((refs_n + 1))
+    case "${ref}" in
+      "${NS}"/jobs/*)
+        job="${ref#"${NS}"/jobs/}"
+        JOB_OID["${job}"]="${oid}"
+        OID_JOBS["${oid}"]+="${job} "
+        jobs+=("${job}")
+        ;;
+      "${NS}"/paths/*)
+        PATHREF_OID["${ref}"]="${oid}"
+        pathrefs+=("${ref}")
+        ;;
+      "${NS}"/sem/*)
+        rest="${ref#"${NS}"/sem/}"
+        name="${rest%%/*}"
+        rest="${rest#*/}"
+        if [[ -z "${SEM_NAMES[${name}]+x}" ]]; then
+          SEM_NAMES["${name}"]=1
+          sems+=("${name}")
+        fi
+        case "${rest}" in
+          meta) SEM_META["${name}"]="${oid}" ;;
+          gen) SEM_GEN["${name}"]="${oid}" ;;
+          slots/*)
+            SEM_SLOT_OIDS["${name}"]+="${oid} "
+            SEM_SLOT_JOBS["${name}"]+="${rest#slots/} "
+            ;;
+          *) finding unknown-ref "${ref}" 'a semaphore ref that is not meta, gen or a slot' ;;
+        esac
+        ;;
+      *) finding unknown-ref "${ref}" 'a ref in the namespace that is not a job, path or semaphore ref' ;;
+    esac
+  done <<<"${rows}"
+
+  # Job records decode and name their own job; collect every path they list.
+  for job in "${jobs[@]}"; do
+    oid="${JOB_OID[${job}]}"
+    doctor_lock_record "${job}" "${oid}" || continue
+    JOB_OK["${job}"]=1
+    field_v rest "${oid}" job
+    [[ "${rest}" == "${job}" ]] || finding job-ref-name "${job}" "the job ref points at a record for job '${rest}'"
+    record_paths_v paths "${oid}"
+    while [[ -n "${paths}" ]]; do
+      p="${paths%%$'\n'*}"
+      if [[ "${p}" == "${paths}" ]]; then paths=''; else paths="${paths#*$'\n'}"; fi
+      [[ -n "${p}" ]] && all_paths+=("${p}")
+    done
+  done
+  doctor_hash_paths "${all_paths[@]}"
+
+  # Every listed path has a path ref pointing at this record; every path ref is listed by the record it points at.
+  local have key
+  for job in "${jobs[@]}"; do
+    [[ -n "${JOB_OK[${job}]+x}" ]] || continue
+    oid="${JOB_OID[${job}]}"
+    record_paths_v paths "${oid}"
+    while [[ -n "${paths}" ]]; do
+      p="${paths%%$'\n'*}"
+      if [[ "${p}" == "${paths}" ]]; then paths=''; else paths="${paths#*$'\n'}"; fi
+      [[ -n "${p}" ]] || continue
+      ref="${NS}/paths/${PATH_HASH[${p}]}"
+      have="${PATHREF_OID[${ref}]:-}"
+      if [[ -z "${have}" ]]; then
+        finding path-ref-missing "${job}" "no path ref for '${p}': a claim on it would not see this lock"
+      elif [[ "${have}" != "${oid}" ]]; then
+        finding path-ref-elsewhere "${job}" "the path ref for '${p}' points at record ${have} (job ${OID_JOBS[${have}]:-of no job ref})"
+      fi
+      EXPECTED_PATHREF["${ref}|${oid}"]=1 # this record lists a path hashing to this ref
+    done
+  done
+  for ref in "${pathrefs[@]}"; do # in ref order, from the sorted rows
+    oid="${PATHREF_OID[${ref}]}"
+    if [[ -z "${OID_JOBS[${oid}]+x}" ]]; then
+      finding path-ref-orphan "${ref}" "points at record ${oid}, which no job ref points at: the path reads as held by nothing a release can name"
+    elif key="${ref}|${oid}" && [[ -z "${EXPECTED_PATHREF[${key}]+x}" ]]; then
+      finding path-ref-stray "${ref}" "points at record ${oid} (job ${OID_JOBS[${oid}]% }) which lists no path hashing to this ref"
+    fi
+  done
+
+  # Families: the parent exists, is live, has the same holder, and the chain has no cycle.
+  local parent pexp pholder holder cur steps
+  for job in "${jobs[@]}"; do
+    [[ -n "${JOB_OK[${job}]+x}" ]] || continue
+    oid="${JOB_OID[${job}]}"
+    field_v parent "${oid}" parent
+    [[ -n "${parent}" ]] || continue
+    if [[ -z "${JOB_OID[${parent}]+x}" ]]; then
+      finding parent-missing "${job}" "names parent '${parent}', which has no job ref: a child cannot outlive its parent"
+      continue
+    fi
+    field_v pexp "${JOB_OID[${parent}]}" expires
+    ((${pexp:-0} > at)) || finding parent-expired "${job}" "parent '${parent}' expired at ${pexp:-0}; sweep removes both"
+    field_v holder "${oid}" holder
+    field_v pholder "${JOB_OID[${parent}]}" holder
+    [[ "${holder}" == "${pholder}" ]] || finding parent-holder "${job}" "held by '${holder}' but parent '${parent}' is held by '${pholder}'"
+    cur="${parent}"
+    steps=0
+    while [[ -n "${cur}" && -n "${JOB_OID[${cur}]+x}" ]] && ((steps <= ${#jobs[@]})); do
+      if [[ "${cur}" == "${job}" ]]; then
+        finding family-cycle "${job}" "its parent chain returns to itself"
+        break
+      fi
+      field_v cur "${JOB_OID[${cur}]}" parent
+      steps=$((steps + 1))
+    done
+  done
+
+  # Semaphores: meta and gen exist, records decode, live slots fit the capacity.
+  local cap live sjob soid slot_oids slot_jobs exp
+  for name in "${sems[@]}"; do
+    if [[ -z "${SEM_META[${name}]+x}" ]]; then
+      finding sem-meta "${name}" 'no meta ref: the semaphore has no capacity'
+      cap=''
+    else
+      field_v rest "${SEM_META[${name}]}" schema
+      if [[ "${rest}" != "${SEM_SCHEMA}" ]]; then
+        finding sem-record "${name}" "meta record ${SEM_META[${name}]} has schema '${rest}', not ${SEM_SCHEMA}"
+        cap=''
+      else
+        field_v cap "${SEM_META[${name}]}" capacity
+        if ! is_int "${cap}" || ((cap < 1)); then
+          finding sem-record "${name}" "meta record ${SEM_META[${name}]} has capacity '${cap}'"
+          cap=''
+        fi
+        field_v rest "${SEM_META[${name}]}" semaphore
+        [[ "${rest}" == "${name}" ]] || finding sem-record "${name}" "meta record names semaphore '${rest}'"
+      fi
+    fi
+    [[ -n "${SEM_GEN[${name}]+x}" ]] || finding sem-gen "${name}" 'no gen ref: acquire and release cannot compare-and-swap'
+    live=0
+    slot_oids="${SEM_SLOT_OIDS[${name}]:-}"
+    slot_jobs="${SEM_SLOT_JOBS[${name}]:-}"
+    while [[ -n "${slot_oids}" ]]; do
+      soid="${slot_oids%% *}"
+      slot_oids="${slot_oids#* }"
+      sjob="${slot_jobs%% *}"
+      slot_jobs="${slot_jobs#* }"
+      doctor_slot_record "${name}/${sjob}" "${soid}" "${name}" "${sjob}" || continue
+      field_v exp "${soid}" expires
+      ((exp > at)) && live=$((live + 1))
+    done
+    if [[ -n "${cap}" ]] && ((live > cap)); then
+      finding sem-capacity "${name}" "${live} live slots over a capacity of ${cap}"
+    fi
+  done
+
+  local _j1 healthy=true
+  ((DOC_FINDINGS == 0)) || healthy=false
+  json_str _j1 "${STORE}"
+  printf '{"event":"doctor","store":%s,"basis":{"refs":%s,"records":%s,"now":%s},"checks":[%s],"findings":%s,"healthy":%s}\n' \
+    "${_j1}" "${refs_n}" "${recs_n}" "${at}" "${DOC_CHECKS}" "${DOC_FINDINGS}" "${healthy}"
+  ((DOC_FINDINGS == 0))
+}

lib/990-main.sh

@@ -26,7 +26,7 @@ main() {
       printf '{"name":"git-locks","version":"%s"}\n' "${VERSION}"
       exit 0
       ;;
-    claim | batch | release | check | list | sweep | store | show | ttl | extend | with | sem) ;;
+    claim | batch | release | check | list | sweep | store | show | ttl | extend | with | sem | doctor) ;;
     *) usage ;;
   esac
   for a in "$@"; do