From 012a539584b3a7463e9f19d4cf5adc8a39b9c304 Mon Sep 17 00:00:00 2001 From: ivanauth Date: Fri, 20 Mar 2026 14:31:12 -0400 Subject: [PATCH] Document fully_consistent nuances on CockroachDB Fixes #525 --- app/spicedb/concepts/consistency/page.mdx | 20 ++++++++++++++++++++ app/spicedb/concepts/datastores/page.mdx | 20 ++++++++++++++++++++ 2 files changed, 40 insertions(+) diff --git a/app/spicedb/concepts/consistency/page.mdx b/app/spicedb/concepts/consistency/page.mdx index ab86dc1e..7d655619 100644 --- a/app/spicedb/concepts/consistency/page.mdx +++ b/app/spicedb/concepts/consistency/page.mdx @@ -106,6 +106,26 @@ If you need read-after-write consistency, consider using a [ZedToken] + + **CockroachDB users:** `fully_consistent` does not guarantee read-after-write consistency on CockroachDB. + +SpiceDB selects a revision using CockroachDB's `cluster_logical_timestamp()` at request time. +Because CockroachDB is a distributed database, this timestamp can differ from other nodes by up to the cluster's configured [`max_offset`][max-offset] (default: 500ms). +If the node handling the read picks a timestamp older than the one assigned to a recent write on a different node, the read will not see that write even though the data has been committed. + +This does not affect CockroachDB's normal read-after-write guarantees -- those are bypassed specifically because SpiceDB chooses its own evaluation revision rather than letting CockroachDB select one. + +The [overlap strategy][overlap-strategy] does not mitigate this. +The overlap strategy prevents two concurrent writes from receiving commit timestamps in the reverse order, but it does not address clock skew between nodes during reads. + +If you need read-after-write consistency with CockroachDB, use a [ZedToken] with [`at_least_as_fresh`](#at-least-as-fresh) instead. + +[max-offset]: https://www.cockroachlabs.com/docs/stable/cockroach-start.html#flags-max-offset +[overlap-strategy]: /spicedb/concepts/datastores#overlap-strategy +[ZedToken]: #zedtokens + + + ```proto Consistency { fully_consistent: true } ``` diff --git a/app/spicedb/concepts/datastores/page.mdx b/app/spicedb/concepts/datastores/page.mdx index 2d5d97f0..fe6e795f 100644 --- a/app/spicedb/concepts/datastores/page.mdx +++ b/app/spicedb/concepts/datastores/page.mdx @@ -30,6 +30,26 @@ AuthZed has standardized our managed services on CockroachDB, but we give self-h - Query and data balanced across the CockroachDB - Setup and operational complexity of running CockroachDB +### Consistency Considerations + + + `fully_consistent` does not guarantee read-after-write consistency on CockroachDB. + +SpiceDB picks a revision using CockroachDB's `cluster_logical_timestamp()`, which can differ across nodes by up to the cluster's [`max_offset`][crdb-max-offset] (default: 500ms). +A read may not see a recently committed write if the reading node's timestamp is behind the writing node's. + +This is specific to CockroachDB's distributed clock model and does not apply to single-node datastores like PostgreSQL. +For read-after-write guarantees, use a [ZedToken] with [`at_least_as_fresh`][at-least-as-fresh]. + +See [Consistency: Fully Consistent][fully-consistent] for details. + +[crdb-max-offset]: https://www.cockroachlabs.com/docs/stable/cockroach-start.html#flags-max-offset +[ZedToken]: /spicedb/concepts/consistency#zedtokens +[at-least-as-fresh]: /spicedb/concepts/consistency#at-least-as-fresh +[fully-consistent]: /spicedb/concepts/consistency#fully-consistent + + + ### Developer Notes - Code can be found [here][crdb-code]