perf(unique): make verifyUniqueWithinMutation linear - #9822
Conversation
Fixes #9814. The in-request duplicate check for @unique predicates compared every unique-predicate edge against every other, calling dql.TypeValFrom once per pair: O(N^2) time and allocations in the number of edges per mutation. At 8k edges one check took 1.67s and 64M allocations, dominating batched writes on @unique predicates. Replace the nested scan with a single pass over a seen-map keyed on (predicate, value), remembering the first subject that set each value. Semantics are unchanged: duplicate values from the same subject remain allowed, nil ObjectValues are skipped, entries pruned by updateMutations are still ignored, and the error message is identical. Value identity still uses the interface{} produced by TypeValFrom, so type identity participates in the comparison exactly as it did with ==. Measured (M4 Pro, benchstat over 6 runs, all p=0.002): 5.20ms -> 32.6us at 500 edges, 1.67s -> 585us at 8000 edges (-99.96%); allocs/op drops from N^2 (64M at 8k) to ~N (8k). The after curve doubles per doubling of N, i.e. linear.
There was a problem hiding this comment.
🟡 Changes recommended
The new linear algorithm should be backed by targeted unit tests asserting the core within-mutation @unique semantics to guard against regressions.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR optimizes the in-request duplicate check for @unique predicates by replacing the previous quadratic pairwise scan with a linear, map-based pass keyed by (predicate, value), reducing CPU and allocations for large batched mutations.
Changes:
- Introduces a
uniqueValueKeyto represent(predicate, value)identity for within-mutation duplicate detection. - Rewrites
verifyUniqueWithinMutationto track first-seen subjects in aseenmap, making the check O(N) in the number of unique edges. - Preserves prior semantics around same-subject duplicates, nil
ObjectValueskipping, and pruned-mutation handling.
File summaries
| File | Description |
|---|---|
edgraph/server.go |
Replaces O(N²) within-mutation @unique duplicate detection with a single-pass seen-map keyed by (predicate, value). |
Review details
- Files reviewed: 1/1 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| type uniqueValueKey struct { | ||
| predicate string | ||
| value interface{} | ||
| } |
There was a problem hiding this comment.
uniqueValueKey.value holds whatever dql.TypeValFrom returned, and five of its ten branches return an unhashable dynamic type: []byte for BytesVal/GeoVal/DatetimeVal/BigfloatVal, and []float32 for Vfloat32Val. Hashing one of those is panic: hash of unhashable type, and there's no recover() anywhere in the mutation path (edgraph/, worker/, dgraph/cmd/alpha/), so it takes the alpha process down.
The old code had the same hazard — == on uncomparable interfaces panics too — but the pred2.Predicate == pred1.Predicate short-circuit meant it took two edges on the same @unique predicate before anything got compared. The new code hashes on the first edge, unconditionally.
It's reachable from a plain JSON mutation, with no vector predicate and no ACL involved. chunker/json_parser.go:224 runs types.ParseVFloat on every string value before it consults the schema, so any string shaped like [...] becomes a Vfloat32Val:
// schema: email: string @unique @index(hash) .
nqs, _, _ := chunker.ParseJSON([]byte(`[{"uid":"_:a","email":"[1.0, 2.0]"}]`), chunker.SetNquads)
// ObjectValue is *api.Value_Vfloat32Val, so TypeValFrom(...).Value is []float32
verifyUniqueWithinMutation(qc)
// panic: hash of unhashable type: []float32
// edgraph.verifyUniqueWithinMutation edgraph/server.go:2477Simplest fix I found is to key on a hashable encoding that keeps the dynamic-type half of the old ==:
type uniqueValueKey struct {
predicate string
value string
}
v := dql.TypeValFrom(pred.ObjectValue).Value
key := uniqueValueKey{predicate: pred.Predicate, value: fmt.Sprintf("%T\x00%v", v, v)}I prototyped that locally: the vfloat case returns could not insert duplicate value [[1 2]] for predicate [email] instead of panicking, and TestVerifyUniqueWithinMutationBoundsChecks still passes. One Sprintf per edge keeps it O(N), and addQueryIfUnique already pays a fmt.Sprintf("%v", ...) per unique edge anyway.
Worth being deliberate about that %T. Keeping it preserves today's behavior where int64(1) and "1" stay distinct while DefaultVal "x" still equals StrVal "x" (both land on Go string), which is what == did. Dropping it would instead align this check with the injected eq() query, which stringifies with plain %v and already conflates the two. That divergence predates this PR so I'd keep %T and leave it alone, but the key encoding is where the choice gets made.
| // the same @unique predicate for different subjects. A single linear pass over a | ||
| // seen-map replaces the earlier every-pair scan, which was O(N^2) in the number of | ||
| // unique-predicate edges and dominated large batched mutations (issue #9814). | ||
| func verifyUniqueWithinMutation(qc *queryContext) error { |
There was a problem hiding this comment.
No new tests here, and this function has history — #9670 was a panic in the same spot. Worth adding alongside TestVerifyUniqueWithinMutationBoundsChecks:
- a non-scalar value on a
@uniquepredicate (regression for the panic above) - duplicate value from the same subject is allowed
- duplicate value from a different subject is rejected
Fixes #9814.
The in-request duplicate check for @unique predicates compared every unique-predicate edge against every other, calling dql.TypeValFrom once per pair: O(N^2) time and allocations in the number of edges per mutation. At 8k edges one check took 1.67s and 64M allocations, dominating batched writes on @unique predicates.
Replace the nested scan with a single pass over a seen-map keyed on (predicate, value), remembering the first subject that set each value. Semantics are unchanged: duplicate values from the same subject remain allowed, nil ObjectValues are skipped, entries pruned by updateMutations are still ignored, and the error message is identical. Value identity still uses the interface{} produced by TypeValFrom, so type identity participates in the comparison exactly as it did with ==.
Measured (M4 Pro, benchstat over 6 runs, all p=0.002): 5.20ms -> 32.6us at 500 edges, 1.67s -> 585us at 8000 edges (-99.96%); allocs/op drops from N^2 (64M at 8k) to ~N (8k).
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.