Repository navigation
reduce: pend joins of history times advanced past upper - #1109
Open
creatifcoding wants to merge 1 commit into
Open
creatifcoding wants to merge 1 commit into
creatifcoding wants to merge 1 commit into
Conversation
With partially ordered times, a reduce advances its input and output history by the meet of the times it will visit, and that can move a history time from before the retired interval to beyond it. Such a time is not interesting, as no batch or pending time lies below it, but its joins with this interval's batch times and current interesting times are. Both tactics skipped every time in advance of `upper` without forming those joins, so the joins were neither evaluated nor carried forward. No later interval forms them either: there both partners are history, which does not seed interesting times. The output then misses corrections at those joins. The cursor tactic now runs the determination step for an uninteresting time even when it is in advance of `upper_limit`; its joins are also in advance of `upper_limit` and so are deferred. The proxy tactic now closes an unreached time against the reached times even out of the interval; `close` pends the joins. Under a total order an uninteresting time has no such joins, so only partially ordered times take the new path. Regression tests: `distinct` over a three-dimensional product order, the same through `iterate` (a transitive closure, as a randomized test found it), and a retire-level test of the proxy tactic.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1108.
With partially ordered times, a reduce advances its input and output history by the meet of the times it will visit. That can move a history time from before the retired interval to beyond it.
Such a time is not interesting, because no batch or pending time lies below it. Its joins with this interval's batch times and current interesting times are interesting. Both tactics skip every time in advance of
upperwithout forming those joins, so the joins are neither evaluated nor carried forward:if !upper_limit.less_equal(&next_time)gate inHistoryReplayer::compute;Sweep::tick.No later interval forms the joins either, because there both partners are history, and history does not seed interesting times. The output then misses corrections at those joins. For example,
distinctholds a record twice at the join of two incomparable times at which it was introduced [observed;tests/reduce.rsbelow].Concretely:
0arrives at(0,1,0), then at(0,0,2)and(1,0,1)in the interval[(0,0,1), (0,1,1)). The meet(0,0,1)advances(0,1,0)to(0,1,1), which isupperitself. Its join with(0,0,2), at(0,1,2), is wheredistinctmust retract the second copy. That time is never visited [observed via instrumentation].Change
cursor/reduce.rs: the gate becomesif !interesting || !upper_limit.less_equal(&next_time). For an uninteresting time beyondupper_limit, the existing determination step forms only the joins with batch times andtimes_current. Those joins are also beyondupper_limit, so the existing split defers them tonew_interesting, under the existinghelddebug assertion. Evaluation stays gated oninteresting, so nothing is emitted at such a time.int_proxy/reduce.rs: only a reached out-of-interval time short-circuits toPended. An unreached one is closed against the reached times, andclosepends its joins (it already joins only with reached times whenreachedis false).Correctness
The round's output change is supported on
cl(prior ∪ novel) ∩ Above(novel)(formal/Differential/RoundCoverage.lean,round_coverage). That set equals the joins that include at least one novel time (cl_inter_above_eq_novelJoins). The dropped time(0,1,2) = (0,1,0) ∨ (0,0,2)is such a join and lies beyondupper, so it must be pended [inferred].Under a total order the change adds no times [observed in code]. The batch-buffer and
times_currententries have already been stepped past, so under a total order each is<= next_time, and any of them makesnext_timeinteresting (cursor/reduce.rsll. 407-408). An uninterestingnext_timetherefore has both buffers empty. Even otherwise, ll. 427-428 join only entries with!time.less_equal(&next_time), and under a total order such a join is the entry itself, which is already a known time. For partially ordered times the extra work is one join per buffered batch time and current time, at uninteresting times beyondupper[inferred].Joins and the other operators' "beyond
upper" gates were not audited for the same pattern; this change touches only the two reduce tactics.Test
Three regression tests, each failing before the change and passing after [observed]:
tests/reduce.rs::distinct_join_with_history_past_upper: the three-update case above, withdistinctoverProduct<u64, Product<u64, u64>>. It checks every accumulation in[0,4)^3. Before:at (0, (1, 2)) left: {0: 2} right: {0: 1}.tests/reduce.rs::iterate_distinct_join_with_history_past_upper: a transitive closure byiterateanddistinct, from a randomized downstream test (seed 219). It is checked against a closure oracle on[0,4)^3. Before:at (2, (1, 2)),(1, 6): 2.tests/int_proxy.rs::reduce_unreached_time_past_upper_pends_joins: the same three updates through threeProxyReduceTactic::retirecalls. Before:at (0, (1, 2)) left: 2 right: 1.Reverting either source change alone fails only that tactic's tests [observed]. The three tests are also red before and green after in
--release[observed]. The downstream program they were reduced from (adistinct-in-iterateclosure over a three-dimensional time) gives weight 2 before and 1 after [observed].cargo test -p differential-dataflow --all-targetspasses before (101 passed, 3 ignored) and after (104 passed, 3 ignored).cargo test -p differential-dataflow --docpasses both times (54 passed, 6 ignored) [observed].Timing:
examples/dynamic.rs 1000000 4000000 1000 10 no -w1(BFS initerateunderProduct<u64, PointStamp>), release, 5 interleaved runs per side. Median total is 5.078 s before and 5.046 s after; the run-to-run spread (up to 0.33 s) exceeds that difference [observed]. This is not a benchmark of the new path itself.This analysis and patch were prepared with AI assistance and reviewed before submission.