Conversation
`interval_diff` advanced past an unready interval whenever the candidate interval started before it. When a daily model depends on an hourly model whose signal marks an hour as not ready, the day containing that hour was compared against the next unready interval instead, so the daily model was evaluated over incomplete upstream data and the interval was recorded as done. Ready intervals after the last overlap were also dropped. Always skip the candidate interval on overlap and keep every remaining interval once the unready intervals are exhausted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: breken-ai <312387581+breken-ai@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
When an upstream model's signal marks an interval as not ready, the scheduler is supposed to hold back every downstream interval that overlaps it.
interval_diffinsqlmesh/core/scheduler.pydoes this, but it only works when the unready interval starts at or before the candidate interval. If the candidate interval is larger and starts first, it takes theindex_b += 1branch. That branch moves past the unready interval and leaves the candidate in place.This happens whenever a model has a coarser interval unit than a parent that has a signal. Example: a daily model reading from an hourly model whose signal reports 2023-01-01 10:00 and 2023-01-03 05:00 as not ready.
The fix skips the candidate interval on any overlap and keeps the unready interval, because it can also overlap the next candidate. Once the unready intervals run out, every remaining candidate starts after the last one, so all of them are kept. The
uninterrupted(depends-on-past) early return is unchanged.Test Plan
test_interval_difffor a large interval that contains a smaller unready interval.test_signals_unready_hourly_parent_blocks_daily_child: a dailyINCREMENTAL_BY_TIME_RANGEmodel downstream of an hourly model with a signal that holds back two hours. Onmainthe child's batches are[(2023-01-01, 2023-01-03)], which evaluates Jan 1 and skips Jan 4. With the fix they are[(2023-01-02, 2023-01-03), (2023-01-04, 2023-01-05)].main(263723f) and pass with the fix.uninterrupted=True) matches.pytest tests/core/test_scheduler.py tests/core/test_snapshot.py: 159 passed.tests/core/test_plan.py,test_plan_evaluator.py,test_context.pyandtests/core/integrationshow the same results with and without the change. The only failures are dbt tests that need dbt, which isn't installed in my environment.ruff check,ruff format --checkandmypy sqlmesh/core/scheduler.pyare clean for the changed files.Checklist
make styleand fixed any issuesmake fast-test)git commit -s) per the DCOI ran ruff and mypy directly rather than the full pre-commit suite. I ran the core scheduler, snapshot, plan and integration suites rather than the full
make fast-test.This fix was found and written with AI assistance (Claude).