Skip to content

Add datasource BACKFILL skip support#17

Open
gnzjgo wants to merge 1 commit into
mainfrom
fix-datasource-backfill-skip-16
Open

Add datasource BACKFILL skip support#17
gnzjgo wants to merge 1 commit into
mainfrom
fix-datasource-backfill-skip-16

Conversation

@gnzjgo
Copy link
Copy Markdown
Member

@gnzjgo gnzjgo commented May 29, 2026

Closes #16.

Summary

  • Adds backfill="skip" to DatasourceOptions / define_datasource.
  • Emits BACKFILL skip in generated .datasource files.
  • Parses legacy .datasource BACKFILL skip during migration and emits the equivalent Python option.
  • Rejects unsupported backfill values so the SDK doesn't silently generate invalid directives.

Reproducibility

Before this change, the following datasource definition accepted the extra option but generated no BACKFILL directive:

from tinybird_sdk import define_datasource, t
from tinybird_sdk.generator.datasource import generate_datasource

daily_page_visits = define_datasource("daily_page_visits", {
    "schema": {
        "date": t.date(),
        "page_url": t.string(),
        "visits": t.uint64(),
    },
    "backfill": "skip",
})

print(generate_datasource(daily_page_visits).content)

After this change, the generated output includes:

ENGINE "MergeTree"
BACKFILL skip

I also added migration coverage for parsing a legacy .datasource containing BACKFILL skip and emitting Python with 'backfill': "skip".

Validation

  • uv run --frozen pytest105 passed
  • Manual repro script above with uv run python confirmed generated output contains BACKFILL skip.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add Data Source BACKFILL skip support

1 participant