Skip to content

feat: Add local Iceberg materialization sink - #6747

Open
HaoXuAI wants to merge 9 commits into
masterfrom
feat/iceberg-local-materialization-sink
Open

feat: Add local Iceberg materialization sink#6747
HaoXuAI wants to merge 9 commits into
masterfrom
feat/iceberg-local-materialization-sink

Conversation

@HaoXuAI

@HaoXuAI HaoXuAI commented Aug 16, 2026

Copy link
Copy Markdown
Collaborator

What changed

  • add PyIceberg-backed catalog access and table upserts to IcebergSource
  • route local compute materialization output into Iceberg-backed derived feature views
  • validate join keys and Arrow schemas before writes, with safe contextual errors
  • add unit and component coverage using a real local PyIceberg SQL catalog
  • document configuration, upsert keys, compatibility, and failure behavior

Why

Feast could read Iceberg sources, but local materialization could not persist results back to Iceberg. This adds a local-compute sink path so derived feature data can be maintained in Iceberg with idempotent upsert semantics.

Impact

Users can configure an IcebergSource as the effective batch source of a derived feature view and have local materialization upsert rows using mapped entity join keys plus the event timestamp. Existing online and offline materialization behavior remains unchanged.

Validation

  • focused Iceberg and local compute suite: 104 passed
  • Ruff formatting and lint: passed
  • mypy for changed Python modules: passed
  • Spark correctness suite with project Python pinned: 5 passed
  • independent code review: no remaining Critical or Important findings

The broader repository unit run reached 2,469 passes. Three existing tests in unchanged on-demand transformation and SQLite paths still fail locally; the failures do not enter the Iceberg sink code. The pre-push lint-push wrapper was skipped because its uv run resolves incompatible unrelated ci and flink extras; the underlying lint and type checks were run directly and passed.

Signed-off-by: HaoXuAI <sduxuhao@gmail.com>
Signed-off-by: HaoXuAI <sduxuhao@gmail.com>
Signed-off-by: HaoXuAI <sduxuhao@gmail.com>
Signed-off-by: HaoXuAI <sduxuhao@gmail.com>
Signed-off-by: HaoXuAI <sduxuhao@gmail.com>
Signed-off-by: HaoXuAI <sduxuhao@gmail.com>
Signed-off-by: HaoXuAI <sduxuhao@gmail.com>
@codecov-commenter

codecov-commenter commented Aug 16, 2026

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 20.58824% with 81 lines in your changes missing coverage. Please review.
✅ Project coverage is 47.04%. Comparing base (f0bc070) to head (8dfd514).

Files with missing lines Patch % Lines
..._sources/contrib/iceberg_catalog/iceberg_source.py 10.98% 81 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #6747      +/-   ##
==========================================
- Coverage   47.09%   47.04%   -0.05%     
==========================================
  Files         419      419              
  Lines       51878    51977      +99     
  Branches     7525     7541      +16     
==========================================
+ Hits        24430    24453      +23     
- Misses      25700    25776      +76     
  Partials     1748     1748              
Flag Coverage Δ
go-feature-server 30.37% <ø> (-0.21%) ⬇️
python-unit 48.36% <20.58%> (-0.04%) ⬇️
Files with missing lines Coverage Δ
...ast/infra/compute_engines/local/feature_builder.py 56.41% <100.00%> (+0.56%) ⬆️
.../python/feast/infra/compute_engines/local/nodes.py 85.02% <100.00%> (+1.18%) ⬆️
..._sources/contrib/iceberg_catalog/iceberg_source.py 24.33% <10.98%> (-0.67%) ⬇️

... and 2 files with indirect coverage changes


Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f0bc070...8dfd514. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@HaoXuAI
HaoXuAI marked this pull request as ready for review August 22, 2026 17:50
@HaoXuAI
HaoXuAI requested a review from a team as a code owner August 22, 2026 17:50
@HaoXuAI HaoXuAI changed the title feat: add local Iceberg materialization sink feat: Add local Iceberg materialization sink Aug 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants