Skip to content

feat: explicit schema for all_events bronze table#69

Open
nkarpov wants to merge 2 commits intomainfrom
feat/bronze-explicit-schema
Open

feat: explicit schema for all_events bronze table#69
nkarpov wants to merge 2 commits intomainfrom
feat/bronze-explicit-schema

Conversation

@nkarpov
Copy link
Copy Markdown
Collaborator

@nkarpov nkarpov commented Apr 2, 2026

Summary

  • Define an explicit StructType schema on the all_events Auto Loader read so ts is ingested as TIMESTAMP and sequence as INT instead of inferred strings
  • Remove the redundant to_timestamp("ts") cast in silver_order_items since ts is now natively typed
  • Improves downstream query performance, especially for time-based filtering when synced to Postgres via reverse ETL

Test plan

  • Run DLT pipeline and verify all_events schema shows ts as TIMESTAMP and sequence as INT
  • Verify silver_order_items and gold tables populate correctly
  • Confirm refund recommender stream handles the typed ts column without errors

This pull request was AI-assisted by Isaac.

nkarpov added 2 commits April 2, 2026 16:34
Replace Auto Loader schema inference with an explicit StructType so ts
is ingested as TIMESTAMP and sequence as INT instead of strings.  Removes
the now-redundant to_timestamp cast in silver_order_items.

Co-authored-by: Isaac
The canonical data replay outputs location_id as an integer, not location
as a string. Aligns the explicit schema with the actual data source.

Co-authored-by: Isaac
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant