HackYourFuture Data Track — Week 13 Assignment

Databricks Lab: PySpark exploration, dbt incremental models on Delta Lake, and Git-backed Job scheduling.

Full instructions live in the curriculum: Week 13 Assignment.

Where to start

Folder / File	What to submit	Points (autograder)
task-1/	PySpark notebook (show() on aggregated results, PySpark-vs-dbt note)	25
task-2/	Ported dbt project + WRITEUP.md (timings + incremental explanation + DESCRIBE HISTORY)	30
task-3/	Git-backed Databricks Job + screenshots + SCHEDULING.md (Jobs vs Airflow)	15
AI_ASSIST.md	Documented LLM usage (prompt, tool, kept/discarded rationale)	15
Required files	Presence of all required files across task-1/, task-2/, task-3/, and AI_ASSIST.md	15
Secrets hygiene	No committed secrets (profiles.yml, .env, tokens)	Blocker if violated

Passing score: 60/100 on the autograder. Your teacher also reviews quality against the rubric (incremental config, > boundary, tool-choice writing, Job configuration).

Repository layout

data-assignment-week-13/
├── task-1/
│   └── pyspark_exploration.ipynb    # or .py export from Databricks
├── task-2/
│   ├── dbt_project.yml              # your ported Week 10 project
│   ├── models/                      # your dbt models
│   ├── profiles.yml.example
│   └── WRITEUP.md                   # incremental build write-up + DESCRIBE HISTORY
├── task-3/                          # Git-backed Job scheduling
│   ├── SCHEDULING.md                # Jobs vs Airflow write-up + Job Run URL
│   └── screenshots/                 # Job config, green run, paused trigger
├── .env.example
├── AI_ASSIST.md                     # LLM interaction log
└── README.md

Setup

cp .env.example .env          # fill in Databricks connection values
cd task-2
cp profiles.yml.example profiles.yml
export $(grep -v '^#' ../.env | xargs)   # or source manually
dbt debug

Use Python 3.11 or 3.12 for dbt if dbt debug crashes on import (uvx --python 3.11 --from dbt-databricks dbt debug).

Check your score locally

bash .hyf/test.sh
cat .hyf/score.json

Scoring ladder (autograder)

Score	What the grader checks
15	Required files present (task-1 notebook, task-2/dbt_project.yml, WRITEUP.md, SCHEDULING.md, AI_ASSIST.md)
25	Task 1 notebook mentions show and borough/payment_type work
30	Task 2 has incremental config (materialized='incremental', merge, unique_key) and a filled WRITEUP.md
15	Task 3 has screenshots in task-3/screenshots/, Job Run URL, and a filled SCHEDULING.md
15	AI_ASSIST.md contains documented prompt and rationale
Pass	Secrets hygiene (no committed .env / profiles.yml / dapi tokens)

Governance and streaming bonuses are teacher-reviewed only; they do not affect the autograder score.

Instructor / track maintainer

This repo is the Week 13 student scaffold. Teacher rubric: week_13__assignment_rubric.md in the datatrack curriculum repo (not shared with students).