Skip to content

About

Hands-on, multi-cloud data engineering projects — built to learn deeply and show the work.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

Data Engineering Lab

Hands-on data engineering projects across clouds and open-source stacks — built to learn deeply and to show the work, by an engineer and an AI teammate.

Portfolio: https://tomasripsky.github.io/data-engineering-lab/ — one card per project, each with its own site.

The world and its countries

flowchart LR
  subgraph world["This repo — the base"]
    agent["agent/<br/>how Claude works<br/>and evolves"]
    lab["lab/<br/>rules · ADRs<br/>tech radar · designs"]
    template["template/<br/>what a new project<br/>starts as"]
    hub["site/<br/>portfolio hub"]
  end
  subgraph countries["The countries — one repository each"]
    pitwall["pitwall"]
    citypulse["CityPulse"]
    next["next project…"]
  end
  vault[("Private second brain<br/>concepts · ideas")]
  template -- bootstraps --> countries
  lab -. rules .-> countries
  agent -. builds and teaches .-> countries
  countries -- lessons --> lab
  hub -- links to each site --> countries
  agent -. writes .-> vault
Loading

This repository is the base: the rules every project follows (lab/), the AI teammate that builds with me (agent/) and the template a new project starts from (template/). Each project is an independent country in its own repository — its code, dependencies, infrastructure, decisions, docs, CI and public site — that can be read, run and torn down on its own (lab ADR 0007). What a project teaches comes back here as a lesson, a convention or a better template.

Projects

Project What it does Stack Status
pitwall Batch ELT over Formula 1 data (OpenF1 → GCS → BigQuery → dbt → Observable site) that explains race strategy — tyres, pit stops and the undercut — to people who don't follow F1. Python · GCS · BigQuery · dbt · GitHub Actions · Observable ✅ v1.0.0 — dashboard
CityPulse Does the weather change how New York rides bikes? Batch ELT over Open-Meteo weather and air quality and 44.5 M Citi Bike trips (GCS → BigQuery → dbt, orchestrated by Airflow 3) — rebuilt from a pre-lab project with the lab's rules. Python · GCS · BigQuery · dbt · Airflow · Terraform · Observable ✅ v2.0.0 — site

The portfolio hub also shows MarineFlow, a streaming project built before the lab and not yet rebuilt with its rules.

Navigate

I want to know… Go to
What a project does and how to run it its repository's README.md
How a project works inside its docs/guide.md
Why a project made a decision its docs/decisions/
How a project was designed and planned its docs/design/
What a new project starts as template/
What a project's site must look like template/site/README.md, lab ADR 0008
The lab's rules lab/conventions.md, lab/security-and-cost.md
Why the lab is built this way lab/adr/
Which tech we use, try, watch or avoid lab/tech-radar.md
How the AI teammate works agent/

Built with an AI teammate

Every project here is built with Claude Code as a senior engineer and tutor: it designs, builds and explains, and I decide. How that collaboration works — the tutoring loop, the memory, the lessons it has learned and how it changes over time — is documented in agent/.

Getting started

git clone https://github.com/TomasRipsky/data-engineering-lab.git
cd data-engineering-lab
pre-commit install   # enables gitleaks, ruff and branch guards on every commit

About

Hands-on, multi-cloud data engineering projects — built to learn deeply and show the work.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages