Async-first foundation for turning oil-field telemetry into clean, readable data products.
Oil Data Back is a compact backend playground for an oil-data service. The repository keeps the core intentionally small: FastAPI at the edge, async SQLAlchemy for persistence, Alembic for migrations, and a static visual lab that explains the direction of the project.
The GitHub Pages showcase is the quickest way to understand the repository. It includes:
- an interactive signal explorer with pressure, temperature, flow, and water-cut controls;
- a deterministic sample payload generator with copy-to-clipboard;
- a visual architecture map for the current backend layers;
- a responsive, dependency-free interface that works as a static GitHub Pages site.
The playground is deliberately a front-end simulation. It makes the product direction visible without pretending that production telemetry already exists.
Deterministic sample data for the visual lab — not a live production feed.
The project is intentionally easy to scan: readings enter through a versioned API boundary, become validated data, move through async persistence, and surface as an understandable insight.
| Part | Role | Status |
|---|---|---|
pipeline/ |
Kafka → Spark → Bronze / Silver / hourly + daily Parquet marts, replay, verification | Working, tested |
app/ |
FastAPI read API over the marts (/snapshot, /wells, /marts/{hourly|daily}) and GET /api/v1/health |
Working, tested |
docs/ |
Interactive GitHub Pages showcase | Ready |
design/specs/ |
Design documents | Reference |
Run the telemetry pipeline (needs Docker):
make up # Kafka + Spark processors
make scenario # duplicates, a late schema-v2 event, bad input, restarts, replays
make verify # observed vs expected counts and aggregates; nonzero exit on mismatch
make api # read API: http://localhost:8000/docs (API_PORT=8010 make api if 8000 is busy)
make down # stop, keeping Kafka, Parquet and checkpoint volumesRun the tests and the API shell without Docker:
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements/dev.txt
pytest
uvicorn app.main:app --reloadThe static showcase needs neither Python nor a database:
python3 -m http.server 8080 --directory docsapp/ # FastAPI read API over the marts, settings, async DB foundation
pipeline/ # contract, generator, Spark jobs, transform, snapshots, verify, CLI
tests/ # unit tests and an in-process recovery scenario
design/ # specs
docs/ # GitHub Pages showcase
alembic/ # migrations configuration
scripts/ # end-to-end scenario
Each materialization rereads all of Bronze, which favors clear late-event correction and replay over throughput. It is sized for a local fixture, not production volume. Kafka retention must exceed any planned processor outage.
Next: connect the visual lab to the read API so the site shows real aggregates.
No license has been selected yet. Add one before publishing this project for reuse.