ETL PIPELINE
In build. A scheduled extract–transform–load pipeline: pull from source APIs, land the raw payloads untouched, then parse, validate and reshape them into warehouse tables a query can trust. Reruns are the design constraint — the same window replayed twice should land the same rows once. Numbers, source and captures go up when it ships.
SCHEMATIC — THE SHAPE, NOT THE BUILDThis one is still being built — the page is here so it has somewhere to land. A scheduled ETL pipeline that pulls from source APIs, lands the raw payloads untouched, and reshapes them into warehouse tables downstream queries can rely on.
The problem
A pipeline is easy to write once and hard to run every day. The failure modes are all in the second category: a source that rate-limits halfway through, a schema that quietly gains a column, a rerun that double-counts the window it already loaded. Getting those right is the actual work.
Approach
- Extract: scheduled pulls against the source APIs, with retry and backoff, writing raw payloads to a landing area before anything touches them.
- Transform: parse and typecast, dedupe, validate, derive the columns the warehouse actually wants, and fail loudly on a schema that has drifted.
- Load: idempotent upserts into the warehouse tables, so replaying a window lands the same rows once rather than twice.
- A run log and a freshness check, because a pipeline nobody can inspect is a pipeline nobody can trust.
Results
To come. Throughput, runtime and the source repo go up here once the pipeline is running on a schedule, along with captures from the real thing in place of the schematic.