01 · What we do
Data and cloud infrastructure
Pipelines, warehouses, and the cloud they run on, so this morning’s numbers came from last night’s load.
We build the path from source system to warehouse to report: the job that pulls the data, the schedule it runs on, the transformation that shapes it, and the check that runs afterward. A load can be re-run from any date without doubling rows, which is what makes a bad night recoverable.
Everything runs in your own cloud accounts under credentials in your name, so the work does not depend on our access continuing. A report is only as current as the last load that succeeded, which is why the checks that catch a late or empty load are built at the same time as the load itself.
What this covers
Data pipelines and integrations
Ingestion, transforms, backfills, and retries. Loads that can be re-run from a point in history without doubling the data.
Scheduling, late-arriving files, and schema drift are designed in. A silent failure that looks like a successful load is the one we watch for.
Data warehouses
Warehouse design on BigQuery, Postgres, and the surrounding cloud. Partitioning, clustering, and the cost of a scan as a design input.
Every table has a named owner and a documented grain, so a question about a number has somewhere to go.
Reporting and BI infrastructure
The tables and views reporting queries, in the same repository as the pipeline. Metrics have a written definition.
When a number in a dashboard moves, you can find the job that produced it and the query that read it.
Cloud migration and setup
Accounts, IAM, networking, and the first production environment, in your cloud, under your name.
We work in your accounts. Credentials stay yours. The environment can be rebuilt from what is in the repo.
Monitoring and alerting
Checks that notify an engineer when a load is late, empty, duplicated, or more expensive than the bound.
Each alert is delivered with a runbook naming who responds and what they do first.
Backups and recovery
Backups that have been restored at least once. The restore is a documented path, timed.
We run the restore during the engagement and time it, so the path is known before you need it.
Deliverables
Pipelines in your repository
Ingestion, transforms, and backfills, each re-runnable from a chosen date.
A warehouse with owners
Tables with a documented grain and a named owner.
Checks that notify an engineer
Late, empty, duplicated, or over budget, each with a runbook.
Runtime and cost per job
Recorded every run, so a regression shows up as a number.
A restore you have watched
The backup path exercised and timed during the engagement.
Example
Example · Data infrastructure
National-scale telecom pipelines
A transformation layer that ran as hand-written SQL, replaced with generated queries and an orchestrator. Runtime fell by about 85 percent.
The source data sat in GCP and BigQuery, at national telecom scale. The transformation layer was a set of hand-run SQL jobs. Failures meant a person re-ran a step; runtime was the constraint the rest of the platform waited on.
Partitioning and parallelization were part of the same change. Loads became re-runnable. Source, warehouse, and the tables reporting queried became one system.
The stack was Python, SQL, GCP, and BigQuery. The runtime number was taken from the same job, before the change and after it.