02.BData Engineering
Pipelines that answer the whole way to source.
Nobody set out to break the chain. Each step was reasonable and none was recorded, so the trace stops at the first hop nobody can explain. These pipelines rebuild any figure they published from the inputs it had at the time.
The problem
Lineage usually stops where the documentation stopped
A number arrives in a report having passed through an ingestion job, three transformations, a manual correction somebody made in a spreadsheet, and a scheduled overwrite. Each step was reasonable. Nobody recorded the chain, so tracing it back means reading code written by someone who has left, and the trace stops at the first step nobody can explain.
An accredited data recipient under the Consumer Data Right has to show where a consumer data element came from and where it went. That is a lineage question, and it is asked after the fact.
What the work produces
Three things you keep
- 01
A lineage map to source
Every hop a number takes, from the system that first recorded it to the report that shows it. Traceable the whole way, not to the last place the documentation stopped.
- 02
Tests on the definition, not just the schema
A schema test proves a column is a number. A definition test proves it is the number the register says it is. Pipelines here fail on the second one, which is the one that matters.
- 03
A history you can reproduce
Any figure the pipeline published can be rebuilt from the inputs it had at the time. A number that cannot be reproduced is an assertion.
The governed spine
How it answers the spine
A pipeline encodes a definition whether or not anybody wrote one down. Building against the register makes the encoding explicit, so when the definition changes there is one place to change it and a lineage map showing everything the change reaches.
Three artefacts, every engagement
- The written record of what a contested term means, who owns it, and the date the wording was agreed.
- Every hop a number takes from the system that first recorded it to the report that shows it.
- When a definition changed and why, written at the point the call was made rather than reconstructed afterward.
02.BCommission the work
Bring the term two systems report differently.
A 30-minute diagnostic, not a sales call. Bring the disagreement and the deadline that made it matter.
Book a 30-minute diagnostic, opens cal.com in a new tabSheet 04, other ways to reach us