Make your systems agree on the same number
You get pipelines that move data reliably, checks that catch bad records before a person acts on them, and a daily proof that two systems still match.
The short version
4 things that decide this
- 01Most reporting problems are not reporting problems. Two systems hold the same fact, nobody decided which one wins, and both are quoted in the same meeting.
- 02A pipeline that silently drops rows is worse than one that fails, because a failure gets fixed and a gap gets reported as a trend.
- 03Data quality is a set of checks that run every day, not a project that finishes.
- 04Hashlogics builds this layer inside its products: TrialTriage tracks 23 audited actions, and TankAware feeds live IoT tank readings into operational reporting.
The number is wrong and everyone still uses it
Bad data does not announce itself. A field changes meaning in one system, a nightly job half-finishes, a timezone gets applied twice. The dashboard still renders. The chart still slopes upward.
Somebody makes a decision on it, and the error only surfaces when two teams compare notes. By then the reports built on that number have been circulating for a month.
The fix is unglamorous and it works. Decide which system owns each fact, check the data every time it moves, and compare the two sides on a schedule so drift is found within a day.
What we have shipped
Counted, not estimated
22
production systems, each with its own data layer
23
tracked actions in TrialTriage's audit trail
50+
ESG topics Greenlight scores per company
10-15
independent sources reconciled behind each Greenlight topic
What we build into the data layer
Each of these exists because of a specific way data goes wrong in production.
- LandRaw copy kept, exactly as it arrived.
- CheckBad batches stop here, not downstream.
- ShapeOne meaning per field, written down.
- ReconcileBoth sides compared, daily.
- AlertOn silence too, not only on errors.
Land keeps the raw copy because reprocessing is the only cheap fix. Once a transformation has overwritten the original, a bug found in June cannot be corrected for April.
Agreeing what a field actually means
The technical work is rarely what stalls a data project. The stall is two departments using the same word for different things, and neither knowing it.
Sales counts a customer from the signature. Finance counts from the first payment. Both are correct, both are quoted as customer count, and the gap between them looks like a bug in the pipeline. Fixing it means writing the definition down and picking an owner, which is an organisational decision the pipeline cannot make for you. You get these surfaced early. They are cheap to settle in week one and expensive to argue about in front of a board.
- Each fact has one owning system, named in writing.
- Money is stored as whole units, never as a floating-point number.
- Timezones are handled once, at the boundary, not in each report.
- A repeated delivery of the same file does not double the numbers.
What we build on
Stores
Movement
Run and watch
Sources we have pulled from
Systems where the data had to be right
“TankAware has revolutionized how we manage petroleum sites. The real-time data and automation have exceeded expectations.”
Blake Sutherland · President, Sutherland Excavating Ltd.
The usual pipeline against ours
Both move data from one place to another. They differ on the day the data is wrong.
| Criterion | The usual approach | How we build |
|---|---|---|
| A bad batch arrives | Loads anyway. The dashboard renders. | Stops at the check, with a report naming what failed. |
| The job stops running | Noticed when a report looks flat. | An alert fires on silence, within the hour. |
| The same file arrives twice | Every number doubles. | Recognised and ignored, because delivery is not counted as truth. |
| A bug is found in old data | Unfixable. The original was overwritten. | Reprocess from the raw copy, which was kept for this. |
| Two systems disagree | Argued about in a meeting. | Reconciled nightly, with the owning system named in advance. |
01Do we need a data warehouse, or is this something smaller?
Most teams asking for a warehouse need three things fixed first: one owner per fact, checks on the way in, and a daily comparison between systems. A warehouse with those problems still gives you two answers to the same question, on nicer hardware. You get the map of where your numbers disagree today, before anyone recommends a platform.
02Have you done this for a data-specialist client?
Our data engineering work sits inside the products we build rather than as standalone warehouse projects, and it is worth being straight about that. TrialTriage moves de-identified patient records through a queue with an audit trail across 23 tracked actions. TankAware turns live IoT tank readings into operational reporting. Judge us on those two, not on a category label.
03How do you find out that a pipeline stopped?
Alert on silence, because a stopped pipeline never throws an error. Every job reports that it ran and how many rows it moved. A job that goes quiet raises an alert on its own. Waiting for a person to notice a flat chart costs you a week every time.
04Our two systems report different revenue. Where do we start?
Start by writing down what each system means by the word, because the difference is usually a definition rather than a defect. One counts at signature, the other at payment, and both are internally consistent. Once the definitions are on paper, a nightly comparison tells you whether a real gap remains.
05Can you work with the data platform we already pay for?
Yes, and we would rather do that than move you. Replacing a working platform is expensive and rarely fixes the actual problem, which is usually ownership and checks. You only hear a change proposed when the current tool cannot do something the workflow genuinely needs.
06Who can see the data once it is all in one place?
Only the people your rules allow, and the rule belongs on the data rather than on the dashboard. Trading CoPilot uses row-level security in Postgres so a query without the right context returns nothing at all. A permission enforced only in the interface is one API call away from being bypassed.
07How long before we can trust the numbers?
Trust arrives in stages, and the first is quick. You learn which system owns each fact, and get a daily report of where two systems disagree. That alone ends most of the arguments. Closing the actual gaps takes as long as the underlying process problems take to fix, and some of those are not engineering problems at all.
08Is this worth doing for a startup, or only at enterprise scale?
Both, at different sizes. A startup needs money recorded correctly and one place that counts customers, which is a week of care rather than a project. An enterprise needs the audit trail, retention rules and access control that a regulated build like TrialTriage carries. The cost of skipping it scales with how many people quote the number.
Related
- RAG development →When the data layer feeds a model.
- API integration →Getting the records between systems in the first place.
- Reconciliation →The daily check that two systems still agree.
- Postgres vs MongoDB →The store decision underneath all of this.
- Why you never use floats for money →The rounding error that shows up in reconciliation.
