Data engineering
The number on the dashboard should survive being questioned
Our engineers build the pipelines under IoT fuel telemetry, multi-source ESG research and queue-backed clinical matching. When a figure looks wrong, they can show you where it came from.
What you are getting
4 things that decide this
- 01Senior engineers who have moved data from sensors, documents and third-party APIs into systems people make decisions with.
- 02They build for the day a source goes quiet or sends nonsense, because both happen and neither should silently change a total.
- 03You interview them yourself before anyone starts, and no is an acceptable answer.
- 04Pipelines, schemas and transformation logic live in your repository from the first commit.
What a data engineer actually does here
Most reporting problems are not reporting problems. They are ingestion problems that nobody noticed for a month.
TankAware collects tank levels from IoT sensors at petroleum sites. Sensors drop offline and come back with a gap, so the pipeline has to know the difference between no reading and a reading of zero. Greenlight pulls 10 to 15 independent sources for each of more than 50 ESG topics, then scores them, which means source quality has to be tracked rather than assumed. TrialTriage moves matching work through Redis job queues so a slow batch never blocks the application.
One habit runs through all three. Record what arrived, what you did to it, and when, so any number can be traced back to a source.
What they build that stops the arguments
Loads that can be run twice
A rerun after a failure must not double the totals. Idempotent loads are the difference between a safe retry and a manual repair at month end.
Checks on the way in
Row counts, ranges and freshness tested before data lands. A source that quietly starts sending empty files should page someone, not just look calm on a chart.
Missing treated as missing
A gap in sensor data is not a zero. Encoding that difference correctly is what stops an outage becoming a reported drop in usage.
Backfills that are safe
History reprocessed without disturbing what is live, so a fixed bug can be applied to old data on purpose.
Traceable numbers
Every figure links back to the rows it came from. This is the first thing anyone asks when a total looks wrong, and the answer should take minutes.
- SourceSensor, API or document
- Land rawKept exactly as received
- ValidateRange, count, freshness
- TransformRules in version control
- ServeWhat the dashboard reads
- TraceAny figure back to its rows
Keeping the raw landing in box two is what makes box six possible. Transforming on the way in saves storage and destroys your ability to answer questions later.
Pipelines feeding decisions people act on
“TankAware has revolutionized how we manage petroleum sites. The real-time data and automation have exceeded expectations.”
Blake Sutherland · President, Sutherland Excavating Ltd.
How hiring works
- 01
Tell us which number is disputed
A free call about the sources, the reports and the figure people argue about. If your problem is a definition rather than a pipeline, we say so.
- 02
Meet the engineers
We shortlist people who have run pipelines that failed and recovered, and you interview them against your own bar.
- 03
They embed
Your repository, your standups, your warehouse. One of our engineers owns the pipeline and the freshness alerts.
- 04
They hand over
Documented sources, the transformation rules and a runbook for a failed load, with someone on your team trained to use it. Where a client prefers we keep watching the loads, we stay on under a service level we agree.
Stack
Pipelines
Storage
Practices
Bring us the report nobody believes
Show us the figure people argue about and where it comes from. The scoping call is free, and you will leave knowing whether the pipeline is wrong or the definition is.
01How is a data engineer different from an analyst?
A data engineer builds and runs the pipeline that delivers trustworthy data; an analyst uses it to answer business questions. If your reports are late, wrong or unexplainable, that is engineering. If the data is sound and you need someone to interpret it, you want an analyst and we will say so.
02Do we need a warehouse, or is our database enough?
Your production database is often enough for a long time, and moving early adds cost with no benefit. A separate warehouse earns its place when reporting queries start slowing the application, or when you need to join sources that live in different systems. We look at your query load before recommending a migration.
03Can they work with data that has to stay on our infrastructure?
Yes. Pipelines can run inside your cloud account or your network, with nothing leaving your boundary. That constraint shapes tool choice, so it is worth stating on the first call rather than discovering it during a build.
04What happens when a third-party API changes without warning?
Validation catches it at the boundary and the load stops rather than writing bad rows. Because the raw payload is kept as received, the fix is usually a change to the transformation and a backfill, not a scramble to reconstruct what was lost.
05How do you handle personal data in a pipeline?
Minimise what is carried, separate identifying fields from the rest, and restrict who can read them. TrialTriage handles this in a clinical setting with field-level encryption and automated masking. We build to the standard your auditors will apply, and document it so the review is short.
06What drives the cost of pipeline work?
The number of sources and how badly they behave. A clean API with documentation is quick. Undocumented exports, sensors that drop offline and files whose format changes without notice are where the time goes. Scoping calls are free. Where we must map an existing system first, a paid two-week diagnostic produces a fixed price.
