Batch Files Still Run the Insurance Industry
The pitch deck says real-time API. A carrier's systems team says an SFTP drop at midnight. Only one of those is how the work actually moves.
The short version
4 things that decide this
- 01Most carriers still exchange data as flat files over SFTP on a nightly or weekly schedule, not through a live API.
- 02A vendor who insists on real-time integration is asking the insurer to rebuild its own back office to fit the vendor's demo.
- 03Batch does not mean slow. A well-built batch pipeline processes thousands of records overnight and reports results before the next business day starts.
- 04We built TrialTriage's insurer path around batch upload for this reason: bulk CSV files, processed in parallel, with progress visible over a live connection while the batch runs.
What the pitch deck assumes
An AI vendor's demo shows a single record going in and a decision coming back in under a second. That shape sells well, because it looks like the future. It also assumes the carrier has a modern API and a sandbox to test against. It assumes their team is free to build a live integration this quarter.
Most carriers do not have that. Core policy and claims systems in this industry were built decades ago. Replacing them is a multi-year program most insurers are not running. The data still moves the way it moved in the 1990s: a fixed-width or CSV file, dropped on an SFTP server on a schedule. A job on the other end picks it up and loads it. Ask a carrier's systems team for a webhook and they will ask you what file format you need instead.
Fighting the batch file, or building for it
A vendor who treats the batch file as a problem to route around usually asks the carrier to change first. Adopt a new API. Stand up a webhook endpoint. Assign an integration engineer. That is a real ask, and it stalls the deal while procurement and IT argue over who owns the work.
A vendor who builds for the batch file meets the carrier where its systems already are. The insurer drops a file on a schedule it already runs. Your platform picks it up and processes every row. Results come back in a format the carrier's team can load into its own system, with no new integration project. Nothing on the carrier's side has to change for the pilot to start.
- 01Real-time architecture built first, with batch bolted on as an afterthought, usually loops through one record at a time and times out under real volume.
- 02Batch built first scales to thousands of rows, because the pipeline was designed for volume from day one.
- 03A carrier's IT team can approve a batch pickup from a known SFTP path far faster than a new inbound API integration.
How this played out on a real build
We built TrialTriage, an AI platform that matches oncology patients to clinical trials, with a nurse reviewing and finalizing every match. Insurers needed a way to check trial eligibility for large groups of members at once, not one patient typed in at a time.
So the insurer path is batch first. An insurer uploads a CSV file. Redis-backed jobs process the rows in parallel, running each patient through the same matching pipeline a nurse would use one at a time. The insurer watches progress over a live connection while the batch runs, then gets a report back. No insurer had to build a new API to use it. They uploaded a file, the way their own systems already produce one.
Some of the systems we have shipped
Questions this raises
01Why do insurance companies still use batch files instead of APIs?+
Core policy and claims systems at most carriers predate modern APIs by decades, and replacing them is a multi-year program few insurers are running. A nightly SFTP file drop is the integration method those systems already support, so it stays the default even as newer tools arrive.
02Is batch processing too slow for AI-driven insurance workflows?+
Not in practice. A batch pipeline built to run in parallel can process thousands of records overnight and return results before the next business day starts. The slowness people picture comes from a single nightly schedule, not from batch processing itself. That schedule can run more than once a day if the business needs it to.
03How do you integrate an AI platform with a legacy insurance system?+
Start with whatever file exchange the carrier already runs. Usually that is a scheduled SFTP drop in a fixed format, so build the pipeline to consume that at volume. A real-time API can come later for the carriers that have one. Starting there instead excludes most of the market on day one.
Related
- API integration →The service behind connecting systems that were never built to talk to each other.
- Legacy modernization →Working with decades-old core systems without a rip-and-replace project.
- AI, automation and custom software for insurance agencies →What insurers actually buy, and the constraints that shape it.
- AI, automation and custom software for firms →The wider build across accounting, insurance and staffing.
