Batch Files Still Run the Insurance Industry
The pitch deck says real-time API. A carrier's systems team says an SFTP drop at midnight. Only one of those is how the work actually moves.
The short version
4 things that decide this
- 01Most carriers still exchange data as flat files over SFTP on a nightly or weekly schedule, not through a live API.
- 02A vendor who insists on real-time integration is asking the insurer to rebuild its own back office to fit the vendor's demo.
- 03Batch does not mean slow. A well-built batch pipeline processes thousands of records overnight and reports results before the next business day starts.
- 04We built TrialTriage's insurer path around batch upload for this reason: bulk CSV files, processed in parallel, with progress visible over a live connection while the batch runs.
What the pitch deck assumes
An AI vendor's demo shows a single record going in and a decision coming back in under a second. That shape sells well, because it looks like the future. It also assumes the carrier has a modern API and a sandbox to test against. It assumes their team is free to build a live integration this quarter.
Most carriers do not have that. Core policy and claims systems in this industry were built decades ago. Replacing them is a multi-year program most insurers are not running. The data still moves the way it moved in the 1990s: a fixed-width or CSV file, dropped on an SFTP server on a schedule. A job on the other end picks it up and loads it. Ask a carrier's systems team for a webhook and they will ask you what file format you need instead.
Fighting the batch file, or building for it
A vendor who treats the batch file as a problem to route around usually asks the carrier to change first. Adopt a new API. Stand up a webhook endpoint. Assign an integration engineer. That is a real ask, and it stalls the deal while procurement and IT argue over who owns the work.
A vendor who builds for the batch file meets the carrier where its systems already are. The insurer drops a file on a schedule it already runs. Your platform picks it up and processes every row. Results come back in a format the carrier's team can load into its own system, with no new integration project. Nothing on the carrier's side has to change for the pilot to start.
- 01Real-time architecture built first, with batch bolted on as an afterthought, usually loops through one record at a time and times out under real volume.
- 02Batch built first scales to thousands of rows, because the pipeline was designed for volume from day one.
- 03A carrier's IT team can approve a batch pickup from a known SFTP path far faster than a new inbound API integration.
How this played out on a real build
We built TrialTriage, an AI platform that matches oncology patients to clinical trials, with a nurse reviewing and finalizing every match. Insurers needed a way to check trial eligibility for large groups of members at once, not one patient typed in at a time.
So the insurer path is batch first. An insurer uploads a CSV file. Redis-backed jobs process the rows in parallel, running each patient through the same matching pipeline a nurse would use one at a time. The insurer watches progress over a live connection while the batch runs, then gets a report back. No insurer had to build a new API to use it. They uploaded a file, the way their own systems already produce one.
Questions this raises
01Why do insurance companies still use batch files instead of APIs?
Core policy and claims systems at most carriers predate modern APIs by decades, and replacing them is a multi-year program few insurers are running. A nightly SFTP file drop is the integration method those systems already support, so it stays the default even as newer tools arrive.
02Is batch processing too slow for AI-driven insurance workflows?
Not in practice. A batch pipeline built to run in parallel can process thousands of records overnight and return results before the next business day starts. The slowness people picture comes from a single nightly schedule, not from batch processing itself. That schedule can run more than once a day if the business needs it to.
03How do you integrate an AI platform with a legacy insurance system?
Start with whatever file exchange the carrier already runs. Usually that is a scheduled SFTP drop in a fixed format, so build the pipeline to consume that at volume. A real-time API can come later for the carriers that have one. Starting there instead excludes most of the market on day one.
