Redis
Excellent glue, risky foundation
We run Redis in four systems for four different jobs: a Celery broker, a Socket.IO adapter across servers, BullMQ queues, and session cache. None of them store their record of truth in it.
The short answer
Redis is the right default for queues, caching, and real-time state shared across servers, but the wrong place to keep data you cannot afford to lose.
Choose differently when the data is your system of record. Redis persists to disk, but its defaults trade some durability for speed, and a queue backlog under memory pressure behaves worse than a slow database. If a lost job or a dropped session would hurt a customer, put a durable store behind it and let Redis sit in front.
What decides this
4 things that decide this
- 01Redis is an in-memory data store, so reads and writes are fast because they rarely touch disk during normal operation.
- 02It is the standard broker for Celery and BullMQ, and the standard adapter for scaling Socket.IO across more than one server.
- 03Persistence is configurable, not automatic durability. The default snapshot interval can lose the last few seconds of writes on a crash.
- 04A queue backlog grows in memory. Left unmonitored, it competes with the cache data on the same instance until something gets evicted.
A fast, single-purpose data store, not a database replacement
Redis keeps data in memory and serves it back with very low latency. It supports a small set of structures: strings, lists, sets, sorted sets and hashes, plus a publish-subscribe mechanism other services can listen to.
That narrow feature set is the point. Redis does one thing well: hand back a value fast, to any process asking. It is not built to enforce relationships between records or run complex queries. It does not guarantee every write survives a crash the way a transactional database does. Teams that forget this end up treating it as their database, then find out the hard way what it drops under pressure.
Where it holds up and where it does not
Strengths
- As a Celery and BullMQ broker, it moves jobs between a web process and a worker with very little overhead, which is why ZhoopZhoop and TrialTriage both run background work through it.
- Its pub-sub model is what lets Socket.IO fan messages out across more than one server. Elevent uses this to keep a trivia game in sync for over a thousand simultaneous players.
- Session and lookup data that changes often and needs to be read fast belongs here rather than round-tripping to Postgres on every request, which is how Cruise Search keeps its search flow responsive.
- Operating it is simple compared with a clustered database. One process, one config file, and most problems show up as memory pressure rather than obscure locking bugs.
Trade-offs
- Persistence is a setting, not a guarantee. The default snapshotting can lose the last few seconds of writes on a crash, which is acceptable for a cache and not for a queue holding jobs nobody has retried yet.
- Under a queue backlog, memory pressure hits the cached data on the same instance. We now run job queues and cache on separate Redis instances after seeing this happen once.
- It has no native way to enforce that a job runs exactly once. Celery and BullMQ both build idempotency handling on top, and skipping that step is how a webhook or a charge gets processed twice.
- A single-threaded command loop means one slow command, like scanning a huge key, blocks every other client waiting behind it.
Four jobs, four different reasons to reach for it
ZhoopZhoop's voice agent takes calls, books appointments and works through a queue of supplier calls to collect parts quotes. Celery reads that queue off Redis, so a slow supplier call never blocks the next incoming customer call. TrialTriage matches oncology patients to clinical trials, and BullMQ handles the background matching work the same way, on Node instead of Python.
Elevent runs live trivia for corporate events, sometimes with over a thousand players in one room. A single Node server cannot hold that many open connections. The game runs across several servers instead, with Socket.IO's Redis adapter keeping every player's state in sync no matter which server they land on. Cruise Search uses Redis more plainly, as a cache in front of an AI-driven search flow. A repeated query does not re-run the same lookup.
- 01Put queues and cache on separate Redis instances once either one has real load, so a backlog in one does not evict the other.
- 02Set an explicit maxmemory eviction policy rather than accepting the default, so you choose what gets dropped under pressure.
- 03Build idempotency into the job itself. Redis will hand a worker the same job twice under some failure conditions, and the job needs to survive that.
- Backlog growsWorkers fall behind producers
- Memory pressureQueue and cache share one instance
- Eviction kicks inCached keys or queued jobs get dropped
- Retry stormLost jobs get resubmitted at once
- No idempotencyA retried job runs twice
None of these failures are unique to Redis. They show up whenever a queue and a cache share memory with no isolation and jobs assume they will only run once.
What teams ask before committing
01Can Redis replace a database?
For most systems, no. Redis can hold your only copy of data, but its defaults favour speed over durability. It is not built to enforce relationships or guarantee every write survives a crash. Keep the record of truth in Postgres or another transactional database, and use Redis for what changes fast and can be rebuilt if lost.
02Redis or a database-backed queue?
Job volume decides it. Redis is faster and the standard choice for Celery, BullMQ and similar libraries. A database-backed queue is slower, using Postgres row locking instead. It gives you the same durability as the rest of your data, with no second system to operate. We reach for Redis once volume is high enough that speed matters, and a lost job can be rebuilt.
03Does Redis lose data?
It can, depending on configuration. The default snapshot interval writes to disk periodically rather than on every command, so a crash between snapshots loses the writes in between. A stricter durability mode exists and costs write throughput. Decide per use case: a session cache can afford to lose a few seconds, a payment queue cannot.
04How do we scale it past one instance?
Redis Cluster shards data across multiple nodes once one instance's memory or throughput runs out. On our current builds, separating queues from cache onto their own instances has kept memory pressure under control well short of that point. That split is the first move, before reaching for cluster mode.
Related
- Supabase in production →The Postgres-based backend Redis pairs with as a cache layer.
- Hire Python developers →Backend engineers who have run Celery and Redis together in production.
- Hire Node.js developers →Engineers who have run BullMQ and Socket.IO's Redis adapter.
- API integration →Where queues and webhook handling get designed, not bolted on.
