The 90% latency rewrite
Rebuilding a field service backend around how it was actually read
- Organisation
- Clean Life Company
- Role
- Backend Engineer
- Period
- Dec 2024 — present
- Published
- 30 August 2026
- NestJS
- TypeScript
- PostgreSQL
- Redis
- BullMQ
- TypeORM
- Zoho FSM
A backend gets slow in two ways. Something in it is doing too much work, or it is doing the right amount of work in the wrong place. This one was doing both, and each one hid the other.
Context
Clean Life runs its field service operations on a platform integrated with Zoho FSM: jobs go out to technicians, and the records those jobs produce come back. I work on the backend behind it — the data layer, the payment integration, and everything that happens outside the request cycle.
The platform had been built quickly and it worked. That is usually the point at which a data model starts charging interest on the decisions that got it shipped.
The problem
Reads had become the bottleneck. Fetching data — the operation somebody is always waiting on — was the slowest thing the system did.
Latency of this kind is not spread evenly. It lands on whoever is looking at a screen, which in a field service platform is a dispatcher deciding what happens next. And it was structural: the cost grew with the data, so it was going to get worse on its own, and no amount of tuning above the data layer was going to reverse that.
Where the time was going
Four causes, and they compounded:
- The schema had grown around writing. Records were shaped by how they arrived rather than by how they were queried, so an ordinary read had to reassemble them.
- The indexes did not match the access paths. Queries that looked cheap were scanning.
- The same data was fetched repeatedly, and every fetch paid the full price.
- Work that did not need to happen inside a request was happening inside it, so the request waited for it.
Any one of these is survivable. Together they meant a single read could be slow for four independent reasons, so fixing one of them barely moved the number — which is exactly why the problem had lasted as long as it had.
Request path
Client
NestJS API
PostgreSQL
Write-shaped schema, indexes that miss the access paths
Heavy work
Runs inline; the request waits, and a failure surfaces to the user
Every read pays for the schema, the missing index and the inline work, on every request.
Diagnosis
With four compounding causes, the order of the work is the whole method. Change several things at once and the measurement tells you nothing: the number moves, and you cannot say which change moved it — or whether one of them made things worse while another hid it.
So: reads first, because that is where the pain was reported. Then the question that splits the problem in half — is the time being spent inside the database, or around it? Everything after that is elimination. One change, measure, keep it or take it back out.
Constraints
- The system was live. Field operations do not pause for a rewrite.
- The Zoho FSM integration was fixed. It defines part of the data shape and was not open for renegotiation.
- Whatever replaced the schema had to be reachable by migration from the schema that already existed, with production data in it.
- It had to stay a system the team could work on. A faster backend that only one person understands is a worse backend.
What changed
- The PostgreSQL schema was redesigned around the read paths — how the data is actually asked for, rather than how it arrives.
- Indexes were built for those paths, once the paths were stable enough to be worth indexing.
- A Redis cache went in front of the results that could tolerate being slightly out of date, with a standardised key convention so that invalidation is something you reason about rather than search for.
- Everything that did not have to happen inside a request moved onto BullMQ queues, with retry handling and distributed locks so that two workers never take the same unit of work.
The ordering is not incidental. Caching a bad query hides it. Indexing a schema you are about to change is wasted work.
Request path
Client
NestJS API
Redis cache
Standardised keys; serves what can tolerate staleness
PostgreSQL
Read-shaped schema, indexed for the paths that run
Outside the request
BullMQ queue
Workers
Retries and distributed locks; one unit of work, one worker
PostgreSQL · Zoho FSM
A failure here is retried, not shown to somebody
The request path is now only the work a person is actually waiting on.
Key decisions
- 01
Change the data model before the code
Application-level fixes to a data-model problem buy you a constant factor. The schema decides what the database can do cheaply, and nothing written above it can undo that.
Trade-offA schema change is a migration against live data, which made it the riskiest part of the project. It bought the largest improvement and carried the largest risk, and those were the same decision.
- 02
Cache selectively, and standardise the keys
Only results that can tolerate being briefly out of date went behind Redis. The value is less in the cache than in the convention: every key follows the same shape, so invalidating one is a decision you can make rather than a search you have to run.
Trade-offEvery cached value is a promise to invalidate it correctly. Data that is stale when it should have been fresh is a much harder bug to see than a slow query.
- 03
Move work out of the request instead of speeding it up
Work nobody is waiting on should not sit in the path they are waiting on. Queued work also gets something an inline call never gets: a retry.
Trade-offThe API now returns before the work has finished, so the product has to represent “in progress” honestly. And failures move off the user's screen into a queue that somebody has to be watching.
- 04
Lock the unit of work, not the code path
Distributed locks keyed on the work itself, following the same naming convention as the cache, so the same unit cannot be picked up twice no matter how many workers are running.
Trade-offA lock is a bet on its own expiry. Too short and duplicates come back; too long and a worker that dies holds a job hostage until the lease runs out.
Performance
≈90%
reduction in data-retrieval latency
A relative measurement, and the aggregate effect of all four changes rather than any one of them. The absolute figures are internal to Clean Life and are not published here.
- Before
- 100
- After
- 10
Relative data-retrieval latency, indexed to before = 100
| Before | After | |
|---|---|---|
| Data-retrieval latency | The baseline | Down by around 90% |
| Schema | Shaped by how records are written | Shaped by how they are read |
| Indexes | Did not match the queries that ran | Built for the access paths that run |
| Repeated reads | Went to PostgreSQL every time | Served from Redis where staleness is acceptable |
| Non-essential work | Inside the request | On BullMQ queues, with retry handling |
| Concurrent workers | Race conditions we had been seeing | Distributed locks; the races stopped |
| Job failures | Recurring | Dropped |
Trade-offs
- Redis is now on the path of cached reads and of every lock. It is a dependency the system did not have before, and it can fail.
- Some data is briefly stale by design. Deciding which data was a product decision as much as a technical one.
- Asynchronous work is easier to operate and harder to read. Following one unit of work now means following it across a queue boundary.
- The schema is better to read from and slightly more work to write to. That is the right side of the trade for this system, and it would be the wrong side for a different one.
Result
- Data-retrieval latency came down by around 90%.
- The work that used to make requests slow now runs on queues, where a failure is retried instead of shown to somebody.
- The race conditions we had been seeing stopped, and job failures dropped.
- The same job system now carries scheduled work: jobs can be registered and rescheduled at runtime without restarting the server, and every run is logged.
What I took from it
- The time was in the data layer, not the application code. That has been true of every latency problem I have worked on since, which is why it is the first place I look.
- Moving work out of the request path beat making the work faster. The fastest version of a task is the one nobody is waiting for.
- The convention around a cache matters more than the cache. Standardised keys are what turn invalidation from a hunt into a decision.
- Migrations are the dangerous part of a change like this. That is a large part of why, when I review code now, I read a migration more slowly than anything else in the diff.