The Production Outage — Where the Fix Was Worse Than the Bug

One missing property line.

Two production outages.

The hot fix between them made the second one inevitable.

This is the story of an outage where every reasonable next step made things worse — and where the eventual resolution was a hard “no” to the reflex answer “just raise the pool size.” It’s also the companion piece to the previous post on Spring Boot concurrency and connection pools. That one taught the mental model: a transaction holds a connection; the pool is capped; slow I/O inside transactions is fatal. This one is what happens when you try to escape that model by giving each service a bigger pool. The database has its own ceiling, and it’s often lower than you think.

If you skip to one section, skip to the arithmetic in section 5. That’s the single formula every backend team on a shared database should be doing at capacity-planning time, and almost none are.

1. The setup

The backend is a Spring Boot monolith split across four cooperating services on Heroku: api (the main HTTP surface), engine (the async job runner), messaging (outbound integrations), and integration (inbound webhooks). All four talk to the same Postgres. Each runs on multiple dynos — small Heroku process containers with their own JVM and their own Hikari connection pool.

In the environment where this incident happened:

• Postgres max_connections = 200 — the hard ceiling the database will accept before refusing new connections with too many connections. It’s a smaller-tier Heroku Postgres plan than we use for the larger production database (which sits at 500).
• Two dynos total — one web mostly for api, one worker mostly for engine. messaging and integration are sidecars distributed across both dynos. api specifically runs on both dynos (one instance per) because it serves HTTP traffic and needs the web dyno slot.
• spring.datasource.hikari.maximum-pool-size was set explicitly for dev, e2e, and the larger production environment. For this production environment specifically, no override existed.

That last line is the whole bug, and the article’s first surprise is what it actually means.

2. The first outage: Connection is not available

The visible symptom was requests that used to be fast starting to time out. In the application logs, over and over:

java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not
available, request timed out after 30000ms.

This is Hikari’s error, not Postgres’s. It means: “a caller asked me for a connection, I couldn’t hand one over within the acquisition timeout (30 seconds by default), giving up.” All the pool’s connections are checked out — presumably by something holding them for too long — and requests are queueing behind them.

Two hypotheses, both true, interacting badly:

1. The pool was small. How small was about to become the article’s first surprise.
2. Something was holding connections for far longer than the DB work alone justified. Specifically, the Gmail-notification batch job was doing external I/O — Gmail download, Cloudinary upload, OCR extraction — inside an active DB transaction. Each invoice’s processing held its connection for approximately 5 seconds, of which maybe 50 milliseconds was actual database work. The rest was HTTP round-trips waiting on remote services.

Every reader of the previous concurrency primer will recognise that second point instantly. It is exactly the “connection held across slow I/O” antipattern the primer’s section 9 exists to name. A five-second transaction consuming one connection means each invoice is five seconds of unavailability for one of the pool’s ten slots. A modest burst of concurrent invoice notifications is enough to saturate a pool of any small size.

Which brings us to the size the pool actually was.

3. The trace: deploys are also where connection demand peaks

Before we can talk about what fixed the runtime timeouts, one more piece: connection demand isn’t constant across the environment’s lifecycle. It peaks during deploys.

The Heroku deploy model is “boot the new dynos before killing the old ones.” For a brief window — 30-60 seconds while the new dynos become healthy — both generations exist simultaneously. Every connection the old dynos still hold, plus every connection the new dynos are opening to become ready, both count against max_connections. Deploy time is the environment’s connection-demand peak.

That fact will matter more in section 4 than it does here. For the runtime timeouts specifically, deploys aren’t the direct culprit — the pool was saturating under normal load. But to understand what happens after the hot fix, we need this piece already in place.

To fix the runtime saturation, we needed to know the effective pool size on each service. And the ugly discovery here was: nobody knew what the effective pool size actually was in this environment.

The larger production environment’s application-*.properties set the pool explicitly (70 for api, lower values for others). The dev and e2e environments did the same. This production environment’s application-*.properties, though — for reasons lost to a git-blame in 2024 — did not.

The base application.properties, sitting quietly at the root of the config tree, had spring.datasource.hikari.maximum-pool-size=10 set explicitly. That value dated to an early-development-era commit when the “sensible default” it was written to protect against was a local developer accidentally exhausting a tiny local Postgres. It had never been revisited.

Spring’s profile-specific properties override individual keys, not whole sections. Every environment that overrode the pool inherited that override, so nobody in review ever noticed the base value. This environment’s profile didn’t override, so it silently inherited 10.

Two properties files, one line missing from one of them, and the entire environment has been running at ten connections per service for months. Nobody ever wrote “10” while thinking about this environment. Ten was a default someone else chose for a different reason years earlier — and every environment except this one had a profile-level override sitting on top of it hiding that fact.

We had been running a pool of 10 for months.

4. The hot fix, and the outage it made inevitable

The hot fix took six minutes to write. Bump api’s maximum-pool-size from 10 to 40 in the environment’s properties file. Redeploy.

Runtime timeouts stopped. Traffic recovered. The team lead sent the “we’re back” Slack message. It was pre-lunch, morale was high, we’d caught the outage fast.

We had also, without realising it, made the next deploy a bigger problem.

The number we’d picked for the pool wasn’t chosen against a total budget. It was chosen against the previous value in the spirit of “we clearly weren’t running with enough headroom, let’s give ourselves more.” Nobody sat down and did the arithmetic across every service, every dyno, and every deploy transition. And when the next deploy landed — six hours later, a routine PR — the outage came back, this time a completely different kind.

The new error was Postgres’s, not Hikari’s:

FATAL: sorry, too many clients already

The math got interesting. After the hot fix:

• api ran with pool 40, on both dynos → up to 80 connections held during load.
• Other services (integration, engine, messaging) still had pool 10 → about 40 connections combined.
• Total steady-state upper bound: 120 connections.

That’s 60% of the 200-connection ceiling — comfortable in isolation. But the deploy transition is where the arithmetic changes. When api deploys, the old-generation api dynos are still holding their ~80 connections while the new-generation api dynos come online and open theirs. For the overlap window, api alone can hold up to 160 connections. Plus the other services’ ~40. Plus Postgres’s own reserved-for-superuser connections.

Peak: at or above 200. The ceiling.

too many clients from the database. Deploy failed. Apps couldn’t complete startup. The rollback was messy — the new dynos held partial connections that took time to release, and the environment sat at ~90% of the ceiling for several minutes while it drained.

Same environment. Two different errors. Two different root causes. Same underlying cause: nobody was computing the deploy-time connection budget, only the steady-state one.

5. The arithmetic every backend team should be doing

If you take one thing from this article, take this formula. It’s not new. It’s not clever. It is the thing that got skipped, and the thing that gets skipped in almost every production incident of this shape I’ve seen.

For any Postgres-backed backend:

(apps × dynos_per_app × pool_size)  +  deploy_overlap  ≤  max_connections

Where:

• apps is the number of services that share this database. For us: four.
• dynos_per_app is how many process instances of each app run concurrently. For us: two (one web, one worker — but shared across apps).
• pool_size is the effective Hikari maximum-pool-size per app process.
• deploy_overlap is the additional connections held by old-generation dynos while new-generation dynos are starting. Under a rolling deploy, this can be up to another steady-state total’s worth for a 30-60 second window — the worst case is when old dynos haven’t drained yet while new dynos are eagerly opening connections. In practice it’s often less than a full doubling, but the bound is what matters for the arithmetic.
• max_connections is what Postgres will actually allow. Not what it’s tuned for — what it will actually accept. Query with SHOW max_connections; in psql. (A small note: Postgres reserves a few connections for the superuser for admin access, so the effective ceiling is a few below the announced max_connections. Don’t plan for the whole number.)

One important nuance about the pool_size term. This is HikariCP’s maximumPoolSize — the upper bound the pool will grow to under load. Under quiet traffic, actual open connections may be much lower (minimumIdle defaults to maximumPoolSize, but Hikari will close idle connections down to minimumIdle after idleTimeout if it’s set lower). For capacity planning, though, you always compute against the max. Under load, or under a sudden burst, Hikari will grow to it — and it will hit whichever ceiling comes first: the pool’s, or the database’s.

For our production environment before the hot fix (all pools at 10):

Steady state:
  api          (10 × 2 dynos)  = 20
  integration  (10 × 2 dynos)  = 20
  engine       (10 × 1 dyno)   = 10
  messaging    (10 × 1 dyno)   = 10
  Total:                         60

Deploy overlap on api alone:
  new-gen api opening another   = up to +20
  Peak:                           80

Available:                        200

Comfortable — 40% of the ceiling at peak. Which is why the first outage wasn’t about deploy overlap at all; it was about long-held transactions saturating the tiny pool at runtime.

After the hot fix (api bumped to 40, others unchanged):

Steady state:
  api          (40 × 2 dynos)  = 80
  integration  (10 × 2 dynos)  = 20
  engine       (10 × 1 dyno)   = 10
  messaging    (10 × 1 dyno)   = 10
  Total:                        120

Deploy overlap on api alone:
  new-gen api opening another   = up to +80
  Peak:                          200

Available:                       200

Peak equals the ceiling. No margin. Add Postgres’s reserved-for-superuser connections, add any concurrent burst of load, and the deploy trips too many clients.

If you don’t have this arithmetic written down for your environments — for each service, each dyno topology, each deploy transition — you don’t know your capacity. You have opinions about your capacity. On shared databases those opinions are wrong more often than right.

6. The paradox: you cannot copy production’s pool size to a smaller environment

The reflex when someone tells you “the pool is too small” is “look at what production uses and copy that.” The larger production environment we run — with higher-tier customers and a higher-tier Postgres — has pool sizes tuned against its own max_connections. What makes 70 comfortable there is that its database has substantial headroom. Copying that number to a smaller environment copies the “70” without copying the “substantial headroom” it depends on. The math that worked in one place presupposes an environment the other one doesn’t have.

Worse, on the smaller environment, there is essentially no “just raise the pool” answer at all.

Take the naïve assumption that all services deploy simultaneously (worst case) and set every pool uniformly. Against the 200-connection ceiling:

(4 apps × 2 dyno-instances × pool) × 2 (deploy overlap)  ≤  200
16 × pool                                                 ≤  100
pool                                                       ≤  ~6

On this database, no service can have a Hikari pool larger than about 6 under worst-case deploy assumptions. And six connections is not enough for a service under normal steady-state load, let alone bursty concurrent work.

So the article’s title isn’t rhetorical. There genuinely is no right pool number here. “Raise it” fails at the ceiling. “Keep it low” fails at real load. You can shuffle the value around and pick the least bad option, but there is no value at which the arithmetic is comfortable. The problem is not the pool size. The problem is the connection budget.

At least — that’s what the naïve arithmetic says. There’s actually one important knob buried inside Hikari that changes the shape of this problem, and it’s the lever that made the larger production environment survivable. Section 7.

7. The lever inside Hikari that changes the arithmetic: minimumIdle

HikariCP has two knobs for pool sizing, and most people only know about one of them.

• maximumPoolSize — the ceiling. Pool will grow up to this many connections under load. Default: 10.
• minimumIdle — the floor. Hikari’s target minimum number of idle connections in the pool. Default: same value as maximumPoolSize.

That default matters enormously. When minimumIdle == maximumPoolSize, Hikari maintains a fixed-size pool. Even when the service is completely idle, it keeps maximumPoolSize connections open, ready to serve traffic instantly. HikariCP’s own documentation recommends this: “For maximum performance and responsiveness to spike demands, we recommend not setting this value and instead allowing HikariCP to act as a fixed size connection pool.”

That recommendation is written from a single-service perspective. It is exactly wrong when your database is shared. If ten services on the same database each keep their maximumPoolSize open at idle, the idle budget of the database is the sum of ten fixed-size pools — the database is never idle from its own perspective. And at deploy time, when old and new generations coexist briefly, both generations hold their full maximumPoolSize idle.

When you set minimumIdle < maximumPoolSize, everything changes:

• Hikari starts with fewer connections (lazily opened up to minimumIdle at first).
• Under load, the pool grows up to maximumPoolSize.
• When load drops back off, Hikari closes idle connections down to minimumIdle.

For deploy overlap, the effect is dramatic. An idle old-generation dyno holds minimumIdle connections, not maximumPoolSize. A new-generation dyno starts with minimumIdle, then grows as load appears. The overlap peak is approximately 2 × minimumIdle × dynos, not 2 × maximumPoolSize × dynos.

Which is exactly how the larger production environment we run — the one on the 500-connection database — survives with maximumPoolSize=70 for api. It also runs with minimumIdle=25. So at idle, each api dyno holds 25 connections, not 70. During deploy overlap, api alone peaks around 2 × 25 × dynos_per_app + whatever short-lived load spikes happen. Plus the other services. Across the full budget, the observed peak sits around 84% of the 500-connection ceiling — high, but survivable, and the deploys don’t trip.

The setting to add for every service that shares a database:

spring.datasource.hikari.maximum-pool-size=40  # ceiling under load
spring.datasource.hikari.minimum-idle=10       # floor at idle

Two things worth internalising about this:

It’s not free. When Hikari starts with only minimumIdle connections and a traffic spike appears, there is a small latency cost on the first few requests while the pool grows. That’s why HikariCP’s docs recommend the default — for a single-service world where “responsiveness to spike demands” is what you’re optimising for, the default is right.

On a shared database, this trade-off flips. The cost of idle dynos hoarding connections is higher than the cost of a slightly slower first spike. Under-provisioned idle capacity that shrinks is better than fully-provisioned idle capacity that starves your neighbours.

Even with minimumIdle set, the smaller production environment is still tight. Peak deploy-time load will still be 2 × minimumIdle × dyno_instances. On a 200-connection database with minimumIdle=10, the arithmetic still isn’t comfortable — but it moves from impossible to tolerable when combined with the primer’s advice on shortening transactions. Section 9.

8. Making the effective pool size visible everywhere

Before we could argue about which fix belonged where, we needed something more basic: to see the effective pool size across every service in every environment. If a team can’t tell you what pool their production is running with in the current five seconds, they can’t be part of any solution — they don’t have the data.

The naive move is to log the property at startup:

env.getProperty("spring.datasource.hikari.maximum-pool-size")

This is the trap the outage was hiding inside. A library’s built-in default is not a property. If the codebase never sets the key, Environment has no entry for it — the property is “absent,” not “defaulted to 10.” Logging the property tells you what someone wrote, not what Hikari is actually using. The one case where logging matters most — the silent-default case — is exactly the case this approach can’t see.

The correct move is to stop asking the configuration text and ask the live bean:

HikariDataSource hikari = dataSource.unwrap(HikariDataSource.class);
int effectivePoolSize = hikari.getMaximumPoolSize();

getMaximumPoolSize() returns whatever Hikari resolved to — from a property file, from the library’s default, from a @Bean override, from anywhere. It is by definition the effective value. That’s the number you want in your logs.

That leaves three implementation subtleties, each of which turned out to be a real design decision, not a formality.

First, when to log. ContextRefreshedEvent fires early and can fire multiple times (context hierarchies, manual refreshes). ApplicationReadyEvent fires once, after every bean is fully built and the app is officially ready to serve. That’s the natural place.

Second, how to reach the bean without breaking startup on any service. ObjectProvider<DataSource> is the defensive-injection idiom: iterate zero, one, or many DataSource beans without ever throwing NoSuchBeanDefinitionException on a service that has none. And DataSource.isWrapperFor(HikariDataSource.class) before unwrap(...) skips proxied-but-non-Hikari data sources silently.

Third, where to put the class so every service picks it up. This is where the design choice matters most.

The obvious answer is “put a @Component in the shared module.” That relies on component-scanning. In a real polyrepo or a multi-service monorepo, each service’s @ComponentScan is likely different: some scan com.company broadly, others use explicit allowlists. A plain @Component in a shared module gets picked up by services that scan broadly and silently skipped by services with allowlists — with no error to warn you. Which is precisely the “works in three services, invisible in the fourth” silent-failure this article is about.

The right pattern is Spring Boot’s auto-configuration mechanism, which does not go through component scanning at all:

@AutoConfiguration
@ConditionalOnClass(HikariDataSource.class)
public class DataSourcePoolLoggingAutoConfiguration {

    @Bean
    public ApplicationListener<ApplicationReadyEvent> logEffectiveHikariPoolSize(
            ObjectProvider<DataSource> dataSources, Environment environment) {
        return event -> dataSources.forEach(ds -> logPoolSize(ds, environment));
    }

    private void logPoolSize(DataSource dataSource, Environment environment) {
        try {
            if (!dataSource.isWrapperFor(HikariDataSource.class)) return;
            HikariDataSource hikari = dataSource.unwrap(HikariDataSource.class);
            log.atInfo()
                .addKeyValue("hikari.poolName", hikari.getPoolName())
                .addKeyValue("hikari.maxPoolSize", hikari.getMaximumPoolSize())
                .addKeyValue("activeProfiles", Arrays.toString(environment.getActiveProfiles()))
                .log("Effective Hikari pool [{}] maximum-pool-size={} for active profiles {}",
                     hikari.getPoolName(), hikari.getMaximumPoolSize(),
                     Arrays.toString(environment.getActiveProfiles()));
        } catch (SQLException e) {
            log.error("Failed to read the effective Hikari pool size at startup", e);
        }
    }
}

Plus a one-line registration file — META-INF/spring/org.springframework.boot.autoconfigure.AutoConfiguration.imports — that lists the class. Spring Boot discovers auto-configurations by reading that file, regardless of any service’s @ComponentScan configuration. Every service that depends on the shared module logs its effective pool size at startup, automatically, forever.

The subtleties in one paragraph so you can revisit later without re-reading: @AutoConfiguration is the classpath-registered variant of @Configuration that Spring Boot discovers without scanning. @ConditionalOnClass(HikariDataSource.class) activates the config only where Hikari is present, without throwing NoClassDefFoundError on services where it isn’t (Spring Boot evaluates the condition via bytecode inspection, not classloading). ObjectProvider<DataSource> handles zero-or-many injection safely. isWrapperFor + unwrap drills through Spring’s proxy layers to reach the real HikariDataSource. log.atInfo().addKeyValue(...) emits structured fields into whatever log platform the service uses, so “the effective pool sizes across every service in every environment” becomes a queryable dashboard instead of a text-grep exercise.

That’s the observability tooling. It doesn’t fix the outage. It just makes it possible to have a real discussion about the outage.

9. Why the fix ultimately isn’t just in the pool

Section 7’s minimumIdle lever is real, and it’s necessary for any service on a shared database — but it’s a reduction in idle waste, not an elimination of the underlying problem. If your services are still holding each connection for five seconds while blocking on external I/O inside a transaction, the pool will still saturate under load, no matter what floor and ceiling you configure.

The concurrency primer’s spine bears repeating here:

Every @Transactional method holds a connection for the entire duration of the method call. Anything else the method does while it holds that line — HTTP calls, file uploads, external OCR, sleep — is time the connection is checked out and unavailable to anyone else.

If a service’s connections are being held — meaning: transactions are open, doing slow I/O, waiting on external calls — then the effective concurrency of the service is bounded by the pool. Raising the pool makes more parallel holds possible. But each hold now has to fit inside the database’s ceiling. On a small database, the pool ceiling is not far above the DB ceiling. Raising one gets you a little more; the other slams the door.

The lever that keeps working, once minimumIdle has done what it can, is making each connection held for less time.

Concretely, and in order of impact for real applications:

1. Move slow I/O out of the transaction window. Do the external HTTP call, the file download, the OCR before opening the transaction, or after closing it. Keep the @Transactional block scoped to the writes only. A method that holds a connection for five seconds while OCR runs, refactored to hold it for fifty milliseconds while it just writes the OCR result, needs one hundredth of the connection capacity to serve the same throughput.
2. Move the whole workflow off the request thread. Even before you shorten the DB window, if the request thread is what’s blocking on the five-second call, that request thread is also unavailable to serve other HTTP requests. Use @Async (with an explicit bounded executor), so the request thread returns immediately and the slow work happens on a pool designed for it.
3. Cap the async pool. If a burst of 500 events would trigger 500 concurrent slow workflows, you don’t want 500 in flight — you’d need 500 connections. A bounded executor with a handful of workers and a queue holds bursty demand at the queue level, without borrowing any connections.

The concurrency primer covers each of these in more depth. The relevant part for this article is that all three of them reduce the connection budget the service consumes. And reducing the budget is the only lever that keeps working on a small-database environment where the pool ceiling can’t safely rise.

The lesson at the top of this section is worth stating one more time in blunt form: on a shared database, connection budget is not a per-service question, it’s a per-database question. The four services share a database. The database has a hard ceiling. Every service’s pool comes out of the same pot. The productive question isn’t “what pool size should service X use” — it’s “what’s the total budget for this database, and how do we split it while each service still functions”. That question has a very different answer, and it’s often “nobody can have as much as they’d want; every service has to work harder to hold connections for less time.”

10. What I’d tell another engineer

The reflex when a service can’t get a connection is “raise the pool.” On a database with lots of headroom that works. On a shared database with a real ceiling — which is most non-toy production environments — it works only for a while. The next deploy, or the next traffic burst, hits the ceiling from the other side, and the same error message shows up meaning the opposite thing. If you see too many clients for the first time and reach for the pool as a knob, you’re likely trading one outage for a bigger one four hours later.

Before touching anything, do the arithmetic across every service that shares the database. Add up steady-state demand, double it for deploy overlap, compare it to max_connections. If the number is close, no per-service pool tweak will save you — the fix has to reduce time-under-transaction somewhere, or the database itself has to grow, or a service has to move to its own database. Those are the three real levers.

And on the observability side: “nobody knows what pool size we’re actually running” is not a soft problem, it’s the hard problem underneath the outage. If you cannot answer “what’s the effective pool size for every service in every environment right now” in less than thirty seconds, the outage is going to happen and you’re going to be surprised by it. Building a small auto-configuration that logs the effective values at ApplicationReadyEvent — reading the live bean, not the Environment — took an afternoon and permanently removed “we didn’t know” from every future incident of this shape.

One missing property line, two production outages, and a discipline that starts with the arithmetic. If your team has the arithmetic written down and refreshed every quarter, you probably aren’t reading this article to learn something. If you don’t — that’s this weekend’s PR. It’ll cost you an afternoon and save you the next outage.