SQS long polling workers and MongoDB connection pool hygiene

By 2019 the Freebird API had been running for over two years and its pull request numbers were past 4,000. It was a Hapi.js service backed by MongoDB, and the same codebase ran in two modes: as the HTTP API, and as a fleet of background workers using SQS long polling for webhook events from ride-hailing partners, payment providers, and analytics tools.

Same code, same config, same database. That was convenient, but it meant both workloads also shared one idea of how many database connections they needed.

Two workloads, one MongoDB connection pool size

Mongoose opens a connection pool per process. Our config gave every process the same pool, sized for the API: 25 connections. The API needed that. It served concurrent requests all day.

The workers didn't. A worker pulls a batch of messages, processes them, and polls again. It rarely needs more than a handful of connections at once. But every worker process reserved the same 25-connection pool as the API, and across a fleet of workers those mostly idle pools add up fast against a database's connection limit.

The fix, in October 2019, was to let the process tell us what it was:

src/mongo/index.js

const poolSize = _.includes(['production', 'staging'], process.env.DOCKER_ENV)
  ? process.env.JOBS === 'true'
    ? 5 // limit jobs to only 5 connections
    : 25 // limit hapi to only 25 connections
  : 5 // limit dev cluster to only 5 connections

It had actually been sitting in the file as a commented-out @TODO: lower jobs pool size before it shipped. That's a pattern I'd see again: the fix is often already written down, waiting for a reason to matter.

Connect per poll, not per process

The second change was about lifetime. The workers used to open a connection at startup and keep it forever. Long-lived connections from long-polling processes are exactly the kind that go stale: the database fails over, a network blip drops the socket, and the process keeps holding a connection that no longer works.

So each poll cycle now makes sure it has a healthy connection, does its work, and closes it:

src/webhooks-sqs/utils/sqs-queue-manager.js

try {
  await Mongo.ensureConnection()
  const response = await self.sqs.receiveMessage(params).promise()
  if (response.Messages) {
    // process each message
  }
  await Mongo.closeConnection()
} catch (error) {
  await Mongo.closeConnection()
  Logger.log('self.sqs.receiveMessage threw an exception', { err: error })
  // NOTE: always return true and let the processMessage determine whether to delete it from the queue
}
return true

ensureConnection reuses a cached connection if its readyState says it's live, and otherwise tears down whatever is left and reconnects. closeConnection removes listeners before disconnecting, so a dead connection can't fire events into code that has moved on.

Don't let one error stop the loop

The highlighted line is the subtle one. Before this change, an exception during a poll returned false, and false ended the worker's polling loop. One transient SQS or database error and the worker quietly stopped consuming its queue.

Now the loop always continues. Whether a message is deleted from the queue is decided per message, by the code that processed it. Failures stay on the queue and get retried; the worker keeps working.

What I'd tell 2019 me

Size resources by workload, not by codebase. One repo can run very different processes. Give them different budgets.
Short-lived connections are easier to reason about for bursty background work. Pay the reconnect cost; skip the stale-socket mysteries.
Audit every path that can end a loop. A worker that silently stops is worse than one that crashes, because nothing restarts it.