Webhook best practices: acknowledge fast, process later

By mid-2017, Freebird Rides was receiving webhooks from a long list of vendors: Stripe, Plaid, Twilio, Zendesk, Uber, Lyft, Branch, AppsFlyer, Mixpanel, and GitHub. Every one of them has its own payload format, retry policy, and opinion about how quickly you should answer. The webhook best practices we landed on were simple.

Our answer was a small repo of Lambda functions on the Serverless Framework, one receiver per vendor, all following the same pattern.

The pattern: acknowledge, then SNS fan out

Each receiver does three things: tag the event with where it came from, publish it to an SNS topic, and return 200. That's it.

lib/AppsFlyerWebhookReceiver.js

handle(event) {
  const response = { statusCode: 200, body: '' }
 
  return this._broadcast(event)
    .then(() => response)
    .catch(err => {
      Logger.log('Caught Error', err)
      return response 
    })
}
 
_broadcast(event) {
  event.webhookType = WEBHOOK_ID
  return this.sns
    .publish({ TopicArn: this.snsTopic, Message: JSON.stringify(event) })
    .promise()
}

The real work (updating a ride, recording a payment, syncing a support ticket) happened in whichever services subscribed to the topic. The receiver doesn't know or care who they are.

Why always 200

Look at the highlighted line: even when publishing fails, the receiver returns 200. One of my commits from November 2017 is literally titled switching to a 200 for appsflyer.

That looks wrong at first. It's deliberate. Most webhook senders retry on anything that isn't a 2xx, some of them aggressively, and some will eventually disable your endpoint if it keeps failing. If your receiver does real work inline, a slow database or a bug in business logic turns into a retry storm from the vendor, which makes the slow database slower.

Separating "we received it" from "we processed it" means:

The vendor sees a fast, boring success, every time.
Failures are ours to handle, in our logs and our retry logic, on our schedule.
Adding a consumer never touches the receiver. New analytics pipeline? Subscribe it to the topic.

The trade is that you need good logging on the receiver, because the vendor will no longer tell you when something went wrong. That's a trade worth making.

Fixtures from real payloads

The other thing this repo did well was testing. Every receiver had a test, and the tests ran against real payloads captured from each vendor and saved as fixtures: a Branch install event, an Uber receipt, GitHub PR events for feature and hotfix branches.

Vendor documentation is often out of date. A captured payload isn't. When a vendor changed a field, the fixture was the first place we noticed.

What I'd do today

The shape holds up. Today I'd reach for a managed queue with built-in retries and dead-lettering (on Vercel, Queues), verify every vendor's signature at the edge, and store the raw payload before anything else touches it. But the core idea, acknowledge fast, process later, and let the bus decouple senders from consumers, is the same one we landed on in 2017.