Billing webhooks: signatures, retries, ordering and replay

A customer pays by SEPA direct debit. The payment provider sends payment.succeeded, your handler marks the invoice paid and unlocks the account. Four days later payment.failed arrives because the bank returned the debit. Then a retry delivers the old payment.succeeded a second time, after the failure. If you apply events in arrival order, the invoice flips back to paid. No dunning, and the account stays open.

In billing, webhooks drive payment status, dunning runs, access rights and bookkeeping entries. This article covers the four places where webhook integrations break in practice: signature verification, retries, ordering, and recovering missed events.

Verify signatures the Standard Webhooks way

A webhook endpoint is a public URL. Anyone who knows it can post an invoice.paid to it. So every delivery has to be signed, and every signature checked before the payload triggers anything.

Many providers have their own signing scheme. The open Standard Webhooks specification unifies this with three headers: webhook-id (unique message ID), webhook-timestamp (Unix time) and webhook-signature (HMAC-SHA256 over ID, timestamp and body, base64-encoded, with a v1, version prefix). Verification in Python looks like this:

import base64, hashlib, hmac, time

def verify(secret: str, headers: dict, raw_body: bytes, tolerance: int = 300) -> bool:
    msg_id = headers["webhook-id"]
    ts = int(headers["webhook-timestamp"])
    if abs(time.time() - ts) > tolerance:
        return False
    key = base64.b64decode(secret.removeprefix("whsec_"))
    signed = f"{msg_id}.{ts}.".encode() + raw_body
    expected = base64.b64encode(hmac.new(key, signed, hashlib.sha256).digest()).decode()
    return any(
        hmac.compare_digest(expected, sig.split(",", 1)[1])
        for sig in headers["webhook-signature"].split(" ")
        if sig.startswith("v1,")
    )

Three details decide whether this holds up in production:

• Use the raw bytes. The signature covers the exact body. If your framework parses the JSON and re-serialises it, key order or whitespace changes and every check fails.
• Compare in constant time. Use hmac.compare_digest, so response timing doesn't leak the signature.
• Accept several signatures. During key rotation the header can carry both the old and the new one. If you only check the first, you drop deliveries for the length of the rotation.

The timestamp plus a tolerance window guards against replay: a captured, valid message can't be resent five minutes later. Keep server clocks in sync or you'll reject genuine deliveries.

Retries are the normal case

Almost every provider delivers webhooks at least once. If your endpoint doesn't return a 2xx within a few seconds, the message is retried, often over hours or days with growing intervals. A handler that spends 20 seconds creating an invoice on the first call and then times out will create another one on the second.

The structure that survives this has two steps:

1. Accept: verify the signature, write the message to a table with webhook-id as a unique key, return 200 immediately. On a key conflict, also return 200, because the message has already arrived.
2. Process: a worker reads pending messages from the table and applies them. Failures here go into an internal retry, not a redelivery from the provider.

The unique message ID is what deduplication rests on. Don't rely on the payload: two different events can concern the same amount and the same invoice. For the same logic applied to usage data that your product sends, rather than receives, see idempotent usage event processing.

Only return an error status when a retry can actually help, for example a 503 because your database was briefly unreachable. An invalid signature won't fix itself on the next attempt, so a 401 or 400 fits better. RFC 9110 defines what the status classes mean, but providers react to 4xx differently: some retry anyway, some disable the endpoint after repeated failures. Read the sender's retry rules.

Ordering comes from the object, not the delivery

Back to the opening example. Webhooks are delivered in parallel and with retries, so arrival order tells you nothing about event order. There are two ways to deal with that.

First, compare versions. If every message carries the timestamp of the state change or a version number of the object, store the highest version applied per invoice and drop older messages.

Second, treat webhooks as a nudge. The handler accepts the message and fetches the current state of the object from the API (GET /invoices/{id}). Arrival order no longer matters because you always work with the latest state. The cost is extra API calls and rate limits.

For payment and invoice status the second pattern is the safer choice, because a wrong intermediate state is expensive there. For events you only log, the first is usually enough.

Recovering missed events

At some point your endpoint will be down for longer than the provider keeps retrying, or a deploy will break signature checks and reject every message for an hour. Then you need a way to find what you missed.

Check whether the provider offers an endpoint that lists events since a given time, or a way to re-trigger failed deliveries. Without either, all that's left is reconciliation: load every invoice changed in the window and compare it with your own state.

A reconciliation job is worth having anyway. Once a day, go through invoices with status open or past_due and compare their status with the provider's. Differences tell you webhooks are going missing before a customer does. These questions belong in a vendor trial; see choosing an API-based billing platform.

Keep tenants apart in the handler

In a multi-tenant product, each event belongs to exactly one tenant. That mapping must not come from a payload field an attacker could set. It should come from a server-side record, such as the mapping of a provider customer ID to your tenant. Where the provider supports it, use a separate signing secret per tenant or per environment, so a leak in staging doesn't open production.

Checklist before go-live

• Signature check on raw bytes, with a tolerance window and support for multiple signatures
• Message ID as a unique key; a duplicate returns 200 and does nothing
• Response within a few seconds, processing asynchronous
• Ordering by object version or by fetching current state, never by arrival time
• A reconciliation job for open invoices and a documented way to backfill missed events
• Tenant mapping held server-side, secrets separated by environment
• Test case: deliver the same message twice, out of order, and ten minutes late

The list applies to any sender. If Kontier is the sender, the Kontier developer documentation is where to start.