Why sending notifications needs a queue (and retries that don't duplicate)

The first version of notification code in almost every system looks like this:

app.post('/orders', async (req, res) => {
  const order = await createOrder(req.body);
  await sendLineMessage(order.customerId, `Order ${order.id} received`);
  res.json(order);
});

It works, until one day it doesn't. This article is about how it breaks and the minimum structure that stops it breaking — written from what I actually ran into building Ting, not from theory.

Failure 1: the downstream API is slow, so you are slow

sendLineMessage is an HTTP call to somebody else. The day LINE takes 5 seconds to answer, your customer taps "buy" and waits 5 seconds. The day LINE returns 500, your order "fails" even though it's already in the database.

Rule one: accepting the work (saving the order) and sending the notification must not share a request. The request should only record that a send is needed and return immediately. The actual sending is a separate process working off that record.

That's a queue. It doesn't have to be Kafka or RabbitMQ — a database table with a status column is a queue, and for most systems it's enough.

Failure 2: a failed send is simply lost

No queue means one attempt; miss and it's gone. A network blip, a rate limit, a token that's briefly expired — the message vanishes and nobody knows.

With a queue, retrying becomes routine: a failed job goes back to "pending" with a time for the next attempt. But a good retry has two conditions:

• Growing gaps — 30 seconds, 2 minutes, 10 minutes. Not a burst every second until you're banned.
• A cap — after 5 attempts, stop and tell a human. Not retrying forever in silence.

Failure 3: duplicates

Worse than a lost message, because the customer sees it and remembers.

The classic scenario: one job is "send to 2,000 recipients". The worker gets through 800 and restarts (deploy, crash, machine reboot). The job is still marked "sending". A new worker picks it up and starts from recipient number one.

The fix isn't "don't restart", because you don't control that. The fix is:

1. Track at the recipient level, not the job level. A table records that job X reached recipient Y on channel Z. Check it before each send.
2. Stuck jobs must be detectable. A worker holding a job updates a heartbeat periodically. If the heartbeat stops for N minutes, treat the holder as dead and put the job back in the queue — and because of point 1, the resumed run skips everyone already sent.
3. Shut down cleanly. On SIGTERM, stop claiming new jobs, finish the current recipient (not the whole job), then exit.

In Ting I tested this by sending to 2,000 real recipients and restarting the server halfway through, twice, before I was willing to write "no duplicates". If you build your own, I'd recommend actually running that test rather than reading the code and trusting it.

Failure 4: two workers grab the same job

As the system grows you'll run more than one worker, and two of them SELECT ... WHERE status = 'pending' at the same moment, get the same row, and you have duplicates of a different kind.

On PostgreSQL the answer is very short:

UPDATE send_queue
SET status = 'sending', claimed_by = $1, claimed_at = now()
WHERE id = (
  SELECT id FROM send_queue
  WHERE status = 'pending' AND next_attempt_at <= now()
  ORDER BY created_at
  FOR UPDATE SKIP LOCKED
  LIMIT 1
)
RETURNING *;

FOR UPDATE SKIP LOCKED lets each worker take a different row with no central lock. Add workers indefinitely without touching the code. Other databases have equivalents (SQL Server uses READPAST).

Failure 5: you can't see what happened

All of the above is in place, but there's nowhere to look. The day a customer says they didn't get it, you're still opening server logs.

The queue table should be something you can view on a screen: which jobs are waiting, how many recipients succeeded, how many failed and why (keep the real error from the far end — e.g. LINE saying the user blocked you), and the actual content that went out.

The minimum structure

As a table:

What you need	What it fixes
Queue table + worker in its own process	Requests no longer slow down with the downstream API
Backoff retry with a cap	Lost sends / retry storms that get you banned
Per-recipient delivery table	Duplicates after a restart
Heartbeat + stuck-job recovery	A dead worker leaving jobs stuck forever
SKIP LOCKED	Multiple workers claiming the same job
Per-recipient, per-channel result log	Being able to tell a customer what happened

All of it is one to two weeks of work if you've done it before, and one to two months if you haven't (most of that goes on the edge cases in failure 3). That's why I made it shared infrastructure in Ting — not because it's theoretically hard, but because it has to be rebuilt in every project and the same mistakes get made every time.

If you're building your own, I hope the table above lets you make fewer of them than I did.