notifications system backend implemention ><

notifire: Understanding Queue Adapters Before Writing One

notifire is a lib for your notifications, which is build to handle hundred thousands msgs, emails, otps, etc in just a few lines of code, where postgresql is default, recommended, production backend but also has BullMQQueueAdapter as an optional adapter for Redis users and proof that the abstraction works across different queue implementations. For its channel providers there's nodemailer and twilio's.

if you've to imagine it's architecture, think it like a building:

Basement — The Database (the storage room) This is just a big room just storage, like a garage.

Floor 1— The QueueAdapter (the building manager) Hand out job safely, to the workers available. concepts like idempotency, Polling vs LISTEN/NOTIFY, lease pattern are implemented.

Floor 2 — The Worker (the person who actually does the task) This person takes the card from the building manager, reads it and fills in a template, which is done by handler code.

Floor 3 — The Provider (the mail carrier) Once the worker has the finished message written out, they hand it to the mail carrier, nodemailer or twilio.

Your App / trigger()

--------------------------------------------------------------------------- trigger() call │ Notifire.trigger() ← fan-out lives HERE (chunking loop) {*Here's come a approach which is called fan out so what it does is instead of sending messages to individuals, it creates batches and then send, but then those batch might contain the same info ?? NO that's the catch, it do not change the text just the approach how it will be delivered!*} │ QueueAdapter.enqueueBatch() ← jobs now sitting in Postgres/Redis │ Worker.dispatch() ← pulls ONE job at a time via queue.consume() │ EmailHandler.process() ← renders template, calls provider.send()

here instead of getting into basic stuff I'll get to the core like deeply understanding like

why actually we need a queue adapter, and how can we build one ??

starting with the WHY, think of it's as a contract that says: "I guarantee no two workers ever process the same job at once, and no job gets silently lost if a worker crashes." And to make sure the contract work fine, there's some concepts like ship locked, Visibility timeout / lease pattern, Polling vs LISTEN/NOTIFY, Connection pool, idempotency etc here each make sure that our adapter work best and prevent race conditions, con-currency issues, msg getting lost or delivering twice or not even once, same msg is given to two workers and a few more.

In this I've used PostgresAdapter, BullMQAdapter just to make sure that the archti. is build in such a way that it won't break over changing adapters as queueAdapter only guarantees delivery of the job to you safely (no double-claims, lease semantics); everything past that boundary rendering, calling the provider, deciding retry vs dead-letter is business logic you own and write, which is where the handler part of the system, lives. they both are join with the queue adapter serving there own specialities like in

PostgreSQL,: WAL — Postgres writes your homework in a diary before saying "done," so even if the power goes out, nothing's lost. MVCC — everyone gets their own snapshot photo of the shelf, so nobody bumps into each other while looking. SKIP LOCKED — if a book is already being read, just skip it and grab the next one — no waiting in line. Optimized for — Postgres is the careful librarian who never loses a single book, even during a fire drill, which resembles durability and transactional consistency

BullMQ: High throughput — Redis keeps everything in its head (memory) instead of writing it down, so it's super fast at handing things out. Lock + heartbeat — a kid holding a toy has to say "still playing!" every few seconds, or someone else gets to take it, so worker has to keep saying I'm doin the job and if it didn't and exceed the lease limit other worker is assign to it Redis persistence — Redis jots quick notes as backup, just in case it forgets everything all at once. Optimized for — BullMQ is the speedy runner who can pass out toys to a hundred kids in seconds, but keeps only quick notes instead of a full diary, which is throughput and low latency.

Both guarantee "no two workers grab the same job" just through different mechanisms (SKIP LOCKED vs. lock+heartbeat). So the real choice isn't "which is faster," it's "do I need Postgres's durability, or Redis's speed."

Now after the job hands out safely, it's where worker & job handler get to action, you can consider worker as (the chef) and handler (the recipe) which comprises of the channel providers (nodemailer, twilio..) so, the handler typically contains the logic that calls the channel provider's SDK or API. It acts as the bridge between your queued job and the external service that sends the notification.

For example: Worker → "Here's an email job." Handler → "I'll send this email." Nodemailer → Actually talks to the SMTP server. SMTP server → Delivers the email.

And at last it ended with job model which defines what a notification job looks like, type: email recipient: disha@example.com template: welcome data: { name: "Disha" } priority: high attempts: 3

and Configuration to tells the adapters and providers how to run.

interface PostgresQueueAdapterOptions { appPool: Pool(process.env.DATABASE_URL); which pg connection should be use to enqueue jobs? workerPool: Pool(process.env.DATABASE_URL); should workers use to fetch and update jobs? (Connection pool = A group of reusable database connections to that database, they don't have a separate connection string as they are made as object, eg, const appPool = new Pool({ connectionString: process.env.DATABASE_URL, }); appPool and workerPool are two separate connection pools created from that same connection string to isolate application traffic from worker traffic.)

concurrency?: number; how many jobs can a worker work at the same time? pollIntervalMs?: number;how often should workers check pq for new jobs? retryDelayMs?: number; If a job fails, how long should the worker wait before retrying? maxAttempts?: number;how many times should a failed job be retried before giving up? leaseTimeoutMs?: number; how long is a worker allowed to own a job before it's considered dead and can be reclaimed by another worker? reaperIntervalMs?: number; how often should the "reaper" scan for expired leases (stuck jobs) and recover them? onError?: (error: unknown) => void; which function should be called to handle or log the error? eg: { concurrency: 20, pollIntervalMs: 1000, maxAttempts: 5 }

There are different methods depending on the workload but apart from that there's some typical ways, like when it's come to decide the maxAttempts uhm like 3-5 times retries, exponential backoff (it increase the delaying time after each response), Permanent failures (e.g., invalid email) where there's no more retry or Transient failures (e.g., network timeout, SMTP temporarily unavailable) each come with there own trade off.

To summarize what each one does,

QueueAdapter ├── Interface (contract) -> the operations every queue backend must support, enqueue(), dequeue() or worker registration, ack() (mark complete).. ├── Backend implementation -> actual code that talks to a specific queue system, can be PostgresAdapter, BullMQAdapter etc ├── Worker/consumer logic -> Handles fetching jobs from the backend, executing them, acknowledging success, and retrying failures. └── Job model + configuration -> Job Model defines what a job is, while Configuration defines how a particular adapter runs and connects to its backend. Like what our msg should include (name,template,to,for etc) and the config of our queue adapters.

That's the theory the interface, the guarantees, why Postgres and BullMQ satisfy them differently. It's deliberately theory-first, as it's my first time building something related to the core backend but I'll soon come with my codebase and I'll publish it as a follow-up, once I have actual numbers instead of assumptions, latency etc.

so, yeah it was it. It might be quiet lengthy and has some other flaws but I'll try to make it better with the follow up, thanks for reading, TBC..