Designing a Distributed Job Scheduler That Doesn't Fire Twice (Usually)

"Send this email in 4 hours." "Retry this webhook at 09:12." "Run billing on the 1st." Every backend accumulates deferred work, and the component that owns it — the job scheduler — has a deceptively hard contract: fire on time, fire once, survive the death of any machine involved, and do this for millions of pending jobs. You cannot have all four perfectly. Understanding which one bends (spoiler: fire-once) is the whole design.

A scheduler is three concerns bolted together: storage for pending jobs, a timing mechanism that notices jobs coming due, and dispatch that hands due jobs to workers exactly-ish once. Take them in order.

The database-as-queue design you should start with

For 90% of systems, the right scheduler is a table:

CREATE TABLE jobs (
  id            BIGINT PRIMARY KEY,
  run_at        TIMESTAMPTZ NOT NULL,
  status        TEXT NOT NULL DEFAULT 'pending',   -- pending|running|done|dead
  attempts      INT NOT NULL DEFAULT 0,
  payload       JSONB NOT NULL,
  locked_until  TIMESTAMPTZ
);
CREATE INDEX idx_due ON jobs (run_at) WHERE status = 'pending';

Workers poll for due work, and the entire correctness story lives in one query:

UPDATE jobs SET status = 'running',
       locked_until = now() + interval '5 minutes',
       attempts = attempts + 1
WHERE id IN (
  SELECT id FROM jobs
  WHERE status = 'pending' AND run_at <= now()
  ORDER BY run_at
  LIMIT 10
  FOR UPDATE SKIP LOCKED
)
RETURNING *;

FOR UPDATE SKIP LOCKED is the magic phrase: ten workers can run this concurrently and the database hands each a disjoint batch — no distributed lock, no coordinator, contention handled by the lock manager you already operate. This one clause is why Postgres-backed queues (Sidekiq-style patterns, pg-boss, Oban, Temporal's early internals) are everywhere.

The locked_until lease handles worker death: a crashed worker's jobs become claimable again when the lease expires. And that is precisely where the exactly-once dream dies — a worker that finishes the work but crashes before marking done leaves a job that ran once and will run again. The lease expiry cannot distinguish "crashed before working" from "crashed after." So the contract is at-least-once, therefore handlers must be idempotent. Every serious queue documents this; every team rediscovers it in an incident anyway.

Polling every second is fine at modest scale and the partial index keeps the due-scan cheap. The design runs out of road around thousands of claims per second, when lock contention and poll pressure start to bite — which is far later than most teams assume.

Scaling the timing tier

Past the database's comfort zone, split the roles: a scheduler tier that watches the clock, and execution via a real queue. The scheduler's only job becomes moving due jobs onto the queue; workers never poll storage.

For in-memory timer mechanics, the tool of choice is the hierarchical timing wheel — buckets on a ring indexed by time, O(1) insert and expiry where a naive priority queue pays O(log n) — the structure inside Kafka's delayed-operation purgatory and most OS timer subsystems. For distributed timer state, a Redis sorted set scored by fire-time gives you the same shape: ZRANGEBYSCORE timers 0 <now> plus an atomic pop-and-move (Lua) so a scheduler crash between "read due" and "enqueue" cannot lose or double-fire a job.

Two sharp edges at this layer:

Redundancy without duplication. One scheduler process is a SPOF; two naive ones fire everything twice. Standard answer: partition the timer keyspace across scheduler instances, each elected leader for its partition — leader election via etcd leases, not hand-rolled.

The thundering herd at 09:00. Cron semantics concentrate load — thousands of "daily at 9" jobs fire simultaneously. Add jitter inside the minute, and rate-limit dispatch so the herd drains as a stream rather than a spike. Also decide misfire policy per job: if the system was down at 09:00, does the job run late, or skip? Backup jobs want run-late; "send morning digest" wants skip. Make it a field, not a philosophy debate during an outage.

Retries, backoff, and the dead-letter end state

Failures route back through the same machinery — a failed attempt is just a reschedule:

run_at = now() + base * 2^attempts + jitter     # exponential backoff
if attempts >= max_attempts: status = 'dead'    # dead-letter, alert, inspect

Full jitter on the backoff is not optional at scale: a dependency outage fails thousands of jobs in the same second, and without jitter they all return in the same second too, synchronized forever — a self-inflicted retry storm. A dead-letter state with tooling to inspect and requeue is the difference between an incident being annoying and being archaeology.

Set the lease (locked_until, or the queue's visibility timeout) longer than the slowest legitimate execution, or your retry mechanism becomes a duplication machine: the lease expires mid-run, another worker claims the job, and now it is running twice concurrently — the exact scenario handlers must survive.

Takeaways

A scheduler is storage plus clock plus claim protocol. Start with the database: SKIP LOCKED claiming, lease-based recovery, exponential backoff with jitter, dead-letter after N attempts — boring, transactional, debuggable. Graduate to a timing-wheel or ZSET scheduler tier feeding a queue when claim volume demands it, partitioned with proper leader election. And accept the theorem the lease forces on you: delivery is at-least-once, so idempotency is not a nice-to-have, it is the API contract between your scheduler and every handler it calls.