Reliability
How Untimely schedules, delivers, retries, and audits every event.
. This page describes the behavior of the running system, not aspirations. It lives in the same repository as the delivery pipeline and is updated when the behavior changes.
Delivery semantics: at-least-once
Untimely delivers events at least once. Before any action runs, the schedule is claimed by inserting a run record guarded by a unique database index — so two workers can never both deliver the same schedule at the same time. If a run crashes mid-flight (a provider timeout, a database blip, a worker eviction), the claim expires after 5 minutes and the next retry re-executes the run instead of giving up.
At-least-once is a deliberate choice: when the system is forced to pick between a rare duplicate delivery and a silent drop, it duplicates. A second copy of an email or a webhook call is annoying; a reminder that never arrives is a broken product. If your webhook consumer must be strictly effect-once, deduplicate on your side. For event creation, the API supports an Idempotency-Key header so retried requests never create duplicate events.
The pipeline, step by step
- A scheduler wakes every 5 minutes and selects every schedule that is due within a 30-minute window — all of them, not a fixed page, so a burst of due events cannot starve the ones behind it.
- Each due schedule is handed to a message queue. The hand-off is recorded on the schedule, so a lagging queue does not cause the same schedule to be enqueued twice.
- A queue consumer claims the run, executes the event's actions in order (Email, Webhook), records every attempt, and marks the schedule triggered.
- Anything that fails is either retried (below) or recorded as an incident — never silently discarded.
Retry policy and dead-letter handling
A failed delivery message is retried up to 3 times with a 60-second delay between attempts, and each failure files an incident with the error and its context. A run whose worker died mid-execution becomes reclaimable after 5 minutes, so retries do real work rather than skipping an abandoned claim.
A message that exhausts every retry lands in a dead-letter queue. That queue has a consumer whose only job is to file a critical incident: a dead-lettered message is a user-visible dropped delivery, and it is treated as exactly that.
Missed schedules are counted, not hidden
If a schedule ages past its 30-minute delivery window without ever being attempted — which should only happen when the pipeline itself is degraded — it is explicitly marked as missed, appears as a failed run in your event history, and files an incident on our side. Untimely does not quietly reschedule a missed event and pretend it never existed.
Run history and retention
Every run writes two layers of records: the schedule run (claim time, start, completion, error) and per-action attempts (status, attempt number, input, output, error message, timing). Both are visible in the event's run history in the dashboard — filterable to failures — and available through the API at GET /api/events/{id}/schedules/{scheduleId}. Completed schedule records are retained for 30 days and then deleted.
How Untimely detects and files its own incidents
Unhandled failures anywhere in the pipeline — API routes, the scheduler, the queue consumer — flow through one incident system:
- The error is fingerprinted over its source, route, error type, message, and top stack frame, so a repeating failure collapses into one incident with an occurrence count instead of a flood.
- Logs and state snapshots are captured with credentials and email addresses redacted before anything is stored.
- A new fingerprint opens an incident record and automatically files a GitHub issue against our own codebase, including the running commit and a debugging brief. Repeat occurrences comment on the same issue with fresh context.
- Near-identical failures that hash differently are deduplicated a second time at triage, so incidents do not fork.
In practice this means that when Untimely breaks, it files its own bug report — usually before anyone notices the symptom.
What you see when a delivery fails
You do not have to take this page's word for anything: check your own events. Each event's run history shows every scheduled run with its status — succeeded, failed, or missed. A failed run shows which action failed, the attempt count, the duration, and the recorded error message. Failures caused by your own configuration (an unreachable webhook URL, a rejected recipient) are recorded in your run history the same way; they are your signal to fix the destination, and they deliberately do not page us.
Status and limitations
- Untimely runs on Cloudflare Workers, Queues, and D1, distributed across Cloudflare's network — there is no single pinned server to lose.
- We do not publish a formal SLA today. If that changes, it will be documented here first.
- At-least-once delivery means rare duplicates are possible during recovery from a crash. We consider that the correct trade.
- Random-window events promise delivery inside the window you chose — that is the product — not at a secret exact minute.
- Suspect a problem? Your event's run history is the fastest ground truth. For anything else: support@untimely.app.