Recurring Messages 6.34

Wolverine has long supported scheduled message delivery for one-off, "run this later" messages. opts.Schedules builds recurring, cron-driven messages on top of exactly that machinery: a single agent per cluster keeps the next occurrence of every registered schedule pre-scheduled through the ordinary scheduled-message pipeline, and everything else — delivery, durability, replay after a crash, management tooling — is the infrastructure that already ships.

var builder = Host.CreateApplicationBuilder();
builder.UseWolverine(opts =>
{
    // Simplest possible usage: a message type with a public, no-argument
    // constructor, published on a cron schedule. The schedule's name defaults
    // to the message type's name
    opts.Schedules.ScheduleRecurring<RunNightlyRollup>("0 2 * * *");

    // Or build the message per occurrence — the factory is handed the
    // occurrence time, so a message can describe the window it covers
    opts.Schedules.ScheduleRecurring(
        "daily-report",
        "0 9 * * *",
        occurrence => new BuildDailyReport(occurrence.AddDays(-1), occurrence));

    // Cron expressions parse into a first class value type that you can
    // construct, hold, and reuse — including with an explicit time zone
    var nineAmCentral = new CronSchedule(
        "0 9 * * *",
        TimeZoneInfo.FindSystemTimeZoneById("America/Chicago"));

    opts.Schedules.ScheduleRecurring<SendMorningDigest>(nineAmCentral);
});

^{snippet source | anchor}

The message types are handled like any other message — the cron machinery only decides when an occurrence is published, never how it is processed:

// Recurring messages are handled like any other message — the cron machinery
// only decides WHEN they are published, never how they are processed
public record RunNightlyRollup;

public record BuildDailyReport(DateTimeOffset From, DateTimeOffset To);

public record SendMorningDigest;

public static class RecurringSampleHandler
{
    public static void Handle(RunNightlyRollup message)
        => Console.WriteLine("Rolling up!");

    public static void Handle(BuildDailyReport message)
        => Console.WriteLine($"Reporting on {message.From} to {message.To}");

    public static void Handle(SendMorningDigest message)
        => Console.WriteLine("Good morning!");
}

^{snippet source | anchor}

Registering the first schedule is the feature's opt-in. A host with zero schedules registered behaves — and migrates its message store — exactly as it did before this feature existed. The first registration wires three things:

1. The recurring-message agent, a singular agent that runs on exactly one node in the cluster and moves on failover. Its whole job is a guarantee: the next occurrence of each schedule is published with a ScheduledTime, so the envelope sits in the same place every other scheduled message sits (the durable inbox when a message store is configured) and fires through the same poller.
2. Occurrence deduplication. Every occurrence carries a deterministic logical deduplication id of the form "{scheduleName}:{occurrenceUtc}" — identical on any node and after any restart — and the deduplication middleware is applied to the handler chains of exactly the cron-scheduled message types. An agent failover or restart that re-publishes the same occurrence is collapsed at consumption rather than executed twice. (The requirement is deliberately non-strict: publishing the same message type by hand, without an id, passes through untouched.)
3. Attribution. Each occurrence carries its schedule's name in a recurring-schedule envelope header and the instant it fired for in a recurring-occurrence header (UTC, round-trippable), surfaced on the handler's OpenTelemetry activity as the wolverine.schedule.name and wolverine.schedule.occurrence tags. Trace consumers can therefore attribute work both to the cron job that caused it and to the specific firing it is serving — which matters because ScheduledTime is cleared by the scheduled machinery before the handler runs, so the occurrence instant would otherwise only be recoverable by parsing the deduplication id. The schedule name is additionally a metrics tag (schedule.name) on every message instrument, so success and failure rates can be sliced per schedule; the occurrence instant is deliberately trace-only, since one distinct value per firing would make those series unbounded in cardinality. Both headers round-trip every transport, so the attribution survives to whichever node actually handles the occurrence. The publish itself happens inside a short wolverine.recurring.occurrence span carrying the same two tags, and that span is the parent the occurrence's envelope points at, so each firing is its own trace rather than a child of whatever activity happened to be current when the agent started.

Short cadences: timer agents 6.52

On the path described above, every occurrence waits in the durable inbox until the scheduled-job poller next looks (DurabilitySettings.ScheduledJobPollingTime, 5 seconds by default), and costs an inbox row, a tracking-row write and a deduplication row. That is the right shape for an hourly or nightly job. For a schedule that fires every few seconds or minutes, the poller dominates the latency, and gets proportionally worse the tighter the schedule is.

So a short-cadence schedule runs on a timer agent of its own instead: one agent per schedule, at wolverine-recurring-timer:///{schedule name} (the name URL-encoded), placed by Wolverine's agent assignment like any other agent and spread evenly across the cluster. The agent computes the next occurrence, waits for it on an in-memory timer, and publishes it the moment it comes due. There is no scheduled inbox row, no poller in between, and no tracking-row write per occurrence.

Nothing changes for the handler. A timer occurrence carries the same recurring-schedule and recurring-occurrence headers, the same wolverine.recurring.occurrence span, and -- most importantly -- the same deduplication id "{scheduleName}:{occurrenceUtc}" as the scheduled path. That shared identity is what makes a rolling deploy (old nodes still pre-scheduling, new nodes on timers) or a brief double run while an agent moves between nodes collapse to one handling per occurrence.

Which schedules run on a timer

A schedule runs on a timer agent when the shortest gap between two of its occurrences over one week is under opts.Schedules.TimerThreshold, which defaults to 15 minutes. Sampling a week rather than the first gap means */5 9-17 * * 1-5 counts as a five-minute schedule. A schedule can also force either path:

// Anything under this runs on a timer agent automatically. 15 minutes is the default
opts.Schedules.TimerThreshold = TimeSpan.FromMinutes(15);

// A daily schedule that should still fire the moment it comes due
opts.Schedules.ScheduleRecurring("daily-timer", "0 9 * * *", _ => new TimerTickMessage())
    .UseTimer();

// A fast schedule that should stay on the scheduled-message path
opts.Schedules.ScheduleRecurring("fast-scheduled", "*/30 * * * * *", _ => new TimerTickMessage())
    .UseScheduledMessages();

^{snippet source | anchor}

Setting TimerThreshold to TimeSpan.Zero keeps every schedule on the scheduled-message path unless it opts in with UseTimer(). Schedules at or above the threshold behave exactly as they always have.

A schedule can fire as often as every second 6.61. Anything faster than every 5 seconds always runs on a timer agent, whatever TimerThreshold says, because the scheduled-job poller cannot honour it. UseScheduledMessages() on such a schedule throws.

// Every five seconds -- well under the 15 minute TimerThreshold, so this
// schedule runs on a timer agent of its own and each occurrence is published
// the moment it comes due
opts.Schedules.ScheduleRecurring(ScheduleName, "*/5 * * * * *",
    occurrence => new TimerOccurrence(occurrence));

^{snippet source | anchor}

What changes on the timer path

• Missed occurrences are still skipped, never back-filled. A (re)started timer agent begins with the first occurrence after "now", so the occurrences that fall inside the window while a dead node's agent is being reassigned are lost. There is no pre-scheduled envelope to fire in the meantime, which is the one real difference from the scheduled path.
• Pause is an agent restriction. IRecurringScheduleControl.PauseAsync pauses the schedule's agent through IAgentRuntime.ApplyRestrictionsAsync -- durable, cluster-wide, and confirmed before the call returns -- and mirrors the pause onto the schedule's tracking row so there is one visible state. ResumeAsync lifts both; the next occurrence is again the first one after the resume. Pausing the agent directly (from an agents screen, say) also shows as the schedule being paused, and the agent honours a pause mark it finds on the tracking row, so a schedule paused while it still ran on the scheduled-message path stays paused when a release moves it to a timer.
• A trigger publishes from the calling node. TriggerAsync publishes one manual occurrence immediately, with the same {schedule}:manual:{requested:O} deduplication id as the scheduled path, and is refused while the schedule is paused.
• Moving to the timer path cleans up after itself. The scheduled-path agent no longer schedules a timer schedule, and cancels the leftover pre-scheduled occurrence an earlier release may have left in the inbox.
• Each occurrence comes from the cron, not from "now + interval". A timer that wakes late fires the late occurrence once and still waits for the next one the cron computes, so a one second schedule does not drift.
• Every occurrence still gets a deduplication row. The id is unique per second, so a one second schedule writes about 86,400 rows a day, each kept for DurabilitySettings.DeduplicationWindow (24 hours by default).

Skipping an occurrence while the previous one runs 6.61

By default every occurrence is published as it comes due, whether or not the previous one is done. For a fast schedule calling something slow, like an external API polled every second, that means overlapping runs. SkipWhileRunning() stops that:

// Every second. Anything under 5 seconds always runs on a timer agent.
// SkipWhileRunning() skips any occurrence that comes due while the
// previous one is still being handled
opts.Schedules.ScheduleRecurring("poll-inventory", "* * * * * *",
        occurrence => new PollInventory(occurrence))
    .SkipWhileRunning();

^{snippet source | anchor}

An occurrence that comes due while the previous one is still running is skipped, not queued, and never back-filled. The schedule picks up again with the first occurrence after the running one finishes. Each skip is logged at debug level, the resume is logged at information level with the number skipped, and skips are counted on the wolverine-recurring-occurrences-skipped counter, tagged with schedule.name.

To know when an occurrence is finished, the timer agent invokes it and waits for it, with the same semantics as IMessageBus.InvokeAsync(), instead of publishing it. Know what that means before you turn it on:

• With a local handler, the occurrence runs inline on the node that owns the schedule's timer agent. There is only one of those agents per schedule in the cluster, so the occurrences do not overlap anywhere in the cluster. The exception is the brief window while the agent moves between nodes.
• With no local handler, the occurrence goes to the one endpoint it routes to by request/reply, and the agent waits for the acknowledgement for at most opts.DefaultRemoteInvocationTimeout (5 seconds by default). After that, the next occurrence can start even if the remote handler is still running.
• The occurrence does not go to any other subscribers, and it does not go through a local queue. There is no inbox row, no queue-level error policy and no dead letter queue. A failed occurrence is logged and the next one runs on schedule.
• A handler that never returns stops the schedule until it does.
• The schedule always runs on a timer agent, even with a daily cron. UseScheduledMessages() and SkipWhileRunning() refuse each other at configuration time, because a pre-scheduled occurrence fires from the scheduled-job poller with nothing waiting on the previous one.
• IRecurringScheduleControl.TriggerAsync still publishes its manual occurrence, and the overlap check does not cover it.

Durability modes

Mode	What the timer path does
Balanced	One agent per schedule, distributed evenly by the leader. Only nodes whose release declares a schedule are eligible, and warm-up nodes never run one.
Solo	The node starts every timer agent itself. Pause stops the agent locally and the Solo loop does not restart it.
No message store	The timer agents start directly, like the scheduled-path agent. Pause and resume stop and start the agent in this process, and are lost on restart.
Serverless, ServerlessWithOutbox, MediatorOnly	No agents run, so -- exactly as on the scheduled path -- Wolverine refuses to start with schedules registered.

ServiceCapabilities.RecurringSchedules reports each schedule's ExecutionPath and, for timer schedules, its AgentUri, so a monitoring console can show which node owns the schedule. wolverine describe shows the execution path too.

Cron expressions

CronSchedule accepts the standard 5-field cron grammar, or 6 fields where the leading field is seconds, parsed by Cronos. Occurrences are computed in the schedule's time zone (UTC unless one is supplied), with daylight-saving transitions handled correctly: a schedule inside a spring-forward gap fires at the adjusted instant, and a fall-back repeat fires once.

An invalid expression throws at the registration call site — a bad schedule is a programming error, and it should fail at the line that wrote it. The fastest cadence is every second (* * * * * *), which is as fine as cron goes. A schedule faster than every 5 seconds always runs on a timer agent, because durable scheduled messages replay on DurabilitySettings.ScheduledJobPollingTime (5 seconds by default) and would deliver it late.

Durable tracking: record and verify

When the message store supports it (every relational database provider — PostgreSQL, SQL Server, MySQL, SQLite), the recurring feature keeps a small tracking table (wolverine_recurring_messages) beside the inbox: one row per registered schedule, mapping the schedule's name to the envelope id(s) of its pre-scheduled next occurrence. The scheduled inbox row is still the materialized occurrence — the tracking table is bookkeeping, never a delivery path — but it upgrades the agent's guarantee from fire-and-forget to record and verify:

• Every publish records the pre-scheduled envelope ids on the schedule's row.
• The agent's loop periodically confirms those envelopes still sit Scheduled in the inbox, and re-publishes the occurrence (same deterministic deduplication id) when something cancelled or lost it out from under the schedule.
• After a failover or restart, the successor agent adopts its predecessor's verified pending occurrence off the row instead of blindly re-publishing it.
• Management tooling can see which schedule owns which pending envelope — and cancel or reschedule an occurrence through the ordinary scheduled-message surface — without parsing deduplication-id strings.

The table is provisioned only on the main message store and only when at least one schedule is registered, so the opt-in stays schema-neutral: a host with zero schedules migrates exactly as it did before the feature existed.

Pausing, resuming and triggering a schedule

Pause/resume/trigger is the one piece of runtime-mutable state the feature has — schedule definitions stay code-first. IRecurringScheduleControl is registered in the container alongside the first schedule:

// Registered alongside the first schedule — resolve it straight
// from the container
var control = host.Services.GetRequiredService<IRecurringScheduleControl>();

// Pause: marks the schedule's durable tracking row AND eagerly cancels
// the pre-scheduled next occurrence, so nothing fires in the gap before
// the scheduling agent's next pass — even when this code runs on a
// different node than the agent
await control.PauseAsync("daily-report");

// Resume: the next occurrence is computed strictly after "now".
// The paused window is never back-filled
await control.ResumeAsync("daily-report");

// Trigger: run the schedule once, right now, without disturbing its
// cron cadence or its pending next occurrence. The request rides the
// same durable row, so this works from any node. Throws if the
// schedule is paused
await control.TriggerAsync("daily-report");

// The tracking rows themselves: which schedule owns which pending
// envelope, next fire times, pause state
var schedules = await control.QueryAsync();

^{snippet source | anchor}

The semantics on the scheduled-message path, precisely (a timer schedule pauses through its agent instead -- see what changes on the timer path):

• Pause marks the durable tracking row and eagerly cancels the pending pre-scheduled envelope. The caller may be on a different node than the scheduling agent, so nothing is allowed to fire in the gap before the agent's next pass — the mark and the cancel happen in one database transaction. Occurrences already executing are unaffected.
• Resume schedules strictly after now. The first occurrence after a resume is the next one the cron computes from the resume instant; the paused window is never back-filled.
• Pause survives restarts and failovers — it lives on the row, not in agent memory.
• Pausing or resuming a name that was never registered throws; pausing an already-paused schedule is a no-op that keeps the original pause timestamp.
• A trigger is extra, never a replacement. TriggerAsync runs the schedule once, now, and leaves the cron cadence and the pending pre-scheduled occurrence completely untouched. The request is recorded on the same durable row and the agent acts on it at its next pass, so it works from any node — and it fires even for a fixed-date schedule whose occurrences have run out.
• A manual run carries its own deduplication id ({schedule}:manual:{requested:O}), so it is never collapsed into a scheduled firing it happens to coincide with. The id is still derived from the request instant rather than a random value, so an agent failover that re-publishes the same outstanding request cannot double-handle it.
• Only one trigger can be outstanding at a time. Two calls landing between two agent passes coalesce into a single run — the row has one slot, deliberately.
• Triggering a paused schedule throws RecurringSchedulePausedException. Pausing says the schedule must not fire, so a trigger is refused rather than allowed to override it; resume first if you meant to let it run.

On a message store without the tracking extension (including no store at all), pause and resume degrade to the local agent's memory: they only take effect when the running agent is in the same process, are lost on restart, and cannot cancel an occurrence that is already pre-scheduled — that one will still fire. This is a documented degradation of the same kind as the no-message-store mode below.

Diagnostics

Registered schedules surface in three places, all populated from the registrations themselves — there is no parallel configuration surface to keep in sync:

• The service capabilities snapshot (ServiceCapabilities.RecurringSchedules) carries one typed descriptor per schedule — name, cron expression, time zone, message type, plus the live pause state and pending next occurrence read from the tracking extension — so monitoring consoles (CritterWatch) can render and manage schedules without parsing deduplication-id strings. The read is best-effort: an unreachable database costs the snapshot the runtime fields, never the schedule definitions.
• wolverine describe renders a Recurring Messages table (name, cron, time zone, message type, execution path, computed next occurrence).
• The Event Model: a slice whose command type has a registered schedule is triggered by the job scheduler — the same treatment TimeoutMessage gets — with the cron expression as the trigger origin's label.

Failure semantics

All of these are deliberate, and worth knowing before relying on the feature:

• Missed occurrences are skipped, never back-filled. The one pre-scheduled envelope still fires even with the agent down — it is an ordinary scheduled message — and only occurrences after it in an agent-less window are lost. This is the reason for the agent shape over a self-perpetuating message that schedules its own successor, which dies permanently the first time its envelope is discarded. A missed firing is a missed firing, not a dead schedule.
• Failover and restart are harmless. The agent's working state is in-memory; re-publishing the same occurrence produces the same deduplication id.
• A failed publish logs and retries on the next tick. It never kills the scheduling loop.
• An occurrence with no subscribers or handlers is not a publish. The agent logs a warning (once per schedule) and keeps retrying each pass, so a subscription that appears later — or a routing table that was still warming up — still gets the occurrence on time rather than silently skipping it.

Running without a message store

A host with no message store still runs recurring messages — the agent starts directly (there is no cluster to coordinate) and occurrences ride the in-process scheduling model. Two things degrade, and Wolverine warns about both at startup:

• An occurrence inside a restart window is lost. The schedule itself survives — the agent re-establishes the next occurrence from the registrations at startup — but anything that was pending in memory when the process died is gone.
• There is no store-backed occurrence deduplication, so an agent restart can double-publish an occurrence where a broker's native deduplication does not cover it.

DurabilityMode.Serverless and DurabilityMode.MediatorOnly are different: those modes run no agents at all, so a registered schedule would be accepted and then silently never fire. Wolverine refuses to start in those modes with schedules registered, naming the schedules in the exception — a schedule that quietly never happens is exactly the failure this feature exists to avoid.