Scheduling Across Time Zones: Cron, DST and Clock Skew

Series: Dates & Time Zones lecture 6/6 (advanced). For developers who run jobs, send reminders or check tokens, and whose users or servers are not all in one zone. After this lecture you can say exactly what a scheduler does on a DST night, store recurring events so they do not drift, size a clock-skew leeway from numbers, and test all of it with an injected clock.

If a nightly report did not arrive on a Sunday in March, if a customer in Madrid got a "weekly" reminder at the wrong hour for three weeks, or if logins failed only on one server, this lecture is for you. These bugs share one cause: code that mixed up three different clocks, the wall clock of a zone, the global UTC timeline, and the monotonic clock inside one machine. We simulate a naive scheduler over the 2026 DST weekends minute by minute, expand a weekly meeting two ways, measure what clock skew does to token checks, look at how five tools treat a leap second, and finish with a DST-safe next_run() and its tests.

Environment: Python 3.9.6 with zoneinfo on system tzdata 2026d, zdump for transitions, Java 26, PHP 8.5.6, SQLite 3.51.0, BSD date, Apple Silicon Mac (M1 Pro). Every command names its zone (TZ=UTC, ZoneInfo("Europe/Madrid"), ...). Simulations use fixed dates and fixed seeds. Paths are shortened to ~/demo/.

Three clocks: wall, UTC and monotonic

A program can ask three different questions about time:

1. What time is it on the wall in Madrid? That is a wall clock in a zone. It jumps on DST nights.
2. What instant is it? That is UTC or Unix time. It does not jump for DST, but it is still adjustable: NTP or an administrator can step it forward or back.
3. How much time has passed since X, inside this process? That is a monotonic clock. It never goes backwards, but its zero point means nothing and it cannot be compared across machines.

Python tells you which clock is which:

import time
for name in ("time", "monotonic", "perf_counter", "process_time"):
    print(name, time.get_clock_info(name))

time          impl=clock_gettime(CLOCK_REALTIME)          monotonic=False adjustable=True  resolution=1e-06
monotonic     impl=mach_absolute_time()                   monotonic=True  adjustable=False resolution=4.16667e-08
perf_counter  impl=mach_absolute_time()                   monotonic=True  adjustable=False resolution=4.16667e-08
process_time  impl=clock_gettime(CLOCK_PROCESS_CPUTIME_ID) monotonic=True  adjustable=False resolution=1e-06

time.time() is adjustable=True. A timeout written as if time.time() - start > 30 can fire early or never if the clock is stepped while it waits. Is the monotonic clock more expensive? We measured 1,000,000 calls, 5 repetitions, min / median per call:

time              62.8 ns/call (min of 5), median  63.0
time_ns           66.1 ns/call (min of 5), median  66.5
monotonic         56.1 ns/call (min of 5), median  56.2
perf_counter      56.3 ns/call (min of 5), median  56.4
perf_counter_ns   53.8 ns/call (min of 5), median  54.0
empty lambda      64.3 ns/call

All of them cost about the same as calling an empty Python function (64.3 ns): the loop dominates, the clock is nearly free. The smallest step we observed between two consecutive reads was 0.715 µs for time.time() and 0.291 µs for perf_counter(). There is no performance reason to measure durations with the wall clock. Rule: monotonic()/perf_counter() for durations and timeouts, time.time() or datetime.now(timezone.utc) for timestamps you store or send.

Cron on a DST night: simulated minute by minute

The transitions we care about, straight from the installed tz database:

$ zdump -v -c 2025,2028 America/New_York | grep 2026
Sun Mar  8 06:59:59 2026 UT = Sun Mar  8 01:59:59 2026 EST isdst=0 gmtoff=-18000
Sun Mar  8 07:00:00 2026 UT = Sun Mar  8 03:00:00 2026 EDT isdst=1 gmtoff=-14400
Sun Nov  1 05:59:59 2026 UT = Sun Nov  1 01:59:59 2026 EDT isdst=1 gmtoff=-14400
Sun Nov  1 06:00:00 2026 UT = Sun Nov  1 01:00:00 2026 EST isdst=0 gmtoff=-18000
$ zdump -v -c 2025,2028 Europe/Madrid | grep 2026
Sun Mar 29 00:59:59 2026 UT = Sun Mar 29 01:59:59 2026 CET isdst=0 gmtoff=3600
Sun Mar 29 01:00:00 2026 UT = Sun Mar 29 03:00:00 2026 CEST isdst=1 gmtoff=7200
Sun Oct 25 00:59:59 2026 UT = Sun Oct 25 02:59:59 2026 CEST isdst=1 gmtoff=7200
Sun Oct 25 01:00:00 2026 UT = Sun Oct 25 02:00:00 2026 CET isdst=0 gmtoff=3600

So in New York, 02:00-02:59 does not exist on 8 March and 01:00-01:59 happens twice on 1 November. In Madrid and London the change is at 01:00 UTC on the last Sundays of March and October. Our naive scheduler walks every UTC minute of a three-day window, converts it to local time, and fires when HH:MM matches:

# cronsim.py (core)
def simulate(zone, hhmm, start_utc, days=3):
    tz = ZoneInfo(zone); fires = []
    t = start_utc
    while t < start_utc + timedelta(days=days):
        local = t.astimezone(tz)
        if local.strftime("%H:%M") == hhmm:
            fires.append(local)
        t += timedelta(minutes=1)
    return fires

America/New_York  02:30  runs per local day: 03-08=0, 03-09=1
America/New_York  01:30  runs per local day: 11-01=2, 11-02=1
       2026-11-01T01:30:00-04:00 fold= 0 EDT
       2026-11-01T01:30:00-05:00 fold= 1 EST
Europe/Madrid     02:30  runs per local day: 03-29=0, 03-30=1
Europe/Madrid     02:30  runs per local day: 10-25=2, 10-26=1
       2026-10-25T02:30:00+02:00 fold= 0 CEST
       2026-10-25T02:30:00+01:00 fold= 1 CET
Europe/London     01:30  runs per local day: 03-29=0, 03-30=1
Europe/London     01:30  runs per local day: 10-25=2, 10-26=1
America/Santiago  00:30  runs per local day: 09-06=0, 09-07=1
America/Santiago  23:30  runs per local day: 04-04=2, 04-05=1
Asia/Tokyo        02:30  runs per local day: 03-08=1, 03-09=1

Madrid's 02:30 is the nastiest: it is skipped in March and doubled in October. Santiago, Chile changes at midnight, so even "00:30 daily" is not safe there. Tokyo has no DST and ran once every day, which is exactly why teams in Japan and Korea discover these bugs only when they get their first overseas users or servers.

What real cron implementations do (not run here, from their documentation): the cron in macOS's base system has an -s flag documented in cron(8): "Enable special handling of situations when the GMT offset of the local timezone changes". With it, jobs that run less often than hourly "are executed exactly once, they are not skipped nor executed twice"; the default (-o) is "the old behavior". Vixie-derived Linux crons have similar special-casing for small clock changes; systemd timers and Kubernetes CronJobs let you name a time zone. The details differ, so do not rely on them: run servers in UTC, schedule business-critical jobs in UTC, and if a job must follow a local wall time, compute its next run yourself (last section). For the logging side of cron jobs, see robust shell scripts for cron; for a real cron-driven push sender, scheduled FCM pushes from PHP cron.

Common mistake: subtracting times across a fold

While computing the gaps between runs, our first script printed 0.0 hours between the two 01:30 runs in New York. That was not a bug in the script's arithmetic. It is how Python defines it:

ny = ZoneInfo("America/New_York")
a = datetime(2026, 11, 1, 1, 30, tzinfo=ny)            # first 01:30 (EDT)
b = datetime(2026, 11, 1, 1, 30, fold=1, tzinfo=ny)    # second 01:30 (EST)

2026-11-01T01:30:00-04:00 2026-11-01T01:30:00-05:00
b - a (same tzinfo): 0:00:00
in UTC:             1:00:00
a == b: True   a < b: False
Sat 02:30 -> Mon 02:30 wall: 2 days, 0:00:00  real: 1 day, 23:00:00

When two aware datetimes share the same tzinfo object, Python subtracts and compares their wall times and ignores the offset. Two instants an hour apart compare as equal. Across the March weekend, "Saturday 02:30 to Monday 02:30" is 2 days of wall time but only 47 real hours.

• What happens: duplicate detection (if run == last_run) misses the second run, and "elapsed hours" for billing or rate limits is off by one hour.
• Fix: convert both sides to UTC (.astimezone(timezone.utc)) before subtracting or comparing. Do all arithmetic on instants in UTC, and use local zones only for input and display.

Recurring events: store the rule, not the first UTC instant

A team meets "Mondays at 09:00 New York time". Model A saves the first occurrence as UTC (14:00, because February is EST) and repeats it weekly. Model B saves the rule, 09:00 + America/New_York, and expands each date in that zone:

date        A: UTC-fixed -> NY   | B: NY 09:00 -> UTC   Madrid  CDMX   Tokyo
2026-03-02  09:00                 | 14:00            15:00   08:00  23:00 (Mon)
2026-03-09  10:00                 | 13:00            14:00   07:00  22:00 (Mon)
2026-03-16  10:00                 | 13:00            14:00   07:00  22:00 (Mon)
2026-03-23  10:00                 | 13:00            14:00   07:00  22:00 (Mon)
2026-03-30  10:00                 | 13:00            15:00   07:00  22:00 (Mon)
2026-04-06  10:00                 | 13:00            15:00   07:00  22:00 (Mon)

Model A moved the meeting to 10:00 for the New York team from 9 March. Model B kept 09:00 in New York, which is what the organiser meant, and the meeting moved for everyone else instead. Madrid changed twice in five weeks: 15:00, then 14:00 for three weeks (the US had switched, Europe had not), then 15:00 again from 30 March. Mexico City stayed on 08:00 → 07:00 because Mexico has had no DST since its last transition in October 2022 (zdump shows none after Sun Oct 30 07:00:00 2022 UT). Tokyo and Seoul (both UTC+9, no DST) moved from 23:00 to 22:00.

Model B also survives tzdata updates: if a country changes its DST rule before the event, expanding the rule with the updated tzdata gives the new correct instant. A precomputed UTC list does not. Calendar standards (iCalendar's RRULE with TZID) use model B for this reason.

Clock skew between machines and token leeway

Each server has its own UTC clock, kept in sync by NTP, and none of them is exactly right. When machine A issues a token with iat/nbf = "now" and machine B checks it 100 ms later, B's "now" may be earlier than A's, and the token is "not yet valid". The check, with a leeway:

def check(now, claims, leeway):
    if now + leeway < claims["nbf"]:
        return "not yet valid"
    if now - leeway >= claims["exp"]:
        return "expired"
    return "ok"

A single token checked 0.1 s after issue (T0 = 2026-03-10 00:00:00 UTC, TTL 300 s):

  verifier clock   +0.0 s: leeway 0 -> ok             30 -> ok             60 -> ok
  verifier clock   -0.5 s: leeway 0 -> not yet valid  30 -> ok             60 -> ok
  verifier clock   -2.0 s: leeway 0 -> not yet valid  30 -> ok             60 -> ok
  verifier clock  -30.0 s: leeway 0 -> not yet valid  30 -> ok             60 -> ok
  verifier clock  -90.0 s: leeway 0 -> not yet valid  30 -> not yet valid  60 -> not yet valid

Then 10,000 requests, verifier skew drawn from Normal(0, σ), network delay 0-200 ms, random.Random(7):

  sigma  0.05 s  leeway  0s: 10.58%  leeway  5s:  0.00%  leeway 30s:  0.00%  leeway 60s:  0.00%
  sigma   1.0 s  leeway  0s: 46.62%  leeway  5s:  0.00%  leeway 30s:  0.00%  leeway 60s:  0.00%
  sigma  30.0 s  leeway  0s: 50.46%  leeway  5s: 43.69%  leeway 30s: 15.79%  leeway 60s:  2.35%

Even 50 ms of skew spread rejected 10.58% of fresh tokens with zero leeway. A few seconds of leeway removes the problem when clocks are synced; no leeway saves you when they are not (30 s spread: still 2.35% failures at 60 s). The order of fixes is: sync clocks with NTP, then add a small leeway (seconds, not minutes), then log the skew (now - iat on every rejection) so the next incident has a number. Leeway also extends the life of a stolen token, which is why it should stay small. The signing side of tokens is covered in JWT HS256 signing pitfalls.

Leap seconds: what five tools do with 23:59:60

UTC has had 27 leap seconds since 1972, the last one at the end of 2016. Unix time pretends they do not exist, so one Unix second number covers two real seconds. We asked five tools about 2016-12-31 23:59:60 (UTC):

Four tools, four behaviours, and none of them errors in the same way. Large operators avoid the step entirely with a leap smear: their NTP servers stretch the seconds around the leap second over many hours, so clocks never show 23:59:60. That is fine inside one fleet and confusing when you mix smeared and non-smeared time sources, because they disagree by up to half a second during the smear. Leap seconds are planned to be discontinued by 2035 under a 2022 international decision; until then, treat them like DST: never store :60, keep durations on the monotonic clock, and accept that wall-clock UTC can repeat or stretch a second. This Mac has no right/ zones, so we could not show a leap-aware zdump.

Testing time: an injected clock and a DST-safe next_run()

Every result above was easy to reproduce because no function asked the system for the time; each took now as a parameter. That is the main idea of testing with time without flakiness. The scheduler, condensed:

# nextrun.py
from datetime import datetime, time, timedelta, timezone
from zoneinfo import ZoneInfo

def _exists(local):
    """True if this wall time really happens (round-trips through UTC)."""
    back = local.astimezone(timezone.utc).astimezone(local.tzinfo)
    return back.replace(tzinfo=None) == local.replace(tzinfo=None)

def next_run(now_utc, at, zone):
    tz = ZoneInfo(zone)
    day = now_utc.astimezone(tz).date()
    for _ in range(3):
        local = datetime.combine(day, at, tz)          # fold=0: first of two 01:30s
        if not _exists(local):                          # in the gap: run at the
            local = local.astimezone(timezone.utc).astimezone(tz)  # shifted time
        run = local.astimezone(timezone.utc)
        if run > now_utc:
            return run
        day += timedelta(days=1)

class FixedClock:
    def __init__(self, t): self.t = t
    def now(self): return self.t
    def advance(self, **kw): self.t += timedelta(**kw)

Policy, written down: a time in the gap runs once at the shifted time (02:30 → 03:30 EDT); a time in the overlap runs once, at its first occurrence. The tests pin the edges, plus one test that compares against a version which captured a fixed offset of -05:00 in January:

def test_spring_gap_runs_once_shifted(self):
    c = FixedClock(datetime(2026, 3, 8, 5, 0, tzinfo=UTC))      # 00:00 EST
    r = next_run(c.now(), time(2, 30), "America/New_York")
    self.assertEqual(r.astimezone(ZoneInfo("America/New_York")).isoformat(),
                     "2026-03-08T03:30:00-04:00")
def test_fall_overlap_runs_once(self):
    c = FixedClock(datetime(2026, 11, 1, 4, 0, tzinfo=UTC))     # 00:00 EDT
    first = next_run(c.now(), time(1, 30), "America/New_York")
    second = next_run(first, time(1, 30), "America/New_York")
    self.assertEqual(second - first, timedelta(hours=25))
def test_fixed_offset_version(self):          # offset captured in January: fails in March
    est = timezone(timedelta(hours=-5))
    naive = lambda now, at: datetime.combine(now.astimezone(est).date() + timedelta(days=1), at, est)
    now = datetime(2026, 3, 9, 0, 0, tzinfo=UTC)
    self.assertEqual(naive(now, time(2, 30)).astimezone(UTC),
                     next_run(now, time(2, 30), "America/New_York"))

test_fall_overlap_runs_once (__main__.T) ... ok
test_fixed_offset_version (__main__.T) ... FAIL
test_madrid_october (__main__.T) ... ok
test_spring_gap_runs_once_shifted (__main__.T) ... ok
...  (traceback trimmed)
AssertionError: datetime.datetime(2026, 3, 9, 7, 30, tzinfo=datetime.timezone.utc) != datetime.datetime(2026, 3, 9, 6, 30, tzinfo=datetime.timezone.utc)
FAILED (failures=1)

The fixed-offset version runs at 07:30 UTC, which is 03:30 in New York after the switch: the "fixed offsets are not zones" lesson from lecture 2, caught by a test in milliseconds instead of by a customer in March. The overlap test passes with exactly 25 hours between the two runs, because next_run returns UTC instants and the subtraction happens in UTC.

Series wrap-up checklist

Six lectures, one page:

• Instants in UTC, always. Store Unix seconds or ISO 8601 with Z; convert to a zone only to show it (lectures 1, 5).
• Zones are IANA names, not abbreviations or fixed offsets. America/New_York, not EST or -05:00 (lecture 2).
• Every string you accept has an explicit format and offset; reject or flag strings without one (lectures 3, 5).
• Calendar arithmetic is a policy: write down what "+1 month" from 31 January means and test it (lecture 4).
• Durations come from a monotonic clock; timestamps come from the UTC clock (lecture 6).
• Future local events store the rule (local time + zone), not a precomputed UTC instant (lecture 6).
• Jobs and servers run in UTC; local-time jobs use a tested, DST-aware next_run (lecture 6).
• Clocks drift: NTP first, small leeway second, log the skew third (lecture 6).
• Tests inject the clock and include a DST gap, a DST overlap, 23:59:59, 1 January and 29 February.
• Update tzdata with your OS and runtimes; zone rules change by law.

Honest limits and version notes

• The scheduler, meeting and skew results are simulations in Python 3.9.6 with tzdata 2026d. We did not install or edit a crontab, change the system clock, or contact an NTP server. Cron behaviour is quoted from the macOS cron(8) manual page and described for others.
• zoneinfo is in the standard library since Python 3.9; fold since Python 3.6. The "same tzinfo compares wall times" rule is documented Python behaviour, not a zoneinfo bug.
• The skew distribution (normal, fixed σ) is a model. Real skew is often one badly configured host far off and the rest close; measure yours by logging now - iat on rejection.
• Java results are from OpenJDK 26, PHP from 8.5.6, SQLite from 3.51.0. Leap-second handling is not standardized across libraries and may change between versions.

Exercise

1. A reminder service stores remind_at as UTC, computed once when the user picks "every day at 08:00" in Europe/Madrid. Users complain twice a year. Describe the change to the schema and to the job that sends reminders.

Answers

1. Store the rule instead: local_time = '08:00', zone = 'Europe/Madrid', plus next_run_utc as an indexed INTEGER. The job (running in UTC every minute) selects rows with next_run_utc <= now, sends them, and recomputes next_run_utc with a DST-aware next_run(now, 08:00, zone). Add tests with a fixed clock on 28/29 March and 24/25 October 2026, and on a day without a transition.

Previous: Dates & Time Zones 5: dates in databases, SQLite and timestamps · Start of the series: Dates & Time Zones 1: Unix timestamps and UTC · Back to the programming learning roadmap.