Flaky Tests: Clocks, Seeds, Test Order and Threads

Series: Testing & Debugging lecture 5/6 (advanced). For developers whose CI is "usually green" and who have started clicking "re-run" without asking why. After this lecture you can reproduce a flaky test on demand, measure its failure rate, and remove the cause instead of the symptom.

If you have been on call, you know the message: "the build failed, I re-ran it and it passed." That is a flaky test, a test that passes or fails on the same code. It costs more than its own reruns: the team learns to ignore red builds, and the one real failure hides among the noise. This lecture takes the five causes we see most often in junior developers' first months (the clock, randomness, test order, threads and fixed sleeps), makes each one fail on purpose, counts how often it fails, fixes it, and counts again.

Environment: Python 3.9.6 (the system Python on macOS), unittest only, no plugins, Apple Silicon Mac (M1 Pro), 8 cores. Every rate below is a real count with the N stated next to it. Seeds are fixed where the point is not the randomness itself. Paths are shortened to ~/demo/…. This lecture builds on lecture 3, first unit tests, and lecture 4, test doubles and fixtures.

What "flaky" means, and the one rule for fixing it

A flaky test has a hidden input. The code under test is the same each time, but something that the test does not control changes between runs: the wall clock, a random number generator, which test ran before it, how the OS schedules threads, how busy the machine is. The fix is always the same idea: find the hidden input and make it an explicit one.

The rule that keeps you honest: a flake is not fixed until its failure rate is measured, before and after, with the same N. One green rerun means nothing when the test was green most of the time anyway. So every section below has a "before" number and an "after" number.

A rerun loop is the simplest measurement tool, and it works for any test runner:

fails=0
for i in $(seq 1 100); do
  python3 -m unittest -q test_coupon 2>/dev/null || fails=$((fails + 1))
done
echo "$fails/100 failed"

Cause 1: randomness without a seed

Here is a coupon code generator. The spec says a code must contain at least one digit, and the test checks exactly that:

# coupon.py
import random
import string

ALPHABET = string.ascii_letters + string.digits   # 62 characters

def make_code(length=8, rng=random):
    """Return a coupon code. Spec: must contain at least one digit."""
    return "".join(rng.choice(ALPHABET) for _ in range(length))

# test_coupon.py
class CouponTest(unittest.TestCase):
    def test_code_has_a_digit(self):
        code = make_code()
        self.assertTrue(any(c.isdigit() for c in code), code)

Eight runs in a row printed OK, FAILED, OK, FAILED, OK, OK, FAILED, OK. A failing run looks like this:

FAIL: test_code_has_a_digit (test_coupon.CouponTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "~/demo/flaky/test_coupon.py", line 8, in test_code_has_a_digit
    self.assertTrue(any(c.isdigit() for c in code), code)
AssertionError: False is not true : ZYRtMYHA

The test is right; the code is wrong. Nothing makes a digit appear. The chance that 8 characters all come from the 52 letters is (52/62)^8 = 24.48%. We measured it: 2,509 failures in 10,000 calls (25.09%), and 2,487 (24.87%) on a second run.

Common mistake → what actually happens → fix. The first instinct is random.seed(1) at the top of the test. With seed 1 the code is iK2ZWeqh, it has a digit, and the test passes forever. With seed 4 it would be ptgUzEjf and fail forever. A fixed seed does not fix the bug; it freezes the coin toss, and you only learn which side it landed on. The useful version runs many seeds, and every failure says which seed reproduces it:

SEED = int(os.environ.get("TEST_SEED", "20261114"))

class CouponTest(unittest.TestCase):
    def check_many(self, fn):
        master = random.Random(SEED)
        for i in range(200):
            seed = master.randrange(2**32)
            code = fn(rng=random.Random(seed))
            with self.subTest(seed=seed):
                self.assertTrue(any(c.isdigit() for c in code), f"{code} (seed={seed})")

Now the bug fails every run, deterministically: FAILED (failures=46) out of 200 cases, first one njJGnFZW (seed=105476702), and make_code(rng=random.Random(105476702)) gives njJGnFZW again in a REPL. The real fix is in the code: put one digit in on purpose.

def make_code_fixed(length=8, rng=random):
    chars = [rng.choice(ALPHABET) for _ in range(length - 1)]
    chars.insert(rng.randrange(length), rng.choice(string.digits))
    return "".join(chars)

After: 0 failures over 10,000 seeds. Two habits come out of this: pass the RNG in (rng=random) so tests can control it, and always print the seed.

Cause 2: the clock, midnight and daylight saving time

This one hurts on-call engineers most, because it fails at night. A due-date label:

# due.py
from datetime import date, datetime, timedelta, timezone

def due_label(due: date, today: date) -> str:
    if due == today:
        return "today"
    if due == today + timedelta(days=1):
        return "tomorrow"
    return "later" if due > today else "overdue"

def label_for_now_flaky(due: date) -> str:
    return due_label(due, datetime.now().date())          # local wall clock

def label_for(due: date, now_utc: datetime) -> str:       # one clock, passed in
    return due_label(due, now_utc.astimezone(timezone.utc).date())

The flaky test builds "tomorrow" in UTC and compares it with the local date:

def test_tomorrow(self):
    due = (datetime.now(timezone.utc) + timedelta(days=1)).date()   # UTC date
    self.assertEqual(label_for_now_flaky(due), "tomorrow")          # local date

With TZ=UTC it passes. We ran it at about 06:00 UTC with the machine clock set to a zone 8 hours behind UTC (TZ=XST8, a made-up zone name), which is still "yesterday" locally:

AssertionError: 'later' != 'tomorrow'
- later
+ tomorrow

How often does it fail? It depends on the time of day, so instead of waiting we swept a fixed clock over all 1,440 minutes of one UTC day and evaluated the same logic at each minute:

That table explains the classic report "it fails on the CI server only in the evening": the CI machine runs in UTC, the developer's laptop does not, and the failure window is exactly the time-zone offset.

The fixed tests pass the clock in and pin the boundaries, so they mean the same thing at any hour:

class FixedDueTest(unittest.TestCase):
    def check(self, now):
        due = (now + timedelta(days=1)).date()
        self.assertEqual(label_for(due, now), "tomorrow")

    def test_tomorrow_one_second_before_midnight(self):
        self.check(datetime(2026, 11, 14, 23, 59, 59, tzinfo=timezone.utc))

    def test_tomorrow_at_midnight(self):
        self.check(datetime(2026, 11, 15, 0, 0, tzinfo=timezone.utc))

    def test_month_and_year_end(self):
        self.check(datetime(2026, 12, 31, 23, 30, tzinfo=timezone.utc))

Daylight saving time. A day is not always 86,400 seconds. We converted local midnights to timestamps under a made-up DST zone (TZ="XST5XDT,M3.2.0,M11.1.0", switching on the second Sunday of March and the first Sunday of November) and divided by 86,400:

2026-03-08 -> 2026-03-09 23 h int(seconds/86400) = 0 | date subtraction = 1
2026-11-01 -> 2026-11-02 25 h int(seconds/86400) = 1 | date subtraction = 1

int(seconds / 86400) says March 8 and March 9 are 0 days apart. Under TZ=UTC all four pairs gave 24 h. Subtracting date objects is always right. Store and compare instants in UTC; convert to local time only to display.

Common mistake: patching datetime.now. The obvious mock fails on the built-in type:

  File "…/python3.9/unittest/mock.py", line 1502, in __enter__
    setattr(self.target, self.attribute, new_attr)
TypeError: can't set attributes of built-in/extension type 'datetime.datetime'

You can patch the name datetime inside your own module with a subclass whose now() returns a fixed value (we did, and the flaky function returned tomorrow at 23:59:59). But passing the clock in as an argument is simpler and makes the dependency visible in the function signature.

Cause 3: test order and shared state

"It passes alone, it fails in the suite" almost always means shared state. A small pricing module:

# pricing.py
DEFAULTS = {"tax_rate": 0.10, "currency": "USD"}
config = DEFAULTS            # bug 1: an alias, not a copy

def set_option(key, value):
    config[key] = value

def price_with_tax(amount):
    return round(amount * (1 + config["tax_rate"]), 2)

class PricingTest(unittest.TestCase):
    def test_default_tax(self):
        self.assertEqual(pricing.price_with_tax(100), 110.0)

    def test_zero_tax_promo(self):
        pricing.set_option("tax_rate", 0.0)       # bug 2: never undone
        self.assertEqual(pricing.price_with_tax(100), 100.0)

    def test_rounding(self):
        self.assertEqual(pricing.price_with_tax(19.99), 21.99)

unittest runs methods in alphabetical order, so test_zero_tax_promo happens to run last, and the suite is green. Add one test whose name sorts later, or split the suite across parallel workers, and it breaks. python3 -m unittest in Python 3.9 has no shuffle option, so we wrote a 25-line runner that loads the tests, shuffles them with random.Random(seed) for seeds 0 to 199, and re-imports the modules each time so every run starts from fresh state. Result: 139 of 200 orders failed (theory: 2/3, the promo test must be last). The first failing order was seed 0:

first failing order (seed 0): ['test_default_tax', 'test_zero_tax_promo', 'test_rounding']
AssertionError: 19.99 != 21.99

The fix has two parts, one per bug. In the code, config = dict(DEFAULTS) plus a reset() function. In the tests, cleanup that runs even when an assertion fails:

class PricingTest(unittest.TestCase):
    def setUp(self):
        self.addCleanup(pricing.reset)          # runs even if the test fails

    def test_zero_tax_promo_patch_dict(self):
        with mock.patch.dict(pricing.config, {"tax_rate": 0.0}):
            self.assertEqual(pricing.price_with_tax(100), 100.0)

After: 0 of 200 shuffled orders failed. Note that config = DEFAULTS alone would have made even reset() useless, because the "defaults" were being mutated too.

Cause 4: a race between threads

Four threads each count 50,000 hits on a shared counter:

class HitCounter:
    def __init__(self):
        self.value = 0
    def hit(self):
        self.value += 1            # read, add, write: three steps

The test expects 200,000. Six runs printed 194148, 156216, 199794, OK, 104471, OK. The GIL does not make += atomic: Python can switch threads between the read and the write. It switches roughly every sys.getswitchinterval() seconds, 0.005 on this machine, which explains a surprising pattern we measured:

The small test (10,000 hits per thread) never failed, because each thread finished its work within one 5 ms time slice. That is how race conditions survive code review and unit tests, then appear in production under real load. The trick worth remembering is the sixth row: shrinking the switch interval turns a rare race into a 100% reproducible one, in a test that takes 2 ms. Once the test fails reliably, add the lock, and the same harsh setting proves it:

class SafeHitCounter:
    def __init__(self):
        self.value = 0
        self._lock = threading.Lock()
    def hit(self):
        with self._lock:
            self.value += 1

The lock doubled the cost here (59.9 ms vs 30.4 ms for 200,000 hits), which is fine for a counter and worth measuring for anything hot. The same logic applies in other runtimes. For how the GIL schedules threads see threads, processes and asyncio; for a single-threaded event loop with its own ordering surprises, see JavaScript promises and the event loop.

Cause 5: sleep-based waits

A background worker computes a result, and the test "waits long enough":

class ThumbnailWorker:
    def __init__(self):
        self.result = None
        self.done = threading.Event()

    def submit(self, n):
        def run():
            self.result = sum(i * i for i in range(n))   # about 16 ms of CPU
            self.done.set()
        threading.Thread(target=run, daemon=True).start()

w = ThumbnailWorker(); w.submit(400_000)
time.sleep(0.02)                      # "should be enough"
assert w.result is not None

On an idle machine: 0 of 200 runs failed. Then we added what a real suite often has, one CPU-busy thread left running by another test. Four batches of 200 runs failed 17, 9, 19 and 21 times (8.3% on average). The busy thread holds the GIL half the time, so the 16 ms job takes about twice as long. Interestingly, OS-level load (12 CPU-burning processes on 8 cores) produced only 1 failure in 200; the in-process thread was the real trigger.

The longer sleep "fixes" the flake by making every run slow. Waiting on the event is both reliable and the fastest, because it returns the moment the work is done; the timeout is only there so a real bug fails instead of hanging.

That busy leftover thread deserves its own test. This mixin fails any test that leaves a thread running:

class NoLeakedThreads(unittest.TestCase):
    def setUp(self):
        before = set(threading.enumerate())
        def check():
            leaked = [t.name for t in threading.enumerate() if t not in before and t.is_alive()]
            self.assertEqual(leaked, [], "test left threads running")
        self.addCleanup(check)

A test that started a poller thread and forgot to stop it failed with AssertionError: Lists differ: ['poller'] != [], and then the whole test process hung after printing FAILED, because a non-daemon thread keeps Python alive. We had to kill it after 5 seconds (exit 142, from alarm). If your CI job "hangs after the tests finish", look for exactly this.

Before and after, in one table

Quarantine or delete? If you cannot fix a flaky test today, mark it clearly (for example @unittest.skip("flaky: issue-123, 54/100")) with its measured rate, so it stops training the team to ignore red builds. Never let "retry three times" become the permanent fix: in lecture 6 we measure how a 10% flaky test makes git bisect blame the wrong commit.

Honest limits / version notes

• Everything ran on Python 3.9.6 only. The switch interval (sys.setswitchinterval) exists since Python 3.2. Free-threaded Python builds (3.13+) and other versions may race at different rates; we did not run them.
• python3 -m unittest in 3.9 has no built-in random order. Plugins such as pytest-randomly exist but were not installed or run here.
• The DST demo uses a made-up POSIX zone string, not a real region; real zone rules come from the tz database and change by law.
• Rates depend on the machine. Treat our numbers as "this is how big it can be", and measure your own with the loop above.
• When debugging a flaky failure, logs that include the seed, the clock value and the test order make the next occurrence reproducible. An example where logs, not a hunch, found the real cause: an AI agent's wrong diagnosis caught by logs.

Exercises

1. A test checks that shuffle_deck() never puts the same card twice. It failed once in CI last week. Which two things do you add before anything else?
2. A report job uses date.today() to choose "yesterday's" file. CI runs in UTC; the server in the office does not. Which test values do you pin?

Answers

1. Pass the RNG into shuffle_deck(rng=...) and run many seeds with subTest, printing the seed on failure; then measure the rate in a loop before you touch the code. A failure that was seen once is real until a measurement says otherwise.
2. Pass now (UTC) into the function and test at least 23:59:59 and 00:00:00 UTC, the first day of a month and January 1. If the function must use local time, also test a DST switch date with a fixed zone.

---

Previous: Testing & Debugging 4: test doubles, mocks and fixtures · Next: Testing & Debugging 6: git bisect regression hunting · Back to the programming learning roadmap.