Test Doubles in Python: Mocks, Fakes, Fixtures

Series: Testing & Debugging lecture 4/6 (intermediate). For developers who can write a plain unit test (lecture 3) and now need to test code that talks to a network service, a database or the file system. After this lecture you can tell a stub from a fake, a mock and a spy, patch the right name, let autospec catch signature drift, write fixtures that clean up even when setup fails, and recognise a test that mocks so much that it can no longer fail.

If you have ever been on call and heard "but all the tests were green" at 2 a.m., there is a good chance a mock was involved. This lecture reproduces the three ways that happens, with real output: a patch that silently hit the real network, a mock that accepted a call with the wrong arguments, and a mocked repository that passed two refactors which broke production. Everything was run on Python 3.9.6 (with messages compared on 3.12.14) and Dart 3.11 on an Apple Silicon Mac (M1 Pro). Paths are shortened to ~/demo/.

The numbers you will see: the plain Mock test passed both broken refactors; the in-memory SQLite fake caught both, and was also the fastest double we measured (47 µs per test, against 141 µs for Mock and 786 µs for create_autospec).

Stub, fake, mock, spy: four jobs, one word

"Mock" is used for everything, but the classic vocabulary (from Gerard Meszaros's xUnit Test Patterns, popularised by Martin Fowler's "Mocks Aren't Stubs") separates four jobs. A test double is any object that stands in for a real dependency during a test.

In Python, one class, unittest.mock.Mock, can play the stub, mock and spy roles. The names describe what you do with it. That matters, because the most common mistake is using a mock (checking calls) where a stub or fake (checking results) would do.

The code under test

A freelancer-style invoice in another currency. rates.py talks to a rates service; invoice.py uses it. Nothing listens on port 9 on this machine, so a real call fails fast instead of reaching the internet.

# rates.py
import json
import urllib.request


def fetch_rate(currency):
    """How many units of `currency` one USD buys (rates service)."""
    url = f"http://127.0.0.1:9/rate?c={currency}"    # nothing listens on port 9
    with urllib.request.urlopen(url, timeout=2) as resp:
        return json.load(resp)["rate"]

# invoice.py
from rates import fetch_rate


def invoice_total(amount_usd, currency):
    return round(amount_usd * fetch_rate(currency))

The rate 0.92 below is a made-up test value, not a real exchange rate.

Patch trap 1: patch where the name is looked up

# test_invoice.py
import unittest
from unittest.mock import patch

from invoice import invoice_total


class InvoiceTest(unittest.TestCase):
    @patch("rates.fetch_rate", return_value=0.92)        # wrong target
    def test_patched_at_definition(self, fake_rate):
        self.assertEqual(invoice_total(1000, "EUR"), 920)

    @patch("invoice.fetch_rate", return_value=0.92)      # where it is looked up
    def test_patched_where_used(self, fake_rate):
        self.assertEqual(invoice_total(1000, "EUR"), 920)
        fake_rate.assert_called_once_with("EUR")

$ python3 -m unittest
E.
======================================================================
ERROR: test_patched_at_definition (test_invoice.InvoiceTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../python3.9/urllib/request.py", line 1346, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  ... 8 more frames inside http.client and socket ...
ConnectionRefusedError: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File ".../python3.9/unittest/mock.py", line 1337, in patched
    return func(*newargs, **newkeywargs)
  File "~/demo/invoice/test_invoice.py", line 10, in test_patched_at_definition
    self.assertEqual(invoice_total(1000, "EUR"), 920)
  File "~/demo/invoice/invoice.py", line 5, in invoice_total
    return round(amount_usd * fetch_rate(currency))
  File "~/demo/invoice/rates.py", line 8, in fetch_rate
    with urllib.request.urlopen(url, timeout=2) as resp:
  ... 6 more frames inside urllib ...
urllib.error.URLError: <urlopen error [Errno 61] Connection refused>

----------------------------------------------------------------------
Ran 2 tests in 0.010s

FAILED (errors=1)

Read the frames from your code down (lecture 1): invoice.py line 5 called rates.py line 8, the real function. The patch worked, but on the wrong name. from rates import fetch_rate copies a reference into the invoice module at import time. patch("rates.fetch_rate") replaces the attribute in rates, and invoice still holds the original. In real life, this "works" until the day the network is up and the test quietly hits a real service, or it hangs for the full timeout in CI.

The rule: patch the name in the module that uses it. And the target follows the import style. When I changed invoice.py to import rates and rates.fetch_rate(currency), the results flipped: patch("rates.fetch_rate") passed, and patch("invoice.fetch_rate") failed with:

AttributeError: <module 'invoice' from '~/demo/alt/invoice.py'> does not have the attribute 'fetch_rate'

Patch trap 2: a mock accepts any call; autospec does not

Next, a realistic refactor. The rates service now needs a date, so rates.py becomes def fetch_rate(currency, on_date):. Nobody updates invoice.py, which still calls fetch_rate(currency). In production:

TypeError: fetch_rate() missing 1 required positional argument: 'on_date'

Three tests against that broken code:

class InvoiceTest(unittest.TestCase):
    @patch("invoice.fetch_rate", return_value=0.92)
    def test_plain_mock(self, fake_rate):
        self.assertEqual(invoice_total(1000, "EUR"), 920)

    @patch("invoice.fetch_rate", return_value=0.92, autospec=True)
    def test_autospec(self, fake_rate):
        self.assertEqual(invoice_total(1000, "EUR"), 920)

    @patch("invoice.fetch_rate", return_value=0.92)
    def test_typo_in_assert(self, fake_rate):
        invoice_total(1000, "EUR")
        fake_rate.called_once_with("USD")          # missing "assert_": checks nothing

$ python3 -m unittest -v          # Python 3.9.6
test_autospec (test_invoice.InvoiceTest) ... ERROR
test_plain_mock (test_invoice.InvoiceTest) ... ok
test_typo_in_assert (test_invoice.InvoiceTest) ... ok

======================================================================
ERROR: test_autospec (test_invoice.InvoiceTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  ...
  File "~/demo/invoice2/invoice.py", line 5, in invoice_total
    return round(amount_usd * fetch_rate(currency))
  File "<string>", line 2, in fetch_rate
  File ".../python3.9/unittest/mock.py", line 180, in checksig
    sig.bind(*args, **kwargs)
  ...
TypeError: missing a required argument: 'on_date'

• The plain mock passed. A Mock accepts any arguments, so it happily pretended the old signature still existed.
• autospec=True failed with the same TypeError production would raise. It builds the mock from the real function's signature.
• The typo test passed on 3.9. called_once_with (without assert_) is just an auto-created attribute; calling it checks nothing, and it even asserted the wrong currency. Python 3.12 rejects it:

AttributeError: 'called_once_with' is not a valid assertion. Use a spec for the mock if 'called_once_with' is meant to be an attribute.. Did you mean: 'assert_called_once_with'?

(The double period is in Python's own message.) If you are still on 3.9–3.11, autospec gives you the same safety, because a specced mock has no called_once_with attribute.

Fixtures: setUp, tearDown, addCleanup and temporary files

A fixture is the known state a test starts from: a temporary directory, a config file, a database with two users. unittest builds it in setUp and destroys it in tearDown. There is a trap: if setUp raises, tearDown never runs.

class TearDownVersion(unittest.TestCase):
    def setUp(self):
        self.tmp = tempfile.mkdtemp(prefix=PREFIX)
        self.config = os.path.join(self.tmp, "config.json")
        with open(self.config, "w") as f:
            json.dump({"currency": "EUR"}, f)
        self.client = json.load(open(os.path.join(self.tmp, "client.json")))  # forgot to create it

    def tearDown(self):
        shutil.rmtree(self.tmp)


class CleanupVersion(unittest.TestCase):
    def setUp(self):
        tmp = tempfile.TemporaryDirectory(prefix=PREFIX)
        self.addCleanup(tmp.cleanup)          # registered before anything can fail
        self.config = os.path.join(tmp.name, "config.json")
        with open(self.config, "w") as f:
            json.dump({"currency": "EUR"}, f)
        self.client = json.load(open(os.path.join(tmp.name, "client.json")))

Both versions have the same bug in setUp (a missing client.json). I ran each test 5 times and counted the directories left behind in the system temp folder:

TearDownVersion: runs=5 errors=5 leftover dirs=5
CleanupVersion: runs=5 errors=5 leftover dirs=0

The error itself was the same for both:

FileNotFoundError: [Errno 2] No such file or directory: '$TMPDIR/invoice-test-9nvxeqa1/client.json'

addCleanup functions run even when setUp fails, in reverse order of registration. Register the cleanup right after creating the resource, and prefer tempfile.TemporaryDirectory over writing to a fixed path, so two test runs (or two developers on one CI runner) never share files. For expensive fixtures that are safe to share, setUpClass runs once per class, but shared state is exactly what makes tests order-dependent (lecture 5).

A fake beats a mock: SQLite in memory

Now a signup service with a repository. Names are normalised so that "Alex" and "alex" are the same user.

# users.py
import sqlite3


class UserRepo:
    def __init__(self, conn):
        self.conn = conn
        conn.execute("CREATE TABLE IF NOT EXISTS users (name TEXT PRIMARY KEY)")

    def exists(self, name):
        row = self.conn.execute("SELECT 1 FROM users WHERE name = ?", (name,)).fetchone()
        return row is not None

    def add(self, name):
        self.conn.execute("INSERT INTO users (name) VALUES (?)", (name,))


def signup(repo, name):
    key = name.strip().casefold()
    if repo.exists(key):
        raise ValueError(f"name already taken: {key}")
    repo.add(key)
    return key

Three ways to test signup:

class SignupWithMock(unittest.TestCase):
    def test_new_name_is_added(self):
        repo = Mock()
        repo.exists.return_value = False
        signup(repo, " Alex ")
        repo.add.assert_called_once()


class SignupWithAutospec(unittest.TestCase):
    def test_new_name_is_added(self):
        repo = create_autospec(UserRepo, instance=True)
        repo.exists.return_value = False
        self.assertEqual(signup(repo, " Alex "), "alex")
        repo.add.assert_called_once_with("alex")


class SignupWithSqliteFake(unittest.TestCase):
    def setUp(self):
        conn = sqlite3.connect(":memory:")
        self.addCleanup(conn.close)
        self.repo = UserRepo(conn)

    def test_new_name_is_added(self):
        self.assertEqual(signup(self.repo, " Alex "), "alex")
        self.assertTrue(self.repo.exists("alex"))

    def test_same_name_different_case_is_rejected(self):
        signup(self.repo, "Alex")
        with self.assertRaises(ValueError):
            signup(self.repo, "alex")

All four tests pass on the correct code. The difference shows only when someone changes the code.

Over-mocking: two broken refactors, one green test

Refactor 1: a teammate renames UserRepo.add to insert and forgets to update signup. Production now raises AttributeError on every signup.

test_new_name_is_added (test_users.SignupWithAutospec) ... ERROR
test_new_name_is_added (test_users.SignupWithMock) ... ok
test_new_name_is_added (test_users.SignupWithSqliteFake) ... ERROR
test_same_name_different_case_is_rejected (test_users.SignupWithSqliteFake) ... ERROR
...
AttributeError: Mock object has no attribute 'add'
...
AttributeError: 'UserRepo' object has no attribute 'add'

Refactor 2: someone removes .casefold() from signup ("the database will handle case"), but the database does not. Now "Alex" and "alex" become two accounts.

test_new_name_is_added (test_users.SignupWithAutospec) ... FAIL
test_new_name_is_added (test_users.SignupWithMock) ... ok
test_new_name_is_added (test_users.SignupWithSqliteFake) ... FAIL
test_same_name_different_case_is_rejected (test_users.SignupWithSqliteFake) ... FAIL
...
AssertionError: 'Alex' != 'alex'
- Alex
? ^
+ alex
? ^
...
AssertionError: ValueError not raised

The plain mock test cannot fail, because it only asserts that some add was called somehow, on an object that invents any attribute you ask for. That is over-mocking: the test restates the implementation and checks nothing the user cares about. The autospec test caught the second refactor only because it also checks the return value; the fake caught it by behaving like a database. Only the fake expressed the actual rule, "the same name in a different case is rejected".

Use mocks at the edges you cannot run (network, payment provider, clock, e-mail sending). For your own code, prefer the real thing or a fake.

A spy: real behaviour, recorded calls

When you want the real repository but also want to see the calls, wrap it:

repo = UserRepo(sqlite3.connect(":memory:"))
with patch.object(repo, "add", wraps=repo.add) as spy:
    signup(repo, "Alex")
print(spy.call_args_list, repo.exists("alex"))

[call('alex')] True

The row really was inserted, and the call was recorded.

The same idea in Dart: the compiler is your autospec

Dart has no patch; you pass dependencies in. A hand-written fake that implements an interface is checked by the compiler. The mocktail package is in the local pub cache too, but a fake needs no package at all:

// lib/rates.dart
abstract interface class RateSource {
  Future<double> rate(String currency);
}

Future<int> invoiceTotal(RateSource rates, int amountUsd, String currency) async =>
    (amountUsd * await rates.rate(currency)).round();

// test/rates_test.dart
class FakeRates implements RateSource {
  final Map<String, double> table;
  final calls = <String>[];          // a spy: records what was asked
  FakeRates(this.table);

  @override
  Future<double> rate(String currency) async {
    calls.add(currency);
    return table[currency]!;
  }
}

void main() {
  test('converts with the fake rate', () async {
    final rates = FakeRates({'EUR': 0.92});
    expect(await invoiceTotal(rates, 1000, 'EUR'), 920);
    expect(rates.calls, ['EUR']);
  });
}

It passed. Then I added the same on_date parameter to the interface (rate(String currency, DateTime onDate)) and ran dart test again:

  Failed to load "test/rates_test.dart":
  test/rates_test.dart:10:18: Error: The method 'FakeRates.rate' has fewer positional arguments than those of overridden method 'RateSource.rate'.
  ...
  lib/rates.dart:6:34: Error: Too few positional arguments: 2 required, 1 given.

The fake and the caller both broke at compile time, which is what autospec approximates at run time in Python.

What each double costs, measured

Mocks are often chosen "because the database is slow". I measured one signup test written four ways: N = 1,000 runs, 5 repetitions, median per test, after one warm-up call. Python 3.9.6, Apple Silicon Mac (M1 Pro), SQLite 3.51.0, run twice with similar results.

On both versions the in-memory database was the fastest option, and create_autospec was the slowest of the in-memory choices, because it inspects the class to copy every method signature. A thousand autospec tests cost under a second, which is still cheap for the safety they buy; the point is that "mocks are faster" is not automatically true. The real database file in a temporary directory was the slowest, mostly from creating and deleting the directory and file.

Decision table

Honest limits / version notes

• Python 3.9.6 was the main runtime; 3.12.14 was used for the typo check and the second timing column. The rejection of called_once_with and similar misspellings arrived in Python 3.12; 3.9 accepted it silently in our run. Python 3.5 and later already reject attributes starting with assert or assret.
• autospec and create_autospec have existed since unittest.mock entered the standard library in Python 3.3. wraps= likewise.
• The rates service is simulated by a closed local port. Against a real but slow service, the wrong patch would hang for the full timeout instead of failing in 10 ms.
• SQLite :memory: is a good fake for code written against SQLite. If production uses another database, SQL dialect differences (types, ON CONFLICT, case sensitivity of comparisons) can make the fake pass where production fails; keep a few integration tests against the real engine. That was not run here.
• Dart 3.11.5, test 1.30.0 from the local pub cache. mocktail was not used in the run.
• The microsecond timings are from one machine and vary between runs (across the two runs, min and max stayed within 12% of the median).

Exercises

1. report.py contains import datetime and calls datetime.date.today(). Which target would you patch, and why does patch("datetime.date.today") fail? (Try it.)
2. Rewrite SignupWithMock.test_new_name_is_added so that it would have caught refactor 2, without using SQLite.

Answers

1. datetime.date is a built-in type written in C, so you cannot set attributes on it: patch("datetime.date.today") raises TypeError: cannot set 'today' attribute of immutable type 'datetime.date' (wording on 3.9: can't set attributes of built-in/extension type 'datetime.date'). Patch the name your module uses, patch("report.datetime"), and set mock_dt.date.today.return_value. Better still, pass a clock or a date into the function, which lecture 5 does.
2. Check the result and the exact call: self.assertEqual(signup(repo, " Alex "), "alex") and repo.add.assert_called_once_with("alex"). Use create_autospec(UserRepo, instance=True) instead of Mock() so the rename in refactor 1 is caught too. It is still weaker than the fake: it cannot catch the duplicate-account bug, because exists is hard-wired to False.

Where next

• Previous: Lecture 3: unit testing basics
• Next: Lecture 5: flaky tests, time, randomness and concurrency
• Earlier: Lecture 1: reading error messages, Lecture 2: print, logging and the debugger
• Later: Lecture 6: git bisect
• Related: Python decorators and context managers (how @patch and with patch(...) work), an AI agent's wrong diagnosis caught by logs (another case where a green check was not the truth), and the programming learning roadmap.