Unit Testing Basics: First Tests and Failing Diffs

Series: Testing & Debugging lecture 3/6 (intermediate). For developers who can read a stack trace and use a debugger (lectures 1 and 2) but have never written a test file of their own. After this lecture you can write a unittest file from scratch, choose test cases by looking at boundaries, read a failing diff without guessing, and decide what is not worth testing.

If you are in your first month on a team and your pull request came back with one comment, "where are the tests?", this lecture is for you. It is also for the person who gets paged at night for a bug that a three-line test would have caught. Everything below was run on Python 3.9.6 (some behaviour also checked on 3.12.14) and Dart 3.11 on an Apple Silicon Mac (M1 Pro); file paths in outputs are shortened to ~/demo/.

The headline facts, all measured below: a test file with a typo in its name ran zero tests and still printed OK on Python 3.9; thinking about boundaries found four inputs that the happy-path tests never touched; and a suite of 10,000 tiny tests took 286 ms, so speed is never a reason to skip small tests.

Your first test file in five minutes

A few terms first. A unit test calls one small piece of code (usually one function) with known inputs and checks the output. An assertion is the check: "this value must equal that value". A test runner finds your tests, runs them and reports. Python ships a runner in the standard library, unittest, so there is nothing to install.

Here is the code we will test. It splits a team dinner bill between people. Money is kept in integer cents, because floats cannot represent 0.1 exactly.

# bill.py
"""Splitting the team dinner bill. Amounts are integer cents."""

FREE_SHIPPING_FROM = 5000   # $50.00
SHIPPING_FEE = 599   # $5.99


def split_bill(total_cents, people):
    """Split total_cents into integer shares that add up to the total.

    >>> split_bill(1000, 3)
    [334, 333, 333]
    >>> split_bill(900, 3)
    [300, 300, 300]
    >>> split_bill(10.0, 3)
    Traceback (most recent call last):
      ...
    TypeError: total_cents must be an int, got float
    """
    if not isinstance(total_cents, int):
        raise TypeError(f"total_cents must be an int, got {type(total_cents).__name__}")
    if people < 1:
        raise ValueError(f"people must be >= 1, got {people}")
    base, extra = divmod(total_cents, people)
    return [base + 1] * extra + [base] * (people - extra)


def shipping_fee(subtotal_cents):
    return 0 if subtotal_cents >= FREE_SHIPPING_FROM else SHIPPING_FEE


def format_share(name, cents):
    return f"{name} owes ${cents // 100}.{cents % 100:02d}"

And the test file next to it:

# test_bill.py
import doctest
import unittest

import bill
from bill import format_share, shipping_fee, split_bill


class SplitBillTest(unittest.TestCase):
    def test_even_total_gives_equal_shares(self):
        # Arrange
        total, people = 9000, 3
        # Act
        shares = split_bill(total, people)
        # Assert
        self.assertEqual(shares, [3000, 3000, 3000])

    def test_remainder_goes_to_first_people(self):
        self.assertEqual(split_bill(10000, 3), [3334, 3333, 3333])

    def test_shares_always_add_up_to_total(self):
        for total in (0, 1, 2, 99, 10000, 10001):
            with self.subTest(total=total):
                self.assertEqual(sum(split_bill(total, 3)), total)

    def test_zero_people_is_rejected(self):
        with self.assertRaises(ValueError):
            split_bill(1000, 0)

    def test_float_total_is_rejected_clearly(self):
        with self.assertRaisesRegex(TypeError, "total_cents must be an int"):
            split_bill(10.0, 3)

    def test_negative_total_still_adds_up(self):
        self.assertEqual(split_bill(-100, 3), [-33, -33, -34])


class ShippingFeeTest(unittest.TestCase):
    def test_one_cent_below_threshold_pays_fee(self):
        self.assertEqual(shipping_fee(4999), 599)

    def test_exactly_threshold_is_free(self):
        self.assertEqual(shipping_fee(5000), 0)


class FormatShareTest(unittest.TestCase):
    def test_pads_cents(self):
        self.assertEqual(format_share("Alex", 3305), "Alex owes $33.05")


def load_tests(loader, tests, ignore):
    tests.addTests(doctest.DocTestSuite(bill))
    return tests


if __name__ == "__main__":
    unittest.main()

Run it from the folder that holds both files:

$ python3 -m unittest -v
test_pads_cents (test_bill.FormatShareTest) ... ok
test_exactly_threshold_is_free (test_bill.ShippingFeeTest) ... ok
test_one_cent_below_threshold_pays_fee (test_bill.ShippingFeeTest) ... ok
test_even_total_gives_equal_shares (test_bill.SplitBillTest) ... ok
test_float_total_is_rejected_clearly (test_bill.SplitBillTest) ... ok
test_negative_total_still_adds_up (test_bill.SplitBillTest) ... ok
test_remainder_goes_to_first_people (test_bill.SplitBillTest) ... ok
test_shares_always_add_up_to_total (test_bill.SplitBillTest) ... ok
test_zero_people_is_rejected (test_bill.SplitBillTest) ... ok
split_bill (bill)
Doctest: bill.split_bill ... ok

----------------------------------------------------------------------
Ran 10 tests in 0.002s

OK

Ten tests: nine methods plus the doctest that load_tests adds (more on doctest below). Note the order: classes and methods run in alphabetical order, not in file order. Never write a test that depends on another test having run first.

The naming trap: "Ran 0 tests" and OK

python3 -m unittest with no arguments uses discovery: it imports files matching test*.py and runs methods whose names start with test. Name the file bill_tests.py and the method check_remainder, and this is what you get, even though the assertion inside is deliberately wrong:

$ python3 -m unittest          # Python 3.9.6
----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK
$ echo $?
0
$ python3.12 -m unittest       # Python 3.12.14, same files
----------------------------------------------------------------------
Ran 0 tests in 0.000s

NO TESTS RAN
$ echo $?
5

Renaming only the file to test_bill.py still ran 0 tests, because the method name was wrong too. On 3.9 a CI job would stay green forever. Habit: after adding a test, make it fail once on purpose and see it fail.

Arrange, act, assert, and names that read as sentences

test_even_total_gives_equal_shares shows the three-part shape most good tests have:

• Arrange: prepare inputs (and later, fake collaborators).
• Act: call the one thing under test, once.
• Assert: check the result.

You will not write the comments every time, but if a test has two "act" steps, it is usually two tests. The name should say what, under which condition and what is expected: test_one_cent_below_threshold_pays_fee tells you the bug before you open the file. test_shipping_2 tells you nothing at 3 a.m.

subTest is the tool for "the same check over many inputs". Without it, the loop stops at the first failing value; with it, each value is reported separately, as you will see in the failure output.

Finding edge cases by thinking about boundaries

The first tests people write follow the happy path: 9,000 cents for three people. Bugs live at the edges. Before writing more tests, I probed the first version of split_bill (the one without the isinstance check) with inputs chosen from boundaries: one person, fewer cents than people, zero, negative, and wrong types.

split_bill(1000, 1)        -> [1000]
split_bill(2, 3)           -> [1, 1, 0]
split_bill(0, 3)           -> [0, 0, 0]
split_bill(-100, 3)        -> [-33, -33, -34]
split_bill(1000, True)     -> [1000]
split_bill(1000, 2.5)      -> TypeError: can't multiply sequence by non-int of type 'float'
split_bill(1000.0, 3)      -> TypeError: can't multiply sequence by non-int of type 'float'
split_bill(1000, '3')      -> TypeError: '<' not supported between instances of 'str' and 'int'
split_bill(1000, 0)        -> ValueError: people must be >= 1, got 0

Four findings the happy-path tests would never show:

The float case is the realistic one: someone passes dollars from a form (10.0) instead of cents. The error message mentions sequences, so the person debugging it starts in the wrong place. That is why split_bill now checks the type, and why test_float_total_is_rejected_clearly checks the message, not just the exception class.

A quick checklist for boundaries: zero, one, many; just below, exactly at, just above a threshold (4999, 5000); empty input; negative; the wrong type; very large values. The shipping tests use exactly the 4999/5000 pair, because >= versus > is the most common one-character bug in pricing code.

Reading a failing test

Now break the code in three small ways, the kind of change a tired reviewer misses: replace the divmod logic with [total_cents // people] * people, change >= to > in shipping_fee, and drop the :02d padding. The real output, trimmed:

$ python3 -m unittest
FF...FF.F
======================================================================
FAIL: test_pads_cents (test_bill.FormatShareTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "~/demo/bill/test_bill.py", line 47, in test_pads_cents
    self.assertEqual(format_share("Alex", 3305), "Alex owes $33.05")
AssertionError: 'Alex owes $33.5' != 'Alex owes $33.05'
- Alex owes $33.5
+ Alex owes $33.05
?               +

======================================================================
FAIL: test_exactly_threshold_is_free (test_bill.ShippingFeeTest)
...
AssertionError: 599 != 0
======================================================================
FAIL: test_negative_total_still_adds_up (test_bill.SplitBillTest)
...
AssertionError: Lists differ: [-34, -34, -34] != [-33, -33, -34]

First differing element 0:
-34
-33

- [-34, -34, -34]
?    ^    ^

+ [-33, -33, -34]
?    ^    ^

======================================================================
FAIL: test_shares_always_add_up_to_total (test_bill.SplitBillTest) (total=1)
...
AssertionError: 0 != 1
...
----------------------------------------------------------------------
Ran 10 tests in 0.003s

FAILED (failures=9)

How to read it, top to bottom:

1. The progress line FF...FF.F has one character per test: . pass, F failure, E error. It shows 9 characters for 10 tests, yet the summary says 9 failures: the subTest failures are counted one per failing value (total=1, 2, 10000 and 10001) but share one character.
2. The traceback ends at your test line, not inside unittest. Line 47 of test_bill.py is the place to look.
3. first != second: unittest calls the two arguments first and second, not "actual" and "expected". Because we always write assertEqual(actual, expected), the - line is what the code returned and the + line is what we wanted. Keep that order in every test, or the diff will lie to you.
4. The ? line points at the exact characters that differ: one + under the missing 0, two ^ under the changed digits.
5. Lists get "First differing element 0" before the diff. Here the negative case shows a second bug the boss would care about: floor division turns a 100-cent refund into 102 cents.
6. (total=1) after the test name is the subTest label. Without subTest you would see only the first failing total.

The exit code was 1, which is what CI looks at.

FAIL versus ERROR

Remove the people < 1 guard and the type check, and two different letters appear:

....F...EF
======================================================================
ERROR: test_zero_people_is_rejected (test_bill.SplitBillTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "~/demo/bill/test_bill.py", line 27, in test_zero_people_is_rejected
    split_bill(1000, 0)
  File "~/demo/bill/bill.py", line 19, in split_bill
    base, extra = divmod(total_cents, people)
ZeroDivisionError: integer division or modulo by zero

======================================================================
FAIL: test_float_total_is_rejected_clearly (test_bill.SplitBillTest)
----------------------------------------------------------------------
TypeError: can't multiply sequence by non-int of type 'float'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "~/demo/bill/test_bill.py", line 31, in test_float_total_is_rejected_clearly
    split_bill(10.0, 3)
AssertionError: "total_cents must be an int" does not match "can't multiply sequence by non-int of type 'float'"

FAIL means an assertion was checked and was false. ERROR means the test crashed before it could decide: assertRaises(ValueError) only catches ValueError, so the ZeroDivisionError escaped. The second block is a chained exception (lecture 1): the original TypeError comes first, then the assertion that rejected its message.

doctest: examples that cannot go stale

The >>> lines in the split_bill docstring are a doctest: Python runs them and compares the printed result to the line below. Run them alone with python3 -m doctest bill.py (silent on success, exit code 0) or add -v. The load_tests function in the test file adds them to the normal unittest run.

When I changed the expected list in the docstring to [333, 333, 334]:

**********************************************************************
File "~/demo/bill/bill.py", line 10, in bill.split_bill
Failed example:
    split_bill(1000, 3)
Expected:
    [333, 333, 334]
Got:
    [334, 333, 333]
**********************************************************************
1 items had failures:
   1 of   3 in bill.split_bill
***Test Failed*** 1 failures.

Use doctest for short examples that also serve as documentation. Avoid it for anything whose printed form is unstable: dict ordering from external data, floats with long tails, object addresses like <Foo at 0x10...>. Those belong in unittest.

A trap I hit while preparing this: I edited the docstring twice within one second, and the second edit kept the file size identical. Python reused the cached bytecode (.pyc), whose recorded modification time and size matched, so the old docstring kept failing even after the fix. Deleting the cache did not help, because this Python stores .pyc files under a separate cache prefix. Touching the file a second later fixed it. You will rarely hit this by hand, but scripts that rewrite source files can.

The same test in Dart, offline

Can you run Dart's package:test without network access? The brief for this machine said: only if dart pub get --offline resolves from the existing package cache. It did, in 1.0 s, resolving test 1.30.0:

// lib/pricing.dart
int shippingFee(int subtotal) {
  if (subtotal < 0) throw ArgumentError.value(subtotal, 'subtotal');
  return subtotal >= 5000 ? 0 : 500;
}

// test/pricing_test.dart
import 'package:pricing/pricing.dart';
import 'package:test/test.dart';

void main() {
  group('shippingFee', () {
    test('charges 500 just below the threshold', () {
      expect(shippingFee(4999), 500);
    });
    test('is free exactly at the threshold', () {
      expect(shippingFee(5000), 0);
    });
    test('rejects a negative subtotal', () {
      expect(() => shippingFee(-1), throwsArgumentError);
    });
  });
}

With >= changed to >, dart test reported (exit code 1):

00:00 +1 -1: test/pricing_test.dart: shippingFee is free exactly at the threshold [E]
  Expected: <0>
    Actual: <500>

  package:matcher              expect
  test/pricing_test.dart 10:7  main.<fn>.<fn>

00:00 +2 -1: Some tests failed.

Dart's matcher labels the sides Expected and Actual, so the argument order matters less than in unittest.

If package:test is not available offline, the fallback is plain assert, but check how your runtime treats it. In this run, all three of these printed "reached the end" with no error:

Inside dart test, asserts are on: a test expecting AssertionError passed. PHPUnit is not installed on this machine, so only assert with zend.assertions=1 was run for PHP.

How long does a suite take as it grows?

A common excuse is "tests are slow". I generated files with N trivial tests (each calls split_bill once) and timed python3 -m unittest as a whole process: 6 runs each, the first (cold) run dropped, median of the other 5. Python 3.9.6, Apple Silicon Mac (M1 Pro).

Up to 100 tests you are measuring Python's startup (about 45 ms). After that, each extra test costs about 19 to 24 microseconds. With -v, the 10,000-test run took 432 ms instead of 272 ms (a separate measurement): printing one line per test costs more than running it.

The same experiment with dart test (3 runs each after a warm-up): 0.91 to 0.95 s for 1 test, 1.07 to 1.24 s for 100, 1.84 to 1.94 s for 1,000 and 6.3 to 6.4 s for 10,000. Dart's fixed cost is larger (it compiles and starts an isolate), and its reporter does more work per test.

The conclusion: small unit tests are cheap. What makes suites slow is a few tests that sleep, touch the network or a real database. Lecture 4 shows how to replace those.

What NOT to test

Tests are code you must maintain. These usually cost more than they return:

• The language and the standard library. A test that divmod(7, 2) == (3, 1) tests Python, not you.
• Private helpers directly. Test them through the public function. Then you can refactor the helper without rewriting tests.
• Exact wording you do not own. Matching Python's own message "can't multiply sequence…" would break on the next Python release. Our test matches the message we wrote.
• Trivial code with no logic, like a property that returns a field.
• The implementation instead of the result. "split_bill calls divmod once" breaks on harmless refactors. "The shares add up to the total" survives them.
• Random or time-dependent output, as-is. Fix the seed or the clock first (lecture 5).

What you should always test: every bug you fix (write the failing test first, then the fix), boundaries of every rule with a number in it, and error handling a caller relies on.

Honest limits / version notes

• Python 3.9.6 was the main runtime; Python 3.12.14 was used only to compare messages. python -m unittest returning exit code 5 for "no tests ran" and the --durations option are both new in 3.12; on 3.9, --durations failed with unrecognized arguments. The first --durations run on 3.12 showed 0.335 s for the doctest (a cold start); later runs showed 0.001 to 0.002 s.
• assertRaisesRegex and subTest exist since Python 3.2 and 3.4.
• Dart 3.11.5 with test 1.30.0 from the local pub cache; nothing was downloaded. Other Dart setups may resolve a different version.
• Timings are from one Apple Silicon Mac (M1 Pro). Absolute numbers will differ on your machine; the shape (fixed startup cost, then a few microseconds per tiny test) should not.
• pytest is the most popular third-party runner and has nicer diffs, but it is not installed here and was not run.

Exercises

1. Add a test that split_bill(5, 5) returns five shares of 1, and one that split_bill(4, 5) returns [1, 1, 1, 1, 0]. Before running them, predict whether they pass.
2. Write one test that fails if someone changes >= to > in shipping_fee, and explain why shipping_fee(4999) alone cannot catch that bug.

Answers

1. Both pass. divmod(5, 5) is (1, 0), so there is no extra cent; divmod(4, 5) is (0, 4), so the first four people get 0 + 1 and the last gets 0. If your prediction was wrong, write down why: that is the boundary you had not thought about.
2. self.assertEqual(shipping_fee(5000), 0). With 4999, both >= and > give the same answer (599), so the test cannot tell them apart. Only the value exactly at the threshold distinguishes the two operators.

Where next

• Previous: Lecture 2: print, logging and the debugger
• Next: Lecture 4: test doubles, mocks and fixtures
• Start of the series: Lecture 1: reading error messages and stack traces
• Later: flaky tests and git bisect
• Related: Python functions, scope and default arguments (mutable defaults are a classic thing to test), profiling and optimization in Python, and the programming learning roadmap for where testing fits.