Float Summation Error, NaN and -0.0, Measured

Numbers lecture 6: summing 10^7 doubles drifted 396 ulps, Python 3.12 sum() was exact, -ffast-math deleted Kahan and turned 0/0 into 1, and NaN broke sorting.

Series: Numbers & Money lecture 6/6 (advanced). This last lecture is for people who add up a lot of floating-point numbers: analytics totals, sensor data, scores, statistics, game physics, or monthly reports that must match last month's. By the end you will know how fast summation error grows, which algorithm removes it and what that costs, what NaN and negative zero do to sorting, max and JSON, and what -ffast-math silently removes from your C code. It closes with a checklist for the whole series.

Lecture 1 showed that 0.1 + 0.2 is 0.30000000000000004. One wrong digit in the 17th place looks harmless. This lecture is about what happens when you do that ten million times, and about the two special values, NaN and -0.0, that slip through code reviews because they compare in surprising ways. Everything was run on an Apple Silicon Mac (M1 Pro): Python 3.9.6 and 3.12.14, Apple clang 21, the macOS JavaScriptCore shell jsc, PHP 8.5.6, Java 26, .NET 10.0.14, Swift 6.4.2 and SQLite 3.51.0. Random data uses random.Random(42), so you can reproduce every number.

0.1 added ten million times

The simplest test: add 0.1 to an accumulator 10,000,000 times. The exact answer is 1,000,000.

Runtime Result
Python 3.9.6 loop and sum() 999999.9998389754
Python 3.12.14 sum() 1000000.0
Python math.fsum (both versions) 1000000.0
jsc for loop 999999.9998389754
PHP 8.5.6 loop (var_export) 999999.9998389754
C# double loop 999999.9998389754
C# decimal loop with 0.1m 1000000.0
SQLite 3.51.0 SELECT sum(0.1) over 10^7 rows 1000000.0

The error is 0.00016, which is visible to anyone checking a report total. Note the quiet change: Python 3.12's built-in sum() switched to compensated summation (Neumaier's variant), so the same script printed a different total after a Python upgrade. SQLite's sum() also gave the exact answer here; recent SQLite versions use the same family of compensated summation (Kahan-Babuska-Neumaier). PHP's default echo printed 999999.99983898, hiding three digits: var_export showed the full value.

A second classic: math.fsum([1e100, 1.0, -1e100]) is 1.0, while the plain loop and Python 3.9 sum() give 0.0, because 1.0 vanished when added to 1e100. Python 3.12 sum() returned 1.0.

How the error grows with N

Each + rounds its result to the nearest double, so a plain loop makes one rounding error per element. The errors partly cancel, so the total error tends to grow slower than N, but it grows. We measured it in ulps (units in the last place: the gap between the true sum and the next double), against math.fsum, which returns the correctly rounded sum. As a check, an exact Fraction sum of the first 10^6 values matched fsum bit for bit.

def naive(xs):
    s = 0.0
    for x in xs: s += x
    return s

def kahan(xs):
    s = c = 0.0
    for x in xs:
        y = x - c; t = s + y; c = (t - s) - y; s = t
    return s

def neumaier(xs):
    s = c = 0.0
    for x in xs:
        t = s + x
        if abs(s) >= abs(x): c += (s - t) + x
        else: c += (x - t) + s
        s = t
    return s + c

Uniform values in [0, 1) (error in ulps against fsum):

N naive loop sorted ascending pairwise Kahan Neumaier
10^3 1 2 0 0 0
10^4 0 5 0 0 0
10^5 54 50 0 0 0
10^6 33 102 0 0 0
10^7 396 64 1 0 0

Mixed signs and magnitudes (uniform(-1, 1) * 10**randint(-8, 8)):

N naive loop sorted ascending sorted by abs pairwise Kahan Neumaier
10^3 15 372 4 8 1 0
10^5 64 5,155 3 2 0 0
10^6 129 3,912 55 0 0 0
10^7 532 54,006 44 1 0 0
Bar chart of summation error in ulps for ten million values: naive loop 396, sorted ascending then naive 64, pairwise 1, Kahan 0, Neumaier and Python 3.12 sum 0, math.fsum 0.
Error of each summation method against the correctly rounded sum (math.fsum, checked against exact Fraction arithmetic at 10^6), for 10^7 uniform random doubles in [0,1) with seed 42. One ulp here is about 9.3e-10. The plain loop is also what Python 3.9 sum() returned; Python 3.12 sum() was exact.

Two lessons. First, the error is not monotonic in N (33 ulps at 10^6, 54 at 10^5); it is a random walk, which is why a test that passes for one dataset fails for another. Second, the folk advice "sort ascending before adding" helped for positive numbers and made things a hundred times worse for mixed signs, because the big negative values get added first and the result is a large number that swallows the small ones. Sorting by absolute value helped, but still left 44 ulps. Compensated algorithms beat both.

What exactness costs

For 10^6 values, median of 5 runs, Apple Silicon Mac (M1 Pro):

Method Python 3.9.6 Python 3.12.14
built-in sum() 2.5 ms (not compensated) 2.5 ms (compensated)
plain for loop 19.2 ms 11.8 ms
math.fsum 16.6 ms 7.3 ms
Kahan for loop 59.8 ms 32.5 ms
sum(sorted(xs)) 169.0 ms 153.4 ms
sum() of Decimal values 39.2 ms 41.7 ms

In Python the answer is easy: on 3.12, sum() is both the fastest and compensated; on older versions use math.fsum, which is exact and still faster than a hand-written loop. Sorting is the slowest option and not the most accurate.

In C, with 10^7 doubles (best of 5, clang -O2): plain loop 9.71 ms, Kahan 37.37 ms. Kahan costs about four times as much because each step depends on the previous one, so the CPU cannot overlap additions. That is a fair price for a ledger, and too high for a game loop.

float32 vs float64

A float (32 bits) keeps about 7 significant decimal digits; a double (64 bits) about 16. The same 10^7 values, summed in C:

double naive             sum=4998269.0432556057  abs err=-3.69e-07  9.71 ms
double kahan             sum=4998269.0432559745  abs err=0          37.37 ms
float naive              sum=4998096.5           abs err=-173       9.32 ms
float data, double acc   sum=4998269.0432599643  abs err=3.99e-06   9.33 ms
float: add 1.0f twenty million times -> 16777216.0

Summing floats in a float accumulator was off by 173 out of five million. Above 4,194,304 (2^22) the spacing between neighbouring floats is 0.5, so every addend in [0, 1) is rounded to 0, 0.5 or 1 before it counts. Storing the data as float but accumulating in double cost nothing extra in time and brought the error down to 4e-6, which is only the rounding of the stored values themselves. The last line is the most dramatic: a float counter stops at 2^24 = 16,777,216, because 16,777,216 + 1 rounds back to 16,777,216. Java printed 1.6777216E7 for the same loop, and jsc's Math.fround(16777217) returned 16777216.

Rule: float32 is fine for storing large arrays (ML weights, vertices) and for GPU math; accumulate totals, means and variances in float64.

-ffast-math: faster, different, and NaN-blind

-ffast-math tells clang it may treat floating-point arithmetic like real-number algebra: reorder additions, assume there are no NaNs or infinities, and ignore the sign of zero. We compiled the same files three ways:

Output -O2 -O2 -ffast-math
double naive sum (10^7) err -3.69e-07, 9.71 ms err 3.45e-08, 1.75 ms
double Kahan sum err 0, 37.37 ms err 3.45e-08, 1.58 ms
float naive sum err -173 err 18
1.0f added 2×10^7 times 16777216.0 20000000.0
z/z with z = atof("0") nan 1.000000
v != v for v = strtod("nan") 1 0

What happened: with reassociation allowed, clang vectorized the plain loop into several partial sums, which is both 5.5 times faster and, by luck, more accurate (it behaves like a shallow pairwise sum). But it also proved that Kahan's correction (t - s) - y is algebraically zero and deleted it: the Kahan function returned exactly the same value as the plain one. The float counter loop was turned into a computed 20,000,000, which is the "right" answer that strict IEEE arithmetic cannot produce. And because the compiler assumes NaN does not exist, 0/0 computed at run time became 1.0, and the self-comparison NaN check returned false. Only the library call isnan() still worked in our noinline helper, and clang printed warning: use of NaN is undefined behavior due to the currently enabled floating-point options [-Wnan-infinity-disabled] when we used the NAN macro.

So: never compile compensated summation, interval code or NaN checks with -ffast-math. If you want the speed, use explicit multiple accumulators or pairwise summation, which give vectorization without lying to the compiler. More on how optimizers exploit assumptions: C undefined behavior and optimization.

NaN: comparison, sorting, max and JSON

NaN ("not a number") comes from 0/0, inf - inf, 0 * inf, failed parses and missing data. Its defining property: every comparison with NaN is false, including NaN == NaN. Sort algorithms and max assume a consistent ordering, so they fail in ways that depend on where the NaN sits:

Python  sorted([3, nan, 1, 2])          -> [3, nan, 1, 2]        (unchanged!)
Python  sorted([nan, 3, 1, 2])          -> [nan, 1, 2, 3]
Python  max([nan, 1, 2]) / max([1, 2, nan]) -> nan / 2
Python  statistics.median([1, nan, 3, 2, 5]) -> 2
Swift   [3, .nan, 1, 2].sorted()        -> [3.0, nan, 1.0, 2.0]
jsc     [3, NaN, 1, 10, 2].sort((a, b) => a - b) -> 3,NaN,1,2,10
jsc     [3, NaN, 1, 10, 2].sort()       -> 1,10,2,3,NaN          (string order)
PHP     sort([3, NAN, 1, 2])            -> [NAN, 1, 2, 3]
PHP     max(NAN, 1) / max(1, NAN)       -> 1 / NAN
Java    Arrays.sort(double[])           -> [-0.0, 0.0, 1.0, 2.0, 3.0, NaN]
C#      Array.Sort(double[])            -> NaN, 0, -0, 1, 2, 3
C#      Math.Max(NaN, 1) / LINQ Max([1, NaN, 2]) -> NaN / 2

Java and C# define a total order for sorting (NaN last in Java, first in C#), which is why they sort correctly; everyone else returns garbage without an error. Even median returned a plausible-looking 2. The dangerous part is not the NaN you see; it is the wrong answer next to it.

Membership is also odd. In Python, nan in [nan] is True (the same object is found by identity), but float('nan') in [float('nan')] is False, and {float('nan'), float('nan')} has two elements. In jsc, [NaN].includes(NaN) is true but [NaN].indexOf(NaN) is -1, and isNaN("abc") is true while Number.isNaN("abc") is false.

JSON has no NaN. Each language reacts differently:

Call Result
Python json.dumps([nan, inf, -0.0]) [NaN, Infinity, -0.0] (not valid JSON)
Python json.dumps(nan, allow_nan=False) ValueError: Out of range float values are not JSON compliant
jsc JSON.stringify([NaN, Infinity, -0]) [null,null,0]
jsc JSON.parse('[NaN]') SyntaxError: JSON Parse error: Unexpected identifier "NaN"
PHP json_encode(NAN) false, Inf and NaN cannot be JSON encoded
.NET JsonSerializer.Serialize(double.NaN) ArgumentException: .NET number values such as positive and negative infinity cannot be written as valid JSON.
SQLite INSERT … VALUES (9e999 - 9e999) stored as NULL

So a Python service that emits NaN breaks a JavaScript client's JSON.parse, a JavaScript client silently turns NaN into null, and SQLite turns it into NULL, which sum() and avg() then skip. Decide at the boundary: reject NaN on input, and send null deliberately when a value is missing. Lecture 2 covered the related JSON problem of large integer IDs; see also how prepared statements change JSON number types in PHP.

Negative zero

IEEE 754 has two zeros. -0.0 == 0.0 is true everywhere we tested, but the sign leaks out:

  • 1 / -0.0 is -inf in C and -Infinity in jsc.
  • Python repr(-0.0) is -0.0; max(0.0, -0.0) returned 0.0 and max(-0.0, 0.0) returned -0.0 (the first argument wins on a tie).
  • Formatting a tiny negative value prints a minus sign: Python f"{-0.004:.2f}", jsc (-0.004).toFixed(2), Java String.format("%.2f", -0.004) and .NET ToString("F2") all printed -0.00; jsc Intl.NumberFormat("en-US", {maximumFractionDigits: 2}) printed -0. PHP number_format(-0.004, 2) printed 0.00.
  • Java Double.valueOf(0.0).equals(-0.0) is false, so a HashMap<Double, …> keeps both keys.
  • JSON.stringify(-0) is 0; PHP json_encode(-0.0) is -0; SQLite stored -0.0 as 0.0.

A refund of 0.004 shown as -0.00 € is the most common real complaint. Round to the currency's minor unit first (lecture 4), then add + 0.0 or test for zero before formatting.

Common mistakes → what actually happens → fix

Mistake What we saw Fix
Summing millions of floats in a loop 396 ulps off at 10^7 math.fsum, Neumaier, or Python 3.12+ sum()
"Sort first, then add" mixed signs: 54,006 ulps off compensated sum instead
float32 accumulator 173 off out of 5 million accumulate in float64
float counter stuck at 16,777,216 integer counter
Kahan + -ffast-math correction deleted no fast-math for that file
sorted() / max() with NaN unsorted list, order-dependent max filter isnan first
json.dumps with NaN invalid JSON for other parsers allow_nan=False, send null
Showing a rounded negative tiny value -0.00 round, then normalize zero
Comparing totals with == Python 3.9 vs 3.12 totals differ integer cents, or math.isclose

When to use int, Decimal or float

Decision diagram. Start with what the number represents. Money, tax and anything that must match a receipt: integer minor units, or Decimal when you need rates and fractional cents. Counts and IDs: 64-bit integers, and strings in JSON when they can exceed 2 to the 53rd. Measurements, statistics and graphics: float64; float32 only for storage or GPU, accumulating in float64. When summing many floats, use fsum, Neumaier or pairwise summation, and reject NaN at the input boundary.
Which number type to use. Every branch is backed by a result in this series: 0.1 added ten million times gave 999999.9998389754 as a double and exactly 1000000.0 as a C# decimal, float32 stopped counting at 16777216, and NaN broke sorting in Python, Swift and jsc.
  • Money, tax, anything that must match a receipt: integer minor units, or Decimal where you need rates and fractional cents (lecture 4). C# decimal added 0.1 ten million times and got exactly 1000000.0.
  • Counts and IDs: 64-bit integers; strings in JSON if they can pass 2^53 (lecture 2).
  • Measurements, statistics, ML, graphics: float64. Use float32 only for storage or GPU work, and accumulate in float64. Sum with fsum or a compensated method when N is large or the result is compared across runs.

Series wrap-up checklist

  1. Know your type: every literal like 0.1 is already rounded in binary (lecture 1). Compare floats with a tolerance, never ==.
  2. Know your limits: 2^31, 2^63, and 2^53 for JavaScript numbers and JSON (lecture 2).
  3. Pick the rounding mode on purpose (half-even vs half-up) and round once, at the end, following your local tax rules (lecture 3).
  4. Store money as integer minor units or Decimal, never as float (lecture 4).
  5. Never store formatted numbers; parse user input with an explicit locale and a strict pattern (lecture 5).
  6. Sum with a compensated algorithm, keep NaN out at the boundary, normalize -0.0 before display, and keep -ffast-math away from numeric-correctness code (this lecture).
  7. Write tests for the edge values: 0.1, 2.5, -0.0, NaN, 2^53 + 1, empty input. See your first unit tests.

Honest limits / version notes

  • Python's sum() uses compensated (Neumaier) summation for floats from 3.12; we ran 3.9.6 (plain) and 3.12.14 (compensated).
  • SQLite's exact sum(0.1) result was measured on 3.51.0 only; older SQLite versions used a plain sum, so check yours.
  • The -ffast-math results are from Apple clang 21 on arm64. GCC and other targets reassociate differently; the deleted-Kahan and NaN-folding behaviour is common, but the exact numbers are not portable. Not run on Linux.
  • Error in ulps depends on the data. Our two datasets come from random.Random(42); other distributions give other numbers, but the ranking of methods was the same.
  • Timings are single-machine measurements (M1 Pro, best or median of 5) and include no warm-up control beyond that.

Exercises

1. A dashboard computes daily revenue by summing 2 million float order totals in Python 3.9 and compares it with yesterday's number from a Python 3.12 job. They differ in the 10th digit. Name the cause and two fixes.

2. Without running it: what does Python print for max([float("nan"), 5.0]) and for max([5.0, float("nan")]), and what would you write instead?

Answers

1. Python 3.12's sum() is compensated and 3.9's is not, so the totals differ in the low digits (we saw 999999.9998389754 vs 1000000.0 for the 0.1 test). Fixes: use math.fsum in both jobs; or, better for revenue, store integer minor units and sum integers, which are exact in every version.

2. nan and 5.0, in that order: max keeps the first element and replaces it only when x > current is true, which never happens with NaN on either side. Use max(x for x in xs if not math.isnan(x)), and decide explicitly what a NaN in the input means (usually: reject it earlier).

Continue

FAQ

Does sorting the numbers before adding them fix float summation?

Not reliably. For ten million positive values in [0,1), sorting ascending cut the error from 396 to 64 ulps, but for mixed signs and magnitudes sorting ascending made it worse: 54,006 ulps against 532 for the plain loop. Kahan, Neumaier, math.fsum and Python 3.12's sum() were exact or within one ulp in every case we ran.

Why does Python's sorted() return an unsorted list when there is a NaN?

Every comparison with NaN is False, so the sort algorithm gets inconsistent answers. sorted([3, nan, 1, 2]) returned [3, nan, 1, 2] unchanged, max([nan, 1, 2]) returned nan but max([1, 2, nan]) returned 2, and statistics.median([1, nan, 3, 2, 5]) returned 2. Swift and jsc broke the same way. Filter NaN with math.isnan before sorting.

Is -ffast-math safe for numeric code?

Only if your code never relies on exact rounding or NaN. With clang -O2 -ffast-math, Kahan summation compiled to the same result as the plain loop, 0.0/0.0 computed at run time printed 1.000000 instead of nan, and v != v was 0 for a NaN read from input. The plain sum got 5.5 times faster and happened to be more accurate.

  • #numbers
  • #advanced
  • #floating point
  • #nan
  • #kahan summation
  • #python
  • #c
  • #javascript
  • #sqlite

t.laboratory

A technical blog that records real problems from the apps, servers and services we build and run — and the code that fixed them.

About · Apps

↑