Naive Bayes in Pure Python on 86 Real Blog Posts

If you want to understand naive Bayes well enough to know when it is enough, this post builds one in about 70 lines of standard-library Python and tests it on real text: the 86 English posts on this blog, predicting each post's category from its body. The headline result is 84.9% leave-one-out accuracy against a 43.0% majority-class baseline, trained in 6.5 ms. The more useful result is where it scored zero, and the single word that caused most of it.

This is lecture 3 of the series Machine learning from scratch (Python stdlib only). Lecture 1 was linear regression with gradient descent, lecture 2 was k-means clustering. Next comes the perceptron and a small neural network, and the series ends with a TF-IDF and BM25 search engine built on the same posts.

Environment for every number below: Python 3.9 standard library, Apple Silicon Mac (M1 Pro). No numpy, no scikit-learn. The code avoids deprecated APIs and runs unchanged on 3.12+.

The dataset: our own 86 posts, 9 unbalanced categories

The blog is generated from Markdown files with YAML frontmatter (the generator is described in a static site generator for a multilingual blog). Each file has a category: line. That is our label. The input is the body text only, without the title, description or FAQ, so the model cannot just read the answer from the slug.

We froze a snapshot of content/en/*.md when this was written. There were 86 posts:

This is a hard shape for any classifier. Always answering "csharp" already gets 43.0%. Four categories have 2 to 4 examples. And this post, plus its siblings published the same day, will join the ai class, so your counts will differ if you run the code later.

With 86 documents, a 70/30 split leaves 26 test posts, and a category with 2 posts would get one test example at best. So the main evaluation is leave-one-out (LOO): train on 85, predict the one left out, repeat 86 times. I also ran a stratified 70/30 split with seeds 0 to 9 as a cross-check (results below).

The whole classifier in about 70 lines

Multinomial naive Bayes scores a document d for class c as

log P(c) + Σ count(w, d) · log P(w | c)

and picks the class with the highest score. P(c) is the share of training posts in class c. P(w | c) is how often word w appears among all words in class c, with a pseudo-count alpha added to every word (Laplace smoothing). "Naive" means it pretends words are independent given the class. They are not, but the ranking is often still right.

Save this as nb.py:

"""Multinomial naive Bayes: predict a blog post's category from its body.
Python standard library only. Usage: python nb.py content/en [--keep-code]
"""
import math, re, sys, time
from collections import Counter, defaultdict
from pathlib import Path

FENCE = re.compile(r"^```.*?^```", re.S | re.M)
WORD = re.compile(r"[a-z][a-z0-9_]*[#+]*")

def load(folder):
    docs = []
    for p in sorted(Path(folder).glob("*.md")):
        _, front, body = p.read_text(encoding="utf-8").split("---", 2)
        cat = re.search(r"^category:\s*(\S+)", front, re.M).group(1)
        docs.append((p.stem, cat, body))
    return docs

def tokenize(body, keep_code=False):
    if not keep_code:
        body = FENCE.sub(" ", body)
    body = re.sub(r"https?://\S+|\[\[fig:[^\]]*\]\]", " ", body)
    return WORD.findall(body.lower())

class NaiveBayes:
    def __init__(self, alpha=1.0):
        self.alpha = alpha

    def fit(self, X, y):                      # X: list of Counter, y: labels
        self.n_docs = Counter(y)
        self.words = defaultdict(Counter)     # class -> word counts
        for counts, c in zip(X, y):
            self.words[c].update(counts)
        self.vocab = set().union(*X)
        self.total = {c: sum(w.values()) for c, w in self.words.items()}
        return self

    def log_scores(self, counts):
        N, V, a = sum(self.n_docs.values()), len(self.vocab), self.alpha
        out = {}
        for c in self.n_docs:
            s = math.log(self.n_docs[c] / N)
            denom = self.total[c] + a * V
            for w, k in counts.items():
                if w not in self.vocab:
                    continue                  # unseen in training: skip
                num = self.words[c][w] + a
                if num == 0:                  # only possible when alpha == 0
                    s = -math.inf
                    break
                s += k * math.log(num / denom)
            out[c] = s
        return out

    def predict(self, counts):
        sc = self.log_scores(counts)
        return max(sc, key=sc.get)

def leave_one_out(X, y, alpha=1.0):
    preds, dead = [], 0
    for i in range(len(X)):
        m = NaiveBayes(alpha).fit(X[:i] + X[i+1:], y[:i] + y[i+1:])
        sc = m.log_scores(X[i])
        if all(v == -math.inf for v in sc.values()):
            dead += 1                         # every class has probability 0
        preds.append(max(sc, key=sc.get))
    return preds, dead

if __name__ == "__main__":
    folder, keep = sys.argv[1], "--keep-code" in sys.argv
    docs = load(folder)
    y = [c for _, c, _ in docs]
    X = [Counter(tokenize(b, keep)) for _, _, b in docs]
    print("docs", len(docs), dict(Counter(y).most_common()))
    t = time.perf_counter()
    preds, dead = leave_one_out(X, y)
    el = time.perf_counter() - t
    acc = sum(p == t_ for p, t_ in zip(preds, y)) / len(y)
    print(f"LOO accuracy {acc:.3f} ({sum(p == t_ for p, t_ in zip(preds, y))}/{len(y)})"
          f"  {el:.2f}s for {len(y)} fits")

$ python3 nb.py content/en
docs 86 {'csharp': 37, 'algorithm': 15, 'regex': 9, 'server': 7, 'winapi': 6, 'app-ops': 4, 'web': 4, 'ai': 2, 'flutter': 2}
LOO accuracy 0.849 (73/86)  0.74s for 86 fits

The vocabulary is built from the training fold only, and a word never seen in training is skipped. Skipping it is the same for every class, so it does not change the ranking.

Tokenizing: strip the code blocks or keep them?

A tech blog is half prose, half code. The tokenizer lower-cases, keeps c# and c++ as single tokens (the [#+]* tail), and removes URLs and figure markers. The open question was code fences: using System; is a very strong hint for C#, but code also brings noise like i, x and printf that every category shares.

The same score, but not the same mistakes. Keeping the code fixed one post, winapi-getcurrentdirectory-getmodulefilename (it had been called csharp), and broke another: litespeed-htaccess-rewrite-non-ascii-urls, a server post, became web by a margin of 75.6: its rewrite-rule code adds tokens such as html, index and htaccess, which the WordPress-migration posts in web also use. The code was a wash. I kept it stripped for the rest because the prose-only model is easier to explain. On a blog where every category has its own language, keeping code would probably win.

Without Laplace smoothing, every post scores zero

Set alpha=0 and P(w | c) becomes exactly 0 for any word that class never used. One such word makes the whole product 0, so the log score is -inf. This is the alpha sweep, run with the same leave_one_out function:

for a in (0, 1e-6, 0.01, 0.1, 0.5, 1, 2, 5):
    preds, dead = leave_one_out(X, y, a)
    ok = sum(p == t for p, t in zip(preds, y))
    print(f"  alpha={a:<6} acc={ok}/{len(y)}={ok/len(y):.3f} all-classes-zero={dead}")

  alpha=0      acc=3/86=0.035 all-classes-zero=86
  alpha=1e-06  acc=70/86=0.814 all-classes-zero=0
  alpha=0.01   acc=73/86=0.849 all-classes-zero=0
  alpha=0.1    acc=74/86=0.860 all-classes-zero=0
  alpha=0.5    acc=73/86=0.849 all-classes-zero=0
  alpha=1      acc=73/86=0.849 all-classes-zero=0
  alpha=2      acc=65/86=0.756 all-classes-zero=0
  alpha=5      acc=54/86=0.628 all-classes-zero=0

Not some posts: all 86 had probability zero for every category. The 3 "correct" answers are max() picking the first key when all scores tie at -inf. For regex-lookaround, the regex class itself was killed by 35 ordinary words that the other 8 regex posts never used, such as ahead, behind, think and colon. Strong regex words in the post, like lookahead, cannot outvote a single zero.

An alpha of one millionth already fixes the failure (81.4%). Large values hurt: at alpha=5 the pseudo-counts drown the real counts and accuracy drops to 62.8%. Keep that in mind for the error analysis.

Log probabilities: the product hits 0.0 after about 175 words

The textbook says "use logs to avoid underflow". Here is the underflow. For the algorithm-sorting post (923 tokens after stripping code), multiply the raw probabilities in a float and note when it becomes 0:

    for j, w in enumerate(toks, 1):
        q = (m.words[c][w] + 1) / denom
        p *= q
        logp += math.log(q)
        if p == 0.0 and zero_at is None:
            zero_at = j

underflow doc tokens 923
  algorithm: product=0.0 zero after token 175  log-sum=-5934.6
  csharp: product=0.0 zero after token 174  log-sum=-6605.2
  min positive float 5e-324 log10 = -323.3062153431158
  token count min/median/max 609 1277 2428

A word probability is around 1e-3 to 1e-4, so 175 of them take the product past the smallest positive double, about 10^-323.3. The true value, e^-5934.6, is about 10^-2577. The shortest post in the set has 609 tokens, so every post underflows without logs, and every class ties at 0.0. In log space the two classes differ by 670.6, a decision no float product could make.

Accuracy and the confusion matrix

LOO with alpha=1, code stripped. Rows are the true category, columns the prediction, in the order csharp, algorithm, regex, server, winapi, app-ops, web, ai, flutter:

  csharp     [37, 0, 0, 0, 0, 0, 0, 0, 0]
  algorithm  [0, 15, 0, 0, 0, 0, 0, 0, 0]
  regex      [0, 0, 9, 0, 0, 0, 0, 0, 0]
  server     [0, 0, 0, 7, 0, 0, 0, 0, 0]
  winapi     [1, 0, 0, 0, 5, 0, 0, 0, 0]
  app-ops    [0, 0, 0, 3, 0, 0, 1, 0, 0]
  web        [0, 0, 0, 4, 0, 0, 0, 0, 0]
  ai         [0, 0, 0, 2, 0, 0, 0, 0, 0]
  flutter    [0, 0, 0, 2, 0, 0, 0, 0, 0]

84.9% overall hides a split result. The five large categories are 73 of 74 correct. The four small ones are 0 of 12, and 11 of those 12 went to server. Averaging the per-class accuracy (balanced accuracy) gives 53.7%, a fairer number for this data.

The stratified 70/30 split agrees with LOO: over seeds 0 to 9 the test accuracy ranged from 80.8% to 88.5%, mean 84.2% (seed 0: 60 train, 26 test, 21 correct). With only 26 test posts, a single post moves the score by 3.8 points, which is why LOO is the main number here.

The most informative words per class

For each class, rank words seen at least 3 times in it by log P(w | c) - log P(w | not c):

  csharp     csharp, linq, ref, lambda, dialogresult, func, goto, collections
  algorithm  nodes, dfs, linked, tail, bfs, asan, algorithm, vertex
  regex      quantifier, pcre2, w+, regexp, backreferences, regexoptions, regex, backreference
  server     localhost, hostname, connection, evening, sql, clients, prepared, budget
  winapi     mingw, sendmessage, getkeystate, w64, findwindow, getasynckeystate, winapi, gcc
  app-ops    landing, fcm, ua, ping, cooldown, token, admob, ctr
  web        wp, category, pillow, ini, noto, hreflang, sans, particle
  ai         human, schema, claude, tasks, agent, verifier, mcp, duration
  flutter    consent, slashed, signboard, curl, …

The big classes have words that are their topic. The small classes have words of one post: pillow and noto come from the OG image card post, evening and budget from the MySQL connection-limit post. With 2 to 4 examples, "informative" means "unusual in that one article". That is the model memorising, not learning a topic.

Error analysis: the word "the" voted for server

I expected the misclassified posts to be fooled by topic words: flutter-migration-16-apps-what-broke talks about servers, so it goes to server. Wrong. Breaking the score into per-word contributions (flutter vs server, positive pushes to server):

flutter-migration-16-apps-what-broke: flutter=-9309.5 server=-8449.2 margin=860.3
  pushes to server [('the', 83, 122.9), ('and', 34, 44.4), ('a', 30, 33.8), ('on', 15, 27.3), …]
  pushes to flutter [('android', 7, -5.1), ('ios', 6, -4.4), ('dart', 2, -3.6), ('debug', 2, -3.6), …]
  train docs flutter 1 tokens 852 | server 7 tokens 8271

The 83 copies of the alone outweigh every Flutter word combined. The cause is the smoothing, not the topic. In this fold the vocabulary has 6,101 words. The flutter class has one training post with 852 tokens, so with alpha=1 its denominator is 852 + 6,101, and 88% of its probability mass is pseudo-counts:

Both classes use the equally often, but smoothing makes it 4.4 times more likely under server. Every common word repeats this, so a small class loses on grammar before topic words are even counted. The same pattern explained the other errors I broke down: ai-coding-agent-live-server-guardrails lost to server by 892.2 while agent (19 times) pushed hard the other way, and winapi-getcurrentdirectory-getmodulefilename lost to csharp by 143.2 even though buffer, getcurrentdirectory and getmodulefilename all pointed to winapi.

Three fixes I tried, all with LOO:

The prior made no difference at all. With 1,000-word documents the likelihood sum is in the thousands and log P(c) is only between -0.8 and -4.4. Lowering alpha helped a little. Nothing got the ai or flutter posts right, because one training example is not a class.

Training and prediction time

  fit on 86 docs: 6.50 ms
  predict: 1.651 ms/doc
  tokenize all: 34.6 ms

Training is just counting, so 86 posts take 6.5 ms. The whole 86-fold LOO took 0.74 s. Prediction is 1.65 ms per post because the pure-Python loop visits every distinct word for each of 9 classes. Tokenizing is the slowest step. For a blog, all of this is free.

Japanese: character bigrams instead of words

Japanese has no spaces, so split() returns whole phrases. The Japanese versions of the same 86 posts were tested two ways, Latin words kept whole in both:

def bigrams(body):
    out = []
    for run in re.findall(r"[^\s\W\d_a-z]+|[a-z][a-z0-9_]*[#+]*", ja_clean(body)):
        if run[0].isascii():
            out.append(run)                   # keep latin words whole
        else:
            out += [run[i:i+2] for i in range(len(run) - 1)] or [run]
    return out

sample bigrams: ['ナイ', 'イー', 'ーブ', 'ブベ', 'ベイ', 'イズ', 'ズで', 'で記', '記事', '事を', 'を分', '分類']
sample split  : ['ナイーブベイズで記事を分類する', 'c#', 'の例']
ja whitespace split          acc 54/86=0.628 balanced=0.281
ja char bigrams              acc 70/86=0.814 balanced=0.481

Bigrams lifted Japanese from 62.8% to 81.4%, close to English (84.9%). Splitting on whitespace failed on server (0/7) and winapi (0/6) because a phrase like the one above is never seen twice. With bigrams, server was 7/7 again, but winapi dropped to 2/6, all lost to csharp. The WinAPI posts are in C, and their Japanese explanations (変数, 関数, 配列 and so on) read much like the C# lessons. The API names that separate them stay one token each, while every Japanese sentence turns into many overlapping bigrams, so in Japanese the prose probably outvotes the API names. A morphological analyser such as MeCab would give real words, but that is a dependency, and bigrams got most of the way with 10 lines.

Naive Bayes, logistic regression or embeddings?

Logistic regression is the next step because naive Bayes treats each word as independent evidence. Our errors were exactly that: dozens of grammar words each adding a little in the same direction. Logistic regression fits all weights together and would push the toward zero. The perceptron lecture builds the same kind of linear model by hand.

Honest limits

• The data is small. 86 posts, and 4 classes with 2 to 4 posts. In LOO the ai and flutter classes train on a single post. No method fixes that. One correct or wrong post moves overall accuracy by 1.2 points.
• The categories overlap. The 19 posts in server, web, app-ops, ai and flutter were written within three days, mostly about the same site and app migration. When I merged them into one ops label, LOO accuracy went to 98.8% (85/86). The only error left was the same winapi post. The classifier was not weak. The labels mark editorial choices that the text alone barely supports.
• The test is not fully independent. Posts in a series share a template (the 37 C# lessons follow one structure), which makes csharp and algorithm easy. A new kind of C# post could do worse.
• The snapshot moves. This post and its siblings are ai posts, so re-running on the live folder gives different counts.
• Accuracy is the wrong single number here. Report balanced accuracy, or per-class recall, whenever the classes are this unbalanced.

The next lecture keeps the bag of words but replaces counting with learned weights: from the perceptron to a small neural network.