Encoding Detection Is Guesswork: Accuracy and Speed

Series: Text & Encoding lecture 6/6 (advanced). For developers who import files from systems they do not control (an old CRM export, a partner's CSV, a mail archive) and have been told to "just detect the encoding". By the end you will know how often detection is right, why it fails on short strings, how fast conversion really is, and how to build an import pipeline that does not depend on luck.

If you have ever written mb_detect_encoding($data) and trusted the answer, or seen It’s arrive as It痴, this lecture is for you. The core problem is simple: a byte sequence does not say which encoding produced it. Detection is statistics on a sample, and a sample of four characters is not much evidence. We measure that with fixed-seed samples in four languages, test PHP's detector against a 40-line Python guesser, time conversion of 10 MB, and fix the chunk-boundary decoding bug that lecture 1 warned about.

Setup: Apple Silicon Mac (M1 Pro), macOS 26.6, Python 3.9.6 (standard library only), PHP 8.5.6 with mbstring and iconv, and the macOS iconv command. The popular Python library chardet (and its relatives) is not installed on this machine and nothing was installed for this lecture, so it was not tested. Windows and Excel were not run.

Why detection can only guess

Take four short strings, encode each in its usual legacy encoding, and decode the same bytes with every candidate:

Most readings decode without any error. Windows-1252 turns nearly every byte into some character, so "it decoded" proves nothing. CP932 and CP949 both use two-byte sequences over overlapping ranges, so Korean bytes become plausible-looking Japanese and vice versa. A detector can only ask: which reading looks most like real text? That is a guess, and it gets better with more text.

One signal is very strong: UTF-8 validity. UTF-8 has strict rules about which bytes may follow which, and random legacy text rarely obeys them. Across our 10,000 legacy samples below (Japanese, Korean, Spanish and English), 0 were valid UTF-8. Check UTF-8 first, and you have already settled most real-world files. Text that is itself mojibake is the exception: the cp1252 bytes of é are exactly the UTF-8 bytes of é.

mb_detect_encoding: three traps before accuracy

PHP's detector has behaviours worth knowing before you look at any accuracy number.

Trap 1: the default list. With no candidate list, PHP uses mb_detect_order(), which here was ASCII, UTF-8. Non-strict mode then returns the "closest" candidate even when none fits:

$b = mb_convert_encoding("文字化け", "SJIS-win", "UTF-8");
var_dump(mb_detect_encoding($b));                         // string(5) "ASCII"
var_dump(mb_detect_encoding("\xff\xfe", ["UTF-8", "ASCII"]));       // string(5) "UTF-8"
var_dump(mb_detect_encoding("\xff\xfe", ["UTF-8", "ASCII"], true)); // bool(false)

Shift_JIS bytes reported as ASCII, and two bytes that can never appear in UTF-8 reported as UTF-8. Always pass an explicit list and true for strict.

Trap 2: candidate order is part of the answer. When two candidates both fit, order and PHP's internal scoring decide. Our results below change dramatically between a "Japanese first" and a "Korean and Western first" list.

Trap 3: the family, not the variant. Detection can tell you "Shift_JIS-ish", but not which table: 0x81 0x60 is U+301C (WAVE DASH) in SJIS and U+FF5E (FULLWIDTH TILDE) in SJIS-win/CP932, in both PHP and Python. That one byte pair is the famous 〜 problem in Japanese data. The same applies to EUC-KR vs CP949 and ISO-8859-1 vs Windows-1252: 0x93Hola0x94 is “Hola” in cp1252 but two invisible C1 control characters around Hola in latin-1.

A stdlib guesser, version 1 and version 2

To compare, we wrote a guesser with the Python standard library only. Version 1: try UTF-8; otherwise decode with each legacy candidate and score the result by character type (kana and hangul +2, kanji +1, printable ASCII +1, half-width katakana −1, control and private-use −5), averaged per character. It was bad at Japanese CP932 (60/200 correct on 4-character samples) because CP932 bytes decoded as CP949 produce real hangul syllables.

Version 2 changed two things after looking at its mistakes: score per byte (so a two-byte kanji does not beat two one-byte Latin letters), and give a penalty to hangul syllables outside the 2,350 common ones of KS X 1001 (the extended CP949 area), because real Korean text rarely needs them but misread Japanese bytes often land there:

import unicodedata
CANDIDATES = ["cp1252", "cp932", "euc_jp", "cp949"]   # on a tie, earlier wins

def char_score(ch):
    o = ord(ch)
    if o < 0x80: return 1.0 if (ch.isprintable() or ch in "\t\r\n") else -5.0
    if 0x3041 <= o <= 0x30FF: return 2.0                       # kana
    if 0xAC00 <= o <= 0xD7A3:                                   # hangul syllable
        b = ch.encode("cp949")
        return 2.0 if (b[0] >= 0xB0 and b[1] >= 0xA1) else -1.0   # rare outside KS X 1001
    if 0x4E00 <= o <= 0x9FFF: return 1.5                        # kanji / hanja
    if 0x3000 <= o <= 0x303F or 0xFF01 <= o <= 0xFF5E: return 1.5
    if 0x2010 <= o <= 0x2026: return 1.0                        # dashes, smart quotes
    if 0xFF61 <= o <= 0xFF9F or 0x3131 <= o <= 0x318E: return -1.0
    cat = unicodedata.category(ch)
    if cat in ("Cc", "Co", "Cn"): return -5.0
    if cat.startswith("L"): return 1.0 if 0xC0 <= o <= 0xFF else 0.0
    return 0.0

def guess(b):
    if b.startswith(b"\xef\xbb\xbf"): return "utf-8-sig"
    try:
        b.decode("utf-8"); return "utf-8"
    except UnicodeDecodeError:
        pass
    best, best_score = None, float("-inf")
    for enc in CANDIDATES:
        try: s = b.decode(enc)
        except UnicodeDecodeError: continue
        score = sum(char_score(c) for c in s) / len(b)
        if score > best_score: best, best_score = enc, score
    return best

Honest warning: version 2 was tuned by looking at failures on one set of texts. To avoid grading our own homework, every number below comes from a second, held-out set of texts (everyday paragraphs about phones, cooking and the weather) that the tuning never saw. Both sets were written by us, so real data will behave differently.

Measuring accuracy: short vs long text

Method: for each language, take random substrings of 4, 8, 16, 64 and 256 characters (random.seed(2026)), skip pure-ASCII samples (every guess is right for those), skip samples the encoding cannot represent (㈱ and ① have no EUC-JP code), encode, and ask each detector. A guess counts as correct if decoding with it gives back exactly the original text. N = 200 per cell, 9,000 samples in all. PHP list A was UTF-8, SJIS-win, EUC-JP, UHC, Windows-1252; list B was UTF-8, UHC, Windows-1252, SJIS-win, EUC-JP (UHC is PHP's name for CP949).

Correct out of 200, 4-character samples:

16-character samples:

At 256 characters every detector was 200/200 except two: v1 and PHP list A both got English CP1252 right only 179 times. All 4,000 UTF-8 samples were right for every detector.

What the failures looked like in English: with either list, PHP read It’s as It痴, ’s g as 痴 g and s “a as s 殿. The smart quote byte 0x92 or 0x93 followed by a letter is a valid Shift_JIS kanji, so the Japanese candidate scored at least as well. In Spanish, léfo became l馭o for the same reason. In Korean, list A turned CP949 into EUC-JP kanji; in Japanese, list B turned EUC-JP into UHC hangul. Our own v2 failed in the opposite direction: all 15 of its misses on 4-character Japanese CP932 were read as cp1252 (天気予報 became “V‹C—\•ñ), because the hiragana lead byte 0x82 is the low quote ‚ in cp1252, and v2 gives smart quotes a bonus. A rule added for English hurt Japanese. Short text plus a candidate list that covers several regions is the worst case, and a person's name, a product code or a CSV cell is exactly that.

Common mistake: detecting on a prefix of a big file

The mistake. Detection on 10 MB is slow (our v2 took 894 ms on 10 MiB of CP932), so you sensibly look at the first 64 KiB: guess(data[:65536]).

What actually happens. The cut lands in the middle of a character two times out of three. Our strict guess then sees invalid UTF-8 and invalid legacy text:

big_cp949_utf8.txt  [:65535] None    [:65536] None    [:65537] utf-8
big_cp932.txt       [:65535] None    [:65536] cp932   [:65537] None
big_cp949.txt       [:65535] cp1252  [:65536] cp949   [:65537] cp1252

A UTF-8 file was reported as "unknown", and a Korean CP949 file as Windows-1252: worse than no answer, because the import then succeeds and writes mojibake.

Fix. Decode the prefix with an incremental decoder and final=False, which treats an unfinished character at the very end as "more bytes coming" instead of an error:

import codecs
def decode_prefix(b, enc):
    return codecs.getincrementaldecoder(enc)().decode(b, final=False)

Using decode_prefix in place of b.decode inside guess, all 15 prefix cuts gave the right answer, and guessing on 64 KiB took 6 ms instead of 894.

The streaming bug: a character split across chunks

The same split happens when you decode a stream chunk by chunk. A 1 MB Japanese file (UTF-8, and the same text in CP932) read in 4,096-byte chunks:

with open(path, "rb") as f:
    while chunk := f.read(4096):
        chunk.decode(enc)          # the bug: each chunk decoded on its own

strict: chunk #1 failed: 'utf-8' codec can't decode bytes in position 4094-4095: unexpected end of data
replace: U+FFFD count = 412
strict: chunk #3 failed: 'cp932' codec can't decode byte 0x83 in position 4095: incomplete multibyte sequence
replace: U+FFFD count = 86

The CP932 case is worse than the count suggests. When a lead byte is replaced, its trail byte starts the next chunk and is read as a character of its own. 基幹システム came out as 幹シス�eムか: the trail byte 0x65 of テ became the letter e. The output was 85 characters longer than the original and contained 59 half-width katakana that were not in the file, with no error raised.

Fix in Python: an incremental decoder keeps the unfinished bytes for the next call.

import codecs
dec = codecs.getincrementaldecoder(enc)()
with open(path, "rb") as f:
    parts = [dec.decode(c) for c in iter(lambda: f.read(4096), b"")]
parts.append(dec.decode(b"", final=True))     # raises if the file ends mid-character

Result for both files: U+FFFD count = 0, identical to decoding the whole file. Simply opening the file in text mode (open(path, encoding="cp932")) uses the same machinery. In PHP, mb_convert_encoding on each fread chunk added 412 (UTF-8) and 87 (CP932) ? characters; appending the stream filter convert.iconv.CP932/UTF-8 to the handle produced output identical to whole-file conversion. JavaScript in jsc has no TextDecoder, so its {stream: true} option was not run here.

Conversion speed on 10 MiB

Each file was 10 MiB of real text in the legacy encoding, converted to UTF-8 in memory; median of 5 runs. The iconv command row includes starting the process and reading the file (cat of the same file took 4 ms).

All four produced byte-identical UTF-8 (cmp and === checks). Python's codecs and mbstring ran at roughly 280 to 1,000 MB/s; the macOS libiconv was about ten times slower. That is still only a quarter of a second for 10 MiB, so for a nightly import any of them is fine; in a request handler, prefer mbstring or Python codecs. Two more numbers: validating 15.6 MB of UTF-8 took 10.3 ms in Python (decode), and PHP's mb_check_encoding took 3.5 to 4.1 ms on a fresh string but about 0.0002 ms on a second call to the same string, because PHP caches the "valid UTF-8" flag on the string. Benchmark the first call, not the median.

A pipeline that does not rely on luck

1. Get the encoding declared. HTTP Content-Type: text/csv; charset=... (HTTP headers lecture), an XML or HTML declaration, a BOM, or simply a line in the contract with the partner. A declared encoding beats any detector.
2. Check for a BOM, then for valid UTF-8. This settled 4,000 out of 4,000 UTF-8 samples and rejected all 10,000 legacy ones.
3. Use the source's known encoding. A given legacy system always writes the same encoding. Keep a table "source → encoding" instead of guessing per file.
4. Decode strictly. errors="strict" in Python, mb_check_encoding before mb_convert_encoding in PHP. A failed import is cheaper than silent mojibake.
5. Guess only as a last resort, on at least a few hundred bytes, with a candidate list limited to the regions you actually receive, and send the result to a human or a quarantine folder.
6. Stream with incremental decoders, never per-chunk decode.
7. Normalize and store UTF-8: NFC (lecture 3), utf8mb4 in MySQL (lecture 5), and write CSV for Excel users as in lecture 4.

Honest limits / version notes

• Our samples come from two small sets of paragraphs we wrote; real data (addresses, product codes, mixed languages) will score differently. Treat the tables as a demonstration of the short-text effect, not as a benchmark of PHP.
• mb_detect_encoding's scoring was reworked in PHP 8.1; results on PHP 7 or 8.0 can differ. Only 8.5.6 was run.
• chardet, charset-normalizer, ICU's CharsetDetector and nkf were not available and were not tested. Linux and GNU iconv were not run.
• Python's euc_kr codec accepts the 8-byte KS X 1001 "make-up" sequences for syllables like 똠 (a4 d4 a4 a8 a4 c7 a4 b1), while the macOS iconv -t EUC-KR refused 똠 with Illegal byte sequence. Different tools disagree even on the same encoding name.
• Speeds are from one Apple M1 Pro; the order (mbstring and codecs fast, macOS libiconv slower) is what to take away.

Exercises

1. Why did mb_detect_encoding($b) return "ASCII" for Shift_JIS bytes, and what one change fixes it?

2. Feed the incremental decoder one byte at a time: decode "똠방각하, 뷁! 한글 깨짐 테스트".encode("cp949") with codecs.getincrementaldecoder("cp949")() called once per byte. Does it equal the one-shot decode?

Answers

1. Without a list, PHP used the detect order ASCII, UTF-8, and in non-strict mode it returns the closest candidate even if none is valid. Pass an explicit list and strict mode: mb_detect_encoding($b, ["UTF-8", "SJIS-win"], true) returned "SJIS-win".

2. Yes. It printed 똠방각하, 뷁! 한글 깨짐 테스트 True. The decoder buffers a lead byte such as 0x8c until the trail byte arrives, so even a chunk size of 1 is safe. A plain UTF-8 example shows the buffering: dec.decode(b"caf\xc3") returned 'caf' and the next call with b"\xa9!" returned 'é!'; calling decode(b"\xc3", final=True) on a fresh decoder raised unexpected end of data.

Series wrap-up

This was the last lecture. From bytes and code points (1), through mojibake (2), normalization (3), CSV and Excel (4) and string length (5), to detection here, the rule has been the same: know which encoding and which unit you are dealing with, and make it explicit at every boundary.

• Previous: Text & Encoding lecture 5/6: string length in 8 languages
• Start again: lecture 1/6: characters, code points and UTF-8 bytes
• The rest of the series: mojibake and how to fix it, Unicode normalization and filenames, CSV encodings, the BOM and Excel
• Related: shell pipes and text tools, Python basics, PHP basics, non-ASCII URLs in .htaccess on LiteSpeed, and the programming learning roadmap