Characters, Code Points and UTF-8 Bytes, Seen in xxd

Series: Text & Encoding lecture 1/6 (beginner). For anyone who has seen don’t on a web page, or watched a username with an emoji break a form, and wants to know what a "character" really is. By the end you will read UTF-8 bytes in a hex dump, predict how many bytes a string takes, and know which "length" each function returns.

This lecture is for developers who handle user text (names, comments, CSV imports) and keep hitting the same three surprises: len("café") says 4 while the database says 5 bytes, a smart quote turns into ’, and a 👍 in a username overflows a column that "should" fit. All three come from one fact: a character, a code point and a byte are three different things, and every tool counts a different one. Every output below was produced by running the code on an Apple Silicon Mac (M1 Pro) with Python 3.9.6, PHP 8.5.6 and the stock macOS xxd.

Characters, code points and bytes are three different things

Let's name the three layers before touching any code.

• A character is what a human sees: é, ñ, 한, 👍. (Lecture 5 shows that even this is fuzzy: 👨‍👩‍👧 looks like one character but is five code points.)
• A code point is the number Unicode assigns to a character. It is written U+ plus hex: é is U+00E9, € is U+20AC, 👍 is U+1F44D. Unicode defines code points from U+0000 to U+10FFFF.
• A byte is what a file, a socket or a database column actually stores. An encoding is the rule that turns code points into bytes. UTF-8 is one such rule; UTF-16, Shift_JIS and Windows-1252 are others.

Text in memory (a Python str) is a sequence of code points. Text on disk or on the wire is bytes. Moving between the two is encoding (str → bytes) and decoding (bytes → str). Almost every encoding bug, including all the mojibake in lecture 2, is a decode done with the wrong rule.

Your first five minutes: look at the bytes with xxd

xxd prints the raw bytes of anything you pipe into it. printf '%s' avoids the newline that echo would add:

for s in A é ñ € 😀 👍; do printf '%s' "$s" | xxd; done

00000000: 41                                       A
00000000: c3a9                                     ..
00000000: c3b1                                     ..
00000000: e282 ac                                  ...
00000000: f09f 9880                                ....
00000000: f09f 918d                                ....

A is one byte, 0x41, the same value it has had in ASCII since the 1960s. é and ñ are two bytes each, € is three, and both emoji are four. The dots on the right are xxd admitting that those bytes are not printable ASCII.

Now the same thing from Python, which can show code points and bytes side by side:

import unicodedata
for s in ["hello", "café", "naïve", "don’t", "I ❤️ UTF-8", "👍", "👨‍👩‍👧"]:
    b = s.encode("utf-8")
    print(f"{s!r:14} len={len(s):2}  bytes={len(b):2}  {b.hex(' ')}")
for ch in "Aé’€👍":
    print(f"{ch} U+{ord(ch):04X} {unicodedata.name(ch)} -> {ch.encode().hex(' ')}")

'hello'        len= 5  bytes= 5  68 65 6c 6c 6f
'café'         len= 4  bytes= 5  63 61 66 c3 a9
'naïve'        len= 5  bytes= 6  6e 61 c3 af 76 65
'don’t'        len= 5  bytes= 7  64 6f 6e e2 80 99 74
'I ❤️ UTF-8'   len=10  bytes=14  49 20 e2 9d a4 ef b8 8f 20 55 54 46 2d 38
'👍'            len= 1  bytes= 4  f0 9f 91 8d
'👨‍👩‍👧' len= 5  bytes=18  f0 9f 91 a8 e2 80 8d f0 9f 91 a9 e2 80 8d f0 9f 91 a7
A U+0041 LATIN CAPITAL LETTER A -> 41
é U+00E9 LATIN SMALL LETTER E WITH ACUTE -> c3 a9
’ U+2019 RIGHT SINGLE QUOTATION MARK -> e2 80 99
€ U+20AC EURO SIGN -> e2 82 ac
👍 U+1F44D THUMBS UP SIGN -> f0 9f 91 8d

Three things to notice. The curly apostrophe in don’t is not the ASCII ' (0x27) but U+2019, three bytes e2 80 99; that is the exact byte sequence that becomes ’ in lecture 2. "I ❤️ UTF-8" has length 10 even though you see 9 symbols, because the red heart is U+2764 followed by an invisible variation selector U+FE0F (ef b8 8f). And the family emoji is five code points glued with zero-width joiners (e2 80 8d).

How UTF-8 lays out bits, from 1 to 4 bytes

UTF-8 uses the high bits of the first byte to say how long the sequence is, and marks every following byte with 10:

Here is é taken apart. Its code point U+00E9 is 11101001 in binary, eight bits, too many for one byte's seven. So it goes into the 2-byte pattern, padded to 11 bits as 00011 101001:

é U+00E9 = 11101001            utf-8: 11000011 10101001   (C3 A9)
👍 U+1F44D = 11111010001001101  utf-8: 11110000 10011111 10010001 10001101

That is the whole format. To prove it, here is a decoder in 10 lines that only uses the table above:

def decode_one(b):
    x = b[0]
    if x < 0x80: return x, 1
    if x >> 5 == 0b110:  return ((x & 0x1F) << 6) | (b[1] & 0x3F), 2
    if x >> 4 == 0b1110: return ((x & 0x0F) << 12) | ((b[1] & 0x3F) << 6) | (b[2] & 0x3F), 3
    if x >> 3 == 0b11110:
        return ((x & 0x07) << 18) | ((b[1] & 0x3F) << 12) | ((b[2] & 0x3F) << 6) | (b[3] & 0x3F), 4
    raise ValueError(f"not a lead byte: {x:#04x}")

data, i, out = "café € 👍".encode(), 0, []
while i < len(data):
    cp, n = decode_one(data[i:]); out.append(f"U+{cp:04X}"); i += n
print(out)

['U+0063', 'U+0061', 'U+0066', 'U+00E9', 'U+0020', 'U+20AC', 'U+0020', 'U+1F44D']

Eight code points from 13 bytes. A real decoder also rejects bad input (overlong forms, surrogates, truncated sequences); Python's does, as the next sections show.

A useful property falls out of the 10xxxxxx marker: from any byte you can find the start of its character by stepping back over continuation bytes. This is called self-synchronization:

data = "año 한글".encode("utf-8")
i = 7                            # somewhere in the middle of 한 (ED 95 9C)
while data[i] & 0xC0 == 0x80:    # 10xxxxxx = continuation byte, step back
    i -= 1
print(i, data[i:i+3].decode("utf-8"))   # 5 한

Which "length"? Python len() vs PHP strlen and mb_strlen

Python 3 len() on a str counts code points; on bytes it counts bytes. PHP strings are byte strings, so strlen counts bytes and the mb_ functions count code points:

<?php
foreach (["hello", "café", "naïve", "👍", "e\u{301}"] as $s) {
    printf("%-8s strlen=%d mb_strlen=%d\n", $s, strlen($s), mb_strlen($s));
}
echo strtoupper("café"), " ", mb_strtoupper("café"), "\n";

hello    strlen=5 mb_strlen=5
café    strlen=5 mb_strlen=4
naïve   strlen=6 mb_strlen=5
👍     strlen=4 mb_strlen=1
é      strlen=3 mb_strlen=2
CAFé CAFÉ

Notice that the columns no longer line up: printf("%-8s") pads to 8 bytes, so every multi-byte string gets fewer spaces. That is the same bug that misaligns fixed-width reports and CLI tables.

The last row is a trap worth seeing early: "e\u{301}" is the letter e followed by a combining acute accent. It looks exactly like é but is two code points and three bytes. That is Unicode normalization, the subject of lecture 3. And strtoupper only knows ASCII, so it left the é alone.

Even the shell has two answers. wc -c counts bytes; wc -m counts characters according to your locale:

$ printf 'café\n' > en.txt
$ wc -c en.txt            →  6 en.txt
$ wc -m en.txt            →  5 en.txt    (LANG=C.UTF-8)
$ LANG=C wc -m en.txt     →  6 en.txt

Same file, two "character counts". The rule of thumb for the whole series:

Common mistake: cutting bytes in the middle of a character

Mistake. You truncate a nickname to "10 characters" with a byte function, or you read a network buffer in fixed-size chunks and decode each chunk.

What actually happens. The cut lands inside a multi-byte sequence. In Python the decode fails loudly; with errors="replace" it fails quietly:

for b in ["café".encode()[:4], "don’t".encode()[:5], b"\xff", b"\xc0\xaf"]:
    try:
        print(repr(b), "->", b.decode("utf-8"))
    except UnicodeDecodeError as e:
        print(repr(b), "->", e)
    print("   replace:", repr(b.decode("utf-8", "replace")))

b'caf\xc3' -> 'utf-8' codec can't decode byte 0xc3 in position 3: unexpected end of data
   replace: 'caf�'
b'don\xe2\x80' -> 'utf-8' codec can't decode bytes in position 3-4: unexpected end of data
   replace: 'don�'
b'\xff' -> 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte
   replace: '�'
b'\xc0\xaf' -> 'utf-8' codec can't decode byte 0xc0 in position 0: invalid start byte
   replace: '��'

0xFF can never appear in UTF-8, and C0 AF is an "overlong" encoding of / that decoders must reject (it was once used to sneak ../ past path filters). In PHP, substr("café", 0, 4) silently returns the four bytes 63 61 66 c3, an invalid string that a browser shows as caf�.

Fix. Cut strings with code point aware functions (s[:10] in Python, mb_substr in PHP), and when you read bytes in chunks, use an incremental decoder that keeps a half-received character for the next chunk. Lecture 6 shows that bug and its fix with real chunk boundaries. If you write C, the C strings lecture covers the NUL terminator side of the same problem.

ASCII is a subset of UTF-8, and why UTF-8 won

Every ASCII byte means the same thing in UTF-8, and no byte of a multi-byte character is below 0x80. So a pure-ASCII file is already valid UTF-8, and it is also identical in most legacy encodings:

t = "Hello, World 123"
print(t.encode("ascii") == t.encode("utf-8"))                     # True
print(t.encode("ascii") == t.encode("cp932") == t.encode("cp1252"))  # True
print(t.encode("utf-16-le")[:8].hex(" "))                          # 48 00 65 00 6c 00 6c 00

UTF-16 is not ASCII-compatible: every English letter gets a 00 byte next to it, which C string functions read as "end of string". The reasons UTF-8 became the default for the web, Linux, Git, JSON and most APIs follow from this:

1. Old ASCII tools keep working. grep, the C library, HTTP headers and / in paths never see a stray 00 or 2F inside a non-ASCII character. A quick check: b"\x00" in "año 한글 👍".encode("utf-8") is False; in UTF-16 it is True.
2. No byte order. UTF-16 needs a BOM or a label to tell FF FE from FE FF ("Hi".encode("utf-16") gave b'\xff\xfeH\x00i\x00'). UTF-8 has one byte order.
3. Self-synchronizing, as shown above: one corrupted byte damages one character, not the rest of the file.
4. Byte order equals code point order. Sorting 10,000 random strings (seed 42) by their UTF-8 bytes gave exactly the same order as sorting the str values.

How big is it? I measured four short sentences, one per language this site publishes in, plus one real page from this site per language:

For CJK prose alone, UTF-8 costs 3 bytes per character against 2 in UTF-16 or the legacy code pages. But nobody ships prose alone. A built lecture page (HTML, CSS classes, URLs, code) is mostly ASCII even in Japanese: 21.5% of its characters were non-ASCII.

The English page was 32,616 bytes in UTF-8 and 65,142 in UTF-16; the Japanese one 39,229 vs 54,842; the Korean one 36,721 vs 54,264. W3Techs surveys put UTF-8 on over 98% of websites (their figure; not measured here).

Honest limits / version notes

• Python 3.9.6's unicodedata uses Unicode 13.0; the current standard is newer, so very recent emoji may print ValueError: no such name from unicodedata.name. The UTF-8 bytes do not depend on the Unicode version.
• PHP 8.5.6 with mbstring; mb_internal_encoding() returned UTF-8, the default since PHP 5.6. Results differ if a legacy php.ini sets another internal encoding.
• wc -m results depend on the locale (LANG), as shown. The macOS wc was used; GNU wc was not run here.
• UTF-8 as defined in RFC 3629 (2003) stops at 4 bytes and U+10FFFF. Older texts describe 5- and 6-byte forms; they are invalid today.
• The size table is four sentences and four pages, not a corpus. Treat it as an illustration of the trend, not a benchmark.

Exercises

1. Without running anything, predict len(s) and len(s.encode("utf-8")) for "naïve café" and for "¿Año?". Then run it.
2. A log shows the bytes E2 82 AC. Decode them by hand with the bit table. Then do the same for F0 9F 8E 89.
3. Why does "I ❤️ UTF-8" have len() 10 when you can count 9 symbols? Which bytes are the hidden part?

Answers

1. "naïve café": len 10, 12 bytes (ï and é take 2 each). "¿Año?": len 5, 7 bytes (¿ is c2 bf, ñ is c3 b1).
2. E2 = 1110 0010, so 3 bytes: 0010 + 000010 + 101100 = U+20AC, €. F0 9F 8E 89 is 4 bytes and decodes to U+1F389, 🎉.
3. The heart is U+2764 (e2 9d a4) followed by VARIATION SELECTOR-16, U+FE0F (ef b8 8f), which asks for the colorful emoji form. It counts as a code point but draws nothing on its own.

Next

The bytes are easy once you can see them. The trouble starts when one program writes bytes with one encoding and another reads them with a different one.

• Next: Text & Encoding lecture 2/6: mojibake, how it happens and how to fix it
• Later in the series: Unicode normalization, NFC/NFD and filenames, CSV encodings, the BOM and Excel, string length, emoji and graphemes in 8 languages, encoding detection and performance
• Related: Python basics: types and mutability, PHP basics: types, strings and arrays, shell pipes and text tools, and the programming learning roadmap