String Length in 8 Languages: Emoji, Graphemes, Bytes

Series: Text & Encoding lecture 5/6 (advanced). For developers who validate, truncate or store user text and have been surprised that a "10-character limit" means different things in different layers. After this lecture you will know which of four "lengths" each API returns, how to cut a string without breaking an emoji, and which count to use for a UI, a database and a protocol.

This is for you if a username like tomás👨‍💻 passed the form check in the browser and failed in the API, if a push notification preview ended in a ?, or if a "Read more…" cut turned a family emoji into a couple. The short answer: there is no single string length. There are UTF-8 bytes, UTF-16 code units, code points and grapheme clusters, and every language picks one as its default. Below, the same 14 strings go through Python, JavaScript, Swift, Dart, Java, C#, PHP and SQLite, with every number taken from a real run.

Setup for every output on this page: Apple Silicon Mac (M1 Pro), macOS 26.6, Python 3.9.6 (Unicode 13.0 database), JavaScriptCore jsc from macOS, Swift 6.4.2, Dart 3.11.5, OpenJDK 26, .NET 10.0.403 (C#), PHP 8.5.6 with intl (ICU 78.3) and SQLite 3.51.0. Node.js, browsers, MySQL and Windows were not run.

The four "lengths" of one string

Lecture 1 separated characters, code points and bytes. To compare languages we need one more layer, because two languages store text in UTF-16:

• UTF-8 bytes: what goes on disk and over the network. PHP strlen, Swift utf8.count, SQLite octet_length().
• UTF-16 code units: 16-bit units. Characters above U+FFFF (most emoji, rare CJK) need two of them, called a surrogate pair. JavaScript .length, Java length(), C# Length, Dart .length.
• Code points: the numbered Unicode characters (U+00E9). Python len(), PHP mb_strlen, SQLite length().
• Grapheme clusters: what a person calls "one character". A flag is two code points, 👍🏽 is a thumb plus a skin tone modifier, and 👨‍👩‍👧 is three people glued by two invisible ZERO WIDTH JOINER (U+200D) characters. Unicode's rule for this is called an extended grapheme cluster (UAX #29).

The big table: 14 strings, measured

Every string was written with escapes ("é", "\u{1F468}‍…") in each language so that no editor could normalize it. Within each unit, every API agreed exactly; the next table lists which API fed which column.

And the strings English-speaking users actually type into sign-up forms:

So tomás👨‍💻 is 6 characters to the user, 8 to Python, 10 to JavaScript and 17 to the database row. A "max 8 characters" rule written once in JS and once in Python will accept it in one place and reject it in the other.

Which API returns which unit

Things the table hides that we checked directly:

• Intl.Segmenter exists in this jsc (typeof Intl.Segmenter printed function), but TextEncoder does not (undefined), so byte counts in jsc need a hand-written loop.
• Python 3.9 has no grapheme support: re.compile(r'\X') fails with re.error: bad escape \X at position 0. The third-party regex module supports \X, but nothing was installed for this lecture.
• Swift compares by grapheme too: "\u{00E9}" == "e\u{0301}" printed true in Swift, false in Dart and False in C# with ==. C#'s string.Compare(..., StringComparison.InvariantCulture) returned 0 (equal).
• PHP's mb_strwidth is a fourth opinion: it gave 2 for 한 and 4 for 文字化け, because East Asian wide characters count as two columns. It is meant for terminal alignment, not limits.

Truncation bugs, with real outputs

Counting wrong is annoying; cutting wrong corrupts data. Here is what each language does when you cut by its default unit.

JavaScript: a lone surrogate. "OK👍".slice(0, 3) keeps only the first half of the surrogate pair:

"OK\ud83d" 3 false OK�
URIError: String contained an illegal UTF-16 sequence.

The values are JSON.stringify(cut), cut.length, cut.isWellFormed() and cut.toWellFormed(). The broken string is still a valid JS string, so nothing fails until it leaves the program: encodeURIComponent threw the URIError above. That is the "search query works for everyone except the user with the emoji" bug.

JavaScript again: a family loses a child. Cutting the family emoji at 5 code units gives "👨‍👩": a valid string, no error, different meaning. Code point slicing ([...s].slice(0, 3)) produces the same two people. Only grapheme slicing is safe:

const seg = new Intl.Segmenter("en", {granularity: "grapheme"});
function truncate(str, n) {
  let out = "", i = 0;
  for (const {segment} of seg.segment(str)) {
    if (i++ === n) return out + "…";
    out += segment;
  }
  return out;
}
print(truncate("Hola \u{1F1F2}\u{1F1FD}!", 6));   // Hola 🇲🇽…

Java and C#: the half emoji becomes ? or � on the way out. Java's "OK👍".substring(0, 3) encoded to UTF-8 as 4f 4b 3f, so the database receives a literal question mark. C#'s Substring(0, 3) encoded as 4F4BEFBFBD, which is U+FFFD (�). With a strict encoder (new UTF8Encoding(false, true)) .NET refuses instead:

EncoderFallbackException: Unable to translate Unicode character \\uD83D at index 2 to specified code page.

The fix in Java is to cut at a code point boundary (s.offsetByCodePoints(0, n)) or, better, a BreakIterator boundary; in C#, new StringInfo(s).SubstringByTextElements(0, 3) returned the whole 가족👨‍👩‍👧.

Python and PHP: cutting bytes. Byte limits are where this really bites, because protocols and columns measure bytes. In Python:

>>> "文字化け".encode()[:10].decode()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe3 in position 9: unexpected end of data

errors="ignore" drops the partial character and gives 文字化, which looks like the fix. It is not: "OK👍🏽" cut to 7 bytes became OK👍. The skin tone modifier vanished silently, and the user's emoji changed. In PHP, substr("한글 깨짐", 0, 4) returned the bytes ed959cea, and json_encode of that returned false with Malformed UTF-8 characters, possibly incorrectly encoded, which is how a whole API response becomes empty. mb_substr cuts by code point, so mb_substr("가족👨‍👩‍👧", 0, 4) ended with a dangling ZWJ (…f09f91a8e2808d).

PHP's intl extension has the right tool: grapheme_extract with GRAPHEME_EXTR_MAXBYTES returns the longest run of whole graphemes that fits in N bytes:

$s = "가족👨‍👩‍👧!";             // 6 + 18 + 1 = 25 bytes
foreach ([23, 24] as $max) {
    $next = 0;
    var_dump(grapheme_extract($s, $max, GRAPHEME_EXTR_MAXBYTES, 0, $next), $next);
}
// string(6) "가족"      int(6)    (the family needs 24 bytes)
// string(24) "가족👨‍👩‍👧" int(24)

Common mistake: one "max length" checked in three layers

The mistake. The product rule says "display names up to 10 characters". The web form checks value.length <= 10 (UTF-16), the API checks len(name) <= 10 (code points), and the column is VARCHAR(10).

What actually happens. tomás👨‍💻 is 10 in JS, so the form accepts it; it is 8 in Python, so the API accepts it too; then a user types five flags 🇲🇽🇯🇵🇰🇷🇪🇸🇺🇸, which is 5 graphemes, 10 code points and 20 UTF-16 units. The form says "too long" for something that looks like five characters. Meanwhile, a name that the API cuts with name[:10] can end in the middle of a ZWJ sequence.

Fix. Decide per layer, on purpose:

Then enforce the strictest one on the server and show the user-facing one in the UI. If you build Flutter apps: Flutter's documentation says TextField.maxLength counts grapheme clusters, while Android's classic maxLength filter counts UTF-16 chars (both described from documentation, not run here).

Database columns: SQLite measured, MySQL described

SQLite treats the declared length as decoration:

sqlite> create table u(name varchar(5));
sqlite> insert into u values ('abcdefghijklmnop');
sqlite> select name, length(name), octet_length(name) from u;
abcdefghijklmnop|16|16
👨‍👩‍👧|5|18

If you need a limit, write it yourself, and note that length() counts code points: with check (length(name) <= 4), 文字化け was accepted and the family emoji was rejected with CHECK constraint failed: length(name) <= 4 (19), although it is one character on screen. substr(fam, 1, 2) returned the bytes F09F91A8E2808D: a man plus a dangling joiner. SQLite also does not normalize: char(0xe9) = 'e' || char(0x301) returned 0, and the built-in upper('año') returned AñO because it only folds ASCII.

MySQL (not run here; from the MySQL reference manual): VARCHAR(n) counts characters (code points), not bytes. The charset called utf8 is an alias of utf8mb3, which stores at most 3 bytes per character, so any 4-byte character (every emoji, and 𠮷) is rejected in strict mode with error 1366 "Incorrect string value". Use utf8mb4, which is the default since MySQL 8.0. Byte limits still exist below the character limit: a row is limited to 65,535 bytes, and index key prefixes have a byte limit (767 or 3,072 bytes depending on the row format), so VARCHAR(255) in utf8mb4 can need up to 1,020 bytes in an index. The SQL basics lecture covers the queries themselves.

Measurement: what counting graphemes costs

Grapheme counting is not free: it must walk the string and apply the UAX #29 rules. We repeated "文字化けcafé한글👍🏽 " 100,000 times (1,200,000 graphemes, 1,300,000 code points, 1,500,000 UTF-16 units, 3,200,000 bytes) and took the median of 5 runs:

Two lessons. Stored lengths (.length, strlen, Python len) are O(1). Grapheme counts are O(n): about 70 ns per grapheme in jsc's Intl.Segmenter, 21 ns in PHP's ICU-based grapheme_strlen. That is nothing for a 30-character name and real money in a loop over a 10 MB log. Count graphemes at the edges (validation, truncation for display), not in hot loops.

Honest limits / version notes

• Intl.Segmenter is in ECMA-402 (the 2022 edition); we only checked that this macOS jsc has it. isWellFormed/toWellFormed are ES2024 and also exist here.
• Java's BreakIterator follows extended grapheme clusters since JDK 20 according to its release notes; checked on 26 only. The \X regex has existed since Java 9.
• .NET's StringInfo switched to extended grapheme clusters in .NET 5 (a documented breaking change); run on .NET 10.
• grapheme_strlen depends on the ICU version bundled with PHP (78.3 here). Newer emoji can segment differently with an older ICU; Python 3.9's unicodedata is Unicode 13.0, older than ICU 78.
• Dart's grapheme support is the characters package (used by Flutter); not run here because nothing could be installed.
• MySQL, Node.js, browsers and Android were not run; their behaviour above is from their documentation.

Exercises

1. Predict, then check: the Korean flag 🇰🇷 in JS .length, Python len, Swift count and PHP strlen.

2. Write a PHP function cut_for_column(string $s, int $maxBytes): string that never splits a grapheme, and test it on "가족👨‍👩‍👧!" with 23 and 24.

Answers

1. Measured: 4, 2, 1, 8. Two regional indicator symbols (U+1F1F0, U+1F1F7), each one 4-byte, 2-unit code point, forming a single grapheme.

2. function cut_for_column(string $s, int $maxBytes): string { $next = 0; return grapheme_extract($s, $maxBytes, GRAPHEME_EXTR_MAXBYTES, 0, $next) ?: ""; } returned 가족 (6 bytes) for 23 and the 24-byte 가족👨‍👩‍👧 for 24, as shown above. The ?: "" covers the false that grapheme_extract returns when the input is not valid UTF-8.

Next

The last lecture asks the question every legacy import ends with: the file has no declared encoding, so can a program guess it? We measure how often it guesses right, how fast conversion is, and fix the chunked-reading bug from lecture 1.

• Previous: Text & Encoding lecture 4/6: CSV encodings, the BOM and Excel
• Next: Text & Encoding lecture 6/6: encoding detection is guesswork
• Earlier in the series: characters, code points and UTF-8 bytes, mojibake and how to fix it, Unicode normalization and filenames
• Related: JavaScript basics: types and equality, PHP basics: strings, C strings and buffer overflows, rendering CJK text into OG images, and the programming learning roadmap