Character Encoding Explained:
How Computers Store Every Character
From ASCII's 128 characters to UTF-8's 1.1 million — a practical, visual guide to how text actually works inside computers, files, databases, and the web. With byte-level examples you can verify with our tools.
Why Character Encoding Exists
Computers don't understand letters. They understand numbers — specifically, binary numbers. When you type the letter A on your keyboard, your computer doesn't store a picture of the letter A. It stores the number 65 — specifically, the binary byte 01000001. When your screen needs to display that character, the system looks up "what character does 65 map to?" and draws an A.
The system that defines which number maps to which character is called a character encoding. Without it, the number 65 is just a number — it could mean anything. The encoding is the dictionary that gives it meaning.
This sounds simple until you realize there are thousands of characters across the world's writing systems — Latin, Arabic, Chinese, Devanagari, Korean, Japanese, emoji — and they all need unique numbers. The history of character encoding is essentially the story of expanding from 128 characters to 1.1 million, without breaking everything that came before.
01 ASCII: Where It All Started (1963)
ASCII (American Standard Code for Information Interchange) was published in 1963 and became the foundation of text encoding. It uses 7 bits per character, which gives it exactly 128 possible values (0–127). Those 128 slots are split into two groups:
33 non-printable characters — tab, newline, carriage return, bell, escape, null, and others used for controlling terminals and printers. Most were designed for teleprinter machines from the 1960s and are rarely used directly today, though \n (newline, decimal 10) and \t (tab, decimal 9) are still everywhere.
95 visible characters — uppercase A–Z (65–90), lowercase a–z (97–122), digits 0–9 (48–57), punctuation, and the space character (32). This is the set most people think of when they hear "ASCII." See all 128 values on our ASCII Table.
Byte-level example: "OK" in ASCII
Verify this yourself: paste 01001111 01001011 into our binary converter.
The Encoding Chaos: 1980s–2000s
With ASCII only covering English, every region built its own encoding to handle their language. This created a patchwork of incompatible standards — all using the same byte range (128–255) to mean completely different characters depending on which encoding you were using.
Western Europe. Added accented characters: é, ñ, ü, ö, ø. Byte 0xE9 = é. Used for HTML before UTF-8 became standard.
Microsoft's extended Latin-1. Replaced control characters 128–159 with printable ones like curly quotes (" "), em-dash (—), and the Euro sign (€). Still the default in many Windows-origin files.
Japanese. Encodes kanji, hiragana, and katakana. Uses variable-width bytes (1–2 bytes per character). Still found in legacy Japanese systems and some video games.
Simplified Chinese. Covers over 20,000 Chinese characters. Still used in legacy Chinese databases and government systems.
Russian Cyrillic. Byte 0xC1 = А (Cyrillic A). Widely used in Russian Unix/Linux systems through the early 2000s.
Arabic. Covers the Arabic script used across the Middle East. Each character takes one byte, limiting it to basic letter forms without contextual shaping.
0xC1 means "Á" in Latin-1, "А" (Cyrillic A) in KOI8-R, and part of a kanji character in Shift-JIS. Without knowing the encoding, the same bytes produce entirely different text. This is why you'd open a file and see garbled characters (mojibake) — your system decoded the bytes with the wrong dictionary.
02 Unicode: One Number for Every Character
Unicode solved the chaos by assigning a unique code point — written as U+XXXX — to every character in every writing system. Instead of competing encoding tables where the same byte means different things, Unicode creates a single, universal registry. As of Unicode 16.0 (September 2024), it defines 154,998 characters across 168 scripts.
Crucially, Unicode is not an encoding by itself — it's the character catalogue. It assigns the numbers. How those numbers are stored as bytes in a file or in memory is the job of an encoding format: UTF-8, UTF-16, or UTF-32. Think of Unicode as the phone book and UTF-8 as the phone network that delivers the calls.
Unicode organizes characters into 17 "planes" of 65,536 code points each. Plane 0 (U+0000 to U+FFFF) is the Basic Multilingual Plane (BMP) and covers the vast majority of characters you'll encounter. Emoji and rare historic scripts live in the supplementary planes (U+10000 and above), which require 4-byte UTF-8 encoding.
UTF-8: The Encoding That Won the Internet
UTF-8 (8-bit Unicode Transformation Format) is a variable-width encoding that represents every Unicode code point using 1 to 4 bytes. Designed in 1992 by Ken Thompson and Rob Pike at Bell Labs, it has become the dominant encoding of the modern web — used by over 98% of all websites in 2026.
Its genius is backward compatibility with ASCII. Any valid ASCII text (bytes 0–127) is also valid UTF-8, byte-for-byte identical. This meant the billions of existing ASCII files, databases, and protocols didn't need to change when the world migrated to UTF-8 — they were already valid.
- 100% backward-compatible with ASCII
- Compact for English and Latin scripts (1 byte each)
- Covers every character in every script (1.1M+ code points)
- No null bytes in text — safe for C-style string processing
- Self-synchronizing (you can jump into the middle of a byte stream and find the next character boundary)
- CJK characters use 3 bytes each (vs. 2 in UTF-16) — larger file size for Chinese/Japanese/Korean-heavy text
- Variable-width means "character count ≠ byte count" —
strlen()and string indexing can be confusing - Emoji use 4 bytes — a message that's 50% emoji uses significantly more space than one that's all ASCII
03 How UTF-8 Encodes Characters: The Byte Structure
UTF-8 uses a clever prefix system where the first few bits of the first byte tell the decoder how many total bytes make up this character. This makes it self-synchronizing — you can jump into the middle of a byte stream and find the next character boundary without backtracking.
| Bytes | Code Point Range | Byte Pattern | Bits for Data | Example |
|---|---|---|---|---|
| 1 | U+0000 – U+007F | 0xxxxxxx | 7 | A → 41 |
| 2 | U+0080 – U+07FF | 110xxxxx 10xxxxxx | 11 | é → C3 A9 |
| 3 | U+0800 – U+FFFF | 1110xxxx 10xxxxxx 10xxxxxx | 16 | 世 → E4 B8 96 |
| 4 | U+10000 – U+10FFFF | 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx | 21 | 😊 → F0 9F 98 8A |
The leading 0, 110, 1110, or 11110 prefix on the first byte tells the decoder exactly how many bytes to read. Every continuation byte starts with 10. This means if you land on a 10xxxxxx byte, you know you're in the middle of a character and can skip forward to the next non-10 byte to find the start of the next one.
Worked example: 😊 in UTF-8 (U+1F60A)
Step 1: Code point 1F60A in binary = 0 0001 1111 0110 0000 1010 (21 bits)
Step 2: Needs 4 bytes (code point > U+FFFF). Template: 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
Step 3: Fill in the x's: 11110000 10011111 10011000 10001010
Step 4: Convert to hex: F0 9F 98 8A
Verify: paste F0 9F 98 8A into our Hex to Text converter — you'll see 😊.
ASCII vs UTF-8 vs UTF-16 vs UTF-32: Side-by-Side
The four encoding formats you're most likely to encounter, compared on the metrics that matter.
| Feature | ASCII | UTF-8 | UTF-16 | UTF-32 |
|---|---|---|---|---|
| Characters covered | 128 | 1,112,064 | 1,112,064 | 1,112,064 |
| Bytes per English letter | 1 | 1 | 2 | 4 |
| Bytes per CJK character | N/A | 3 | 2 | 4 |
| Bytes per emoji | N/A | 4 | 4 | 4 |
| Fixed-width? | Yes (1 byte) | No (1–4) | No (2 or 4) | Yes (4 bytes) |
| ASCII-compatible? | Yes (is ASCII) | Yes | No | No |
| Endianness issues? | No | No | Yes (LE/BE) | Yes (LE/BE) |
| Web usage (2026) | Legacy | 98.3% | <0.01% | <0.001% |
| Primary use in 2026 | Legacy protocols | Web, APIs, files, databases | Windows internals, Java, JS | Internal processing only |
Mojibake: When Encoding Goes Wrong
Mojibake (文字化け, Japanese for "character transformation") is the garbled text you see when bytes are decoded with the wrong encoding. It's one of the most common bugs in software that handles international text, and understanding encoding is the only real fix.
The word "café" in UTF-8 is 5 bytes: 63 61 66 C3 A9. The last two bytes (C3 A9) are the UTF-8 encoding of é (U+00E9). But if you decode those same bytes as Latin-1, you get two separate characters: à (C3) and © (A9). The result: café instead of café.
Going the other direction: the byte E9 is "é" in Latin-1, but it's an illegal byte in UTF-8 (it starts with 1110, indicating a 3-byte sequence, but no continuation bytes follow). The UTF-8 decoder either replaces it with the replacement character (�) or throws an error.
<meta charset="UTF-8">. In databases, set the connection charset to match the column charset. In APIs, set the Content-Type: application/json; charset=utf-8 header. In Python, open files with open('file.txt', encoding='utf-8'). The single most effective rule: use UTF-8 everywhere, declare it explicitly, and never assume the default is correct.
The BOM: Those Mysterious Bytes at the Start of Files
You may have encountered the BOM (Byte Order Mark) — the bytes EF BB BF at the very beginning of a UTF-8 file. In a hex editor they're invisible, but they cause real problems: PHP scripts that output a blank line before any content, CSV files that Excel misreads, JSON parsers that throw errors on otherwise valid files, and diff tools that flag two identical files as different.
The BOM was designed for UTF-16 and UTF-32, where it serves an essential purpose: it tells the decoder which byte order (big-endian or little-endian) the file uses. But UTF-8 has no byte-order issue — every byte has a fixed position — so the UTF-8 BOM is technically unnecessary. It's allowed by the Unicode standard but explicitly discouraged in most contexts. The W3C, IETF, and most modern tooling treat it as an optional signal that the file is UTF-8, but many parsers don't expect it and choke on it.
Which Encoding Should You Use in 2026?
The answer is almost always UTF-8. Here's the short decision tree for the few cases where it isn't.
Web pages, APIs, JSON, config files, source code → UTF-8
No exceptions. Set charset=utf-8 in your HTML meta tag and HTTP headers. Every modern browser, framework, and CDN expects it.
Databases → UTF-8 (usually utf8mb4 in MySQL)
MySQL's utf8 charset only supports 3-byte characters and silently truncates emoji. Use utf8mb4 to get full 4-byte UTF-8 support. PostgreSQL's UTF8 handles 4 bytes by default.
Windows internal APIs, Java, JavaScript strings → UTF-16 (not your choice)
These environments use UTF-16 internally. You don't choose this — it's what the runtime uses. When serializing to files or network, convert to UTF-8. JavaScript's TextEncoder does this natively.
Legacy system that requires a specific encoding → match what it expects
If you're interfacing with a system that requires Shift-JIS, Windows-1252, or Latin-1, encode your output in that format at the boundary. Keep your internal processing in UTF-8 and convert only at the edges.
See Encoding in Action With Our Tools
Every concept on this page maps to a tool you can experiment with right now — all running in your browser with zero data sent to our servers.
Frequently Asked Questions
What is character encoding?
What is the difference between ASCII and UTF-8?
What is the difference between Unicode and UTF-8?
Why do I see garbled text or question marks instead of characters?
<meta charset="UTF-8">. In databases, match the connection charset to the column charset. In APIs, set the Content-Type header with charset=utf-8.