character encoding

basicwidespread

Character encoding is the rule set that defines which number sequences correspond to letters and punctuation marks in a computer; today, UTF-8 is the most widespread.

In plain terms

Computers do not recognize letters, only numbers. Character encoding is a shared code table: it specifies which number sequence represents which letter, digit, punctuation mark, or emoji. It is like Morse code, but with zeros and ones instead of dots and dashes.

Problems arise when text is saved with one code page and read back with another. In such cases, 'árvíztűrő' becomes 'Ã¡rvÃztÅ±rÅ‘', or the letter 'ő' turns into 'õ' and 'ű' into 'û'. The text is not corrupted; it was simply decoded using the wrong character set.

UTF-8 became the global standard because almost every writing system fits into a single encoding: Hungarian accented letters, Cyrillic, Chinese, and emojis. Previously, every region and writing system had its own table, and these were incompatible with each other.

How it works

Every character gets a unique number in the international Unicode registry; for example, the number for "ő" is 337.
UTF-8 converts this number to 1–4 bytes: letters of the English alphabet take 1 byte, Hungarian accented letters take 2, and emojis mostly take 4.
Ideally, the file or webpage indicates the encoding used, such as the "charset=utf-8" label in the webpage header.
The program that opens the file converts the bytes back into characters based on this; if there is no indication, it guesses, and that is when strange symbols appear.
Language models and AI agents receive text this way: bytes first become characters, and then tokens, so incorrect encoding causes incorrect reading for them as well.

An everyday example

You download a customer list from your online store in a CSV file, open it in Excel, and instead of "Kőszegi Győző" it shows "KĹ‘szegi GyĹ‘zĹ‘". The file is UTF-8 encoded, but Excel tried to read it with an older Windows Central European code page. If you select UTF-8 during import, the names are restored.

Who is affected

everyoneIf strange characters appear in place of accented letters in an email, subtitle file, or webpage, the cause is almost always an encoding mismatch.

employeeWhen moving tables, exports, and imports, poor encoding can corrupt names and addresses, requiring later correction. → Work & HR

developerYou must ensure that the database, files, website, and AI agent all use the same encoding. → Code

entrepreneur, executiveA customer name appearing incorrectly on an invoice or newsletter is awkward and complicates data reconciliation.

A common misconception

MythIf there are strange characters in the text, the file is corrupted.

In realityThe data is mostly intact; it was simply opened with the wrong code table. Selecting the correct encoding restores the original text. Real loss occurs only if the corrupted version is saved and further work is done on it.

Limits and risks

The file's encoding is often not visible, and programs guess, but the guess can be wrong.
If you save a text opened incorrectly and then convert it again, the error can double and become difficult to reverse.
Legacy systems, bank and billing exports may still use older Central European code pages, so it is worth checking for ő and ű characters upon import.
In UTF-8, a character is not always one byte, so the character count and the length measured in bytes differ, which can cause surprises in length-limited fields (for example, in a database or form).

Similar, but not the same

Unicode – Unicode is a numbered list of characters, while UTF-8 is one method for storing these numbers in bytes.
encryption – Encoding is a publicly reversible transformation, while encryption renders data unreadable without a key.
font – The coding decides which character it is, while the font only determines how it looks on the screen.

Related concepts

Broader

data visualization, standard, File format

Narrower

UTF-8, ASCII, Latin-2

On the edge

Unicode, token, tokenization, CSV, byte, byte order mark

A brief history

The ASCII code table, adapted to the English alphabet, was standardized in 1963; the first version of Unicode appeared in 1991. UTF-8 was designed in 1992 by Ken Thompson and Rob Pike, and by the end of the 2000s it became the web's most common encoding.

Where things stand

At the end of 2026, the vast majority of websites use UTF-8, and it is the default in most programming languages, in Linux and macOS, and in AI tools. On Windows, some programs still work with an old code page, so problems mainly arise from files coming from legacy systems and certain spreadsheet imports.

Status: 2026 November

In the news

Appears in 233 Agent Age stories; the curve shows weekly occurrence in the news magazine.

AI coding agents still need human rework on three-quarters of tasksNov 8, 2026 · Code
AWS launches Kiro bots to coordinate AI coding agentsNov 5, 2026 · Code
Meta buys AI startup to push to the front of the coding marketNov 4, 2026 · Agent Age
OpenAI to acquire Pierre Computer to strengthen CodexNov 4, 2026 · Code
Meta acquires AI coding startup ConductorNov 4, 2026 · Code
Cognition launches SWE-2 Flash to make coding agents cheaper and fasterNov 4, 2026 · Code
Revolut switches from Claude Code to Chinese open-weight model, cutting costs by 90%Nov 3, 2026 · Code
AI coding agents are installing MCP servers that will never be patchedNov 3, 2026 · Code

Also known as

character encoding · coding · code page · character set · UTF-8 · character encoding · charset · text encoding

Sources

This entry was machine-edited and has undergone machine expert review; the “In the news” section updates automatically from Agent Age stories.