indexing

basicwidespread

Indexing is the process where a search engine processes and stores the content of pages in an organized way so it can find them again in seconds.

In plain terms

Imagine the index at the end of a thick technical book: you do not need to flip through the whole book; it is enough to check which page a word appears on. Search engines do the same with the internet: they prepare the "index" in advance, and when you search, they look it up there, not on the live web.

This is why Google can respond in a fraction of a second. It has already done most of the work well before your question: it crawled the pages, interpreted what they are about, and stored the information in its own database, the index.

What is not in the index does not exist for the search engine. No matter how great a website is, if the search engine has not indexed it, no one will find it in the results.

AI-based search engines and agents also rely on indexes. They partly use the indexes of traditional search engines and partly build their own index of documents that can be searched by meaning.

How it works

Crawling: the search engine's automated program (bot) discovers and downloads pages by moving from link to link.
Processing: the system reads the text, headings, and image descriptions, and tries to understand what the page is about and in what language it was written.
Filtering: the search engine decides whether to store the page. Copied, empty, low-quality, or owner-blocked pages may be excluded.
Storage: content enters the index, typically by recording which pages contain each word or meaning.
Update: the robot returns periodically, and if the page changes or disappears, the index is adjusted.

An everyday example

You launch your small business's new website, search for its name on Google, and find nothing. Nothing is broken: the search engine simply has not indexed the page yet, which can take from a few days to several weeks.

Who is affected

marketerThe first requirement of search engine optimization is that pages are included in the index at all; otherwise, they cannot appear for any keyword. → Marketing

entrepreneur, executiveIf your company's website or webshop is not indexed, customers will not find you through search engines, regardless of how much you spent on it.

developerThe technical structure and settings of a website determine whether search crawlers can navigate and index its content. → Code

everyoneWhat you see in the search engine comes from the index, and AI assistants' answers often build on this, meaning the index largely shapes what you find and what you do not.

A common misconception

MythIf a page goes online, Google finds it immediately and automatically, and it appears in the search results.

In realityIndexing takes time and is not guaranteed: the search engine decides what to store. Even an indexed page may not appear in the right place, as ranking is a separate step.

Limits and risks

The index always reflects a previous state, so search results may include outdated or already deleted content.
Search engines do not index everything: password-protected and blocked pages are excluded, and those deemed low quality may also be left out.
If you accidentally make a page indexable, personal or internal company data can also become searchable. Permanent removal from search results takes time, and copies already saved by others cannot be recalled.
Being indexed alone does not bring visitors, as search ranking depends on content quality and many other factors.

Similar, but not the same

mapping – Crawling is the discovery and downloading of pages, while indexing is the subsequent processing and storage.
ranking – Indexing decides whether a page enters the search engine's database, while ranking determines at what position it appears for a given search.
caching – A cache is a saved copy of a page for faster loading, while an index is a searchable, organized directory of content.

Related concepts

Broader

search engine, information retrieval, search engine optimization

Narrower

inverted index, vector indexing, mobile-first indexing

On the edge

mapping, ranking, Search bot, sitemap, RAG

A brief history

The idea of the book index comes from books, and computer search took it over. Archie, considered the internet's first search tool, was already building an index in 1990, at that time from lists of downloadable files. Search engines launched in the mid-1990s, such as AltaVista, made indexing the full text of web pages widely available. Google, launched in 1998, turned this into a dominant, everyday service.

Where things stand

Indexing is a fundamental operation of every search engine, a technology refined over decades. At the end of 2026, what changes is that AI search engines and agents also rely on indexes, often supplementing them with live page retrieval. Websites therefore now have to be "readable" for them as well.

Status: 2026 November

Also known as

indexing · indexing · indexing · search engine indexing · web indexing · search index · Google indexing · getting indexed

Sources

This entry was machine-edited and has undergone machine expert review; the “In the news” section updates automatically from Agent Age stories.