Common Crawl

advancedwidespread

Common Crawl is a non-profit organization and its free, public web archive containing copies of billions of websites, serving as the foundation for training data for many AI models.

In plain terms

Imagine someone regularly touring the public parts of the internet, photocopying every page they come across, and putting the bundles in a warehouse open to anyone. This is what Common Crawl does: it is a non-profit organization that crawls the web automatically and publishes the results for free.

Previously, such large web crawls were typically available only to major search companies, because crawling the web was expensive and complex. Thanks to Common Crawl, a university researcher, a small startup, or a journalist can also work with web data of billions of pages.

It is important from an AI perspective because large language models learn from vast amounts of text, much of which often comes from here. So if you ever wrote a public blog post or forum comment, it is quite possible it was included in the archive. However, it cannot be certain, as Common Crawl only crawls a part of the web.

This is not a search engine: you cannot browse comfortably or ask questions. It is a raw dataset that must be processed by programs.

How it works

An automated program, a web crawler called CCBot, goes through publicly accessible websites link by link.
The robot respects the website's robots.txt file, a small 'house rule' in which the site owner can state what must not be downloaded.
It saves downloaded pages in a standard archive format and creates separate versions for the extracted text and links.
Collections of several billion pages each, produced roughly monthly, are uploaded to cloud storage where anyone can download them for free.
Users, such as AI developers, filter and clean the material (removing garbage, duplicates, unwanted languages) and only then use it for model training or research.

An everyday example

Years ago you wrote a recipe on your own blog and since then forgot about it. If CCBot reached your page back then, the text may still be in the archive and could have entered the training data of a language model. There it is just a drop in the ocean among billions of pages, but it got there without anyone asking you.

Who is affected

everyoneWhat you published publicly on the web may have been archived and included in the training data of AI models.

developerFree, massive raw material for training models, building search engines, and web research that you do not need to collect with your own bot. → Code

entrepreneur, executiveYou must decide whether to allow the CCBot robot on your company's website or block it in the robots.txt file.

lawyer, citizenA central issue in copyright and data privacy debates is whether it is permissible to train AI on such collected content. → Law

A common misconception

MythCommon Crawl contains the entire internet, and anything within it can be freely used for any purpose.

In realityIt only samples a part of the public web, and the fact that the archive is freely downloadable does not remove the copyright protection of the pages. The person using the data is responsible for the legality of its use.

Limits and risks

The quality is mixed: alongside good articles, there is a vast amount of spam, ads, duplicate, and erroneous content, making it poor training material without filtering.
English-language content is heavily dominant, while the share of Hungarian and other smaller languages is small, which is also reflected in the language skills of models trained on it.
If you block the robot today, it only affects future collection: previously saved copies may remain in old archives and in models trained from them.
Publishers criticized it because text from articles behind paywalls was included in the archive, meaning the boundary of 'public content only' is disputed in practice.

Similar, but not the same

Internet Archive (Wayback Machine) – The Internet Archive is for people to view old versions of a page, while Common Crawl is a raw dataset intended for machine processing.
Search Engine – A search engine returns results for queries from the crawled pages, while Common Crawl only publishes the raw copies, with no search interface.
web scraping – Web scraping is the method by which anyone can collect data from the web, while Common Crawl is a specific organization's ready-made, published collection.

Related concepts

Broader

Training data, Dataset, open data

Narrower

CCBot

On the edge

web scraping, Webrobot, robots.txt, Large language model, Copyright and AI

A brief history

The Common Crawl foundation was established by American entrepreneur Gil Elbaz in 2007, with the goal that a copy of the web should not be the privilege of major search companies only. It published its first public archive in 2011. It became widely known in the early 2020s, when the success of large language models drew attention to the fact that a significant part of their training material comes from there.

Where things stand

At the end of 2026, Common Crawl is one of the fundamental, widely used sources of raw material for AI research and model training. Meanwhile, more and more publishers are blocking its crawler, and copyright disputes have reached the organization too.

Status: 2026 November

Also known as

Common Crawl · CommonCrawl · Common Crawl Foundation · Common Crawl Foundation · Common Crawl archive · Common Crawl dataset · CCBot

Sources

This entry was machine-edited and has undergone machine expert review; the “In the news” section updates automatically from Agent Age stories.