web scraping

basicwidespread

Web scraping is when a program automatically reads through websites, instead of a person clicking through them, and extracts the data it needs.

In plain terms

Imagine having to write down every price tag in a store in a notebook. You could do it by hand, aisle by aisle, or entrust it to a tireless assistant who scans the entire store in minutes. Web scraping is this assistant on the internet: a program that opens pages and compiles what it finds into a table.

Many everyday services are built on this: price comparison sites, flight search engines, real estate listing portals, and media monitors. In the world of artificial intelligence, it became a key term because a significant part of the training material for large language models was compiled this way, gathered from the public web.

It differs from usual data transfer in that the website owner may not know about it and may not be happy about it. The site was designed for people, while the program reads it unsolicited. That is why it attracts so much legal and ethical debate.

How it works

The program receives the list of web addresses to visit, or starting from an initial page, it follows the links itself.
It downloads the page just as your browser would, but instead of displaying it, it reads the underlying code.
In the page structure, it locates the parts important to it, such as the product name, price, or article text.
The extracted data is cleaned and saved in an organized format, such as a table or database.
It repeats this across thousands or millions of pages, often regularly, to keep the data fresh.

An everyday example

You are looking for a new washing machine, and on a price comparison site, you see the current prices of thirty online shops in a single list. These are typically not typed in by people: the data comes partly from price lists sent by the shops and partly from programs that regularly scan their websites.

Who is affected

everyoneWhat you publish publicly online can be collected by programs and may even end up in the training material of AI models.

entrepreneur, executiveThis allows you to monitor competitors' prices and the market, while others may also collect content from your website.

developerYou must write and maintain the scraper, and you must know where the legal and technical boundaries lie. → Code

lawyer, citizenThe border area between copyright and data protection, where your personal data is protected even if publicly visible. → Law

A common misconception

MythAnything publicly visible on the internet can be freely scraped and used by anyone.

In realityPublic availability does not mean free usability: copyright, database protection, the GDPR regarding personal data, and the site's terms of use can all impose restrictions. The technique itself, however, is not prohibited.

Limits and risks

Collected data can be incorrect, outdated, or incomplete: traditional collection programs do not understand the content, they only copy from the specified location, while AI-based tools can also misread it.
If the website structure changes, the crawler may break or silently start saving incorrect data.
Excessively frequent queries can overload the target site, so many operators use technical defenses to filter or block scrapers.
Mass collection of personal data may be illegal even if the data is visible to anyone.

Similar, but not the same

web crawling – Crawling discovers what pages exist by following links, while scraping extracts specific data from those pages.
API – An API is an officially offered data channel for machines provided by the service provider, whereas scraping extracts the same data from pages designed for humans, even without permission.
data mining – Data mining searches for correlations in existing data, while scraping is solely about collecting data.

Related concepts

Broader

data collection, data extraction, automation

Narrower

price monitoring

On the edge

web crawling, screen scraping, training data, robots.txt, text and data mining, API

A brief history

The first web-crawling programs appeared in 1993. The robots.txt agreement, which governs crawling and relies on voluntary compliance, was created in 1994 and became an official internet standard (RFC 9309) in 2022. The method moved to the center of public debate in the 2020s because of the training-data demands of large language models.

Where things stand

At the end of 2026, web scraping is an established technique used at mass scale, increasingly carried out by AI agents. Meanwhile, publishers and site operators are in more and more places blocking AI companies' crawlers or putting their access behind a paywall, and legal disputes continue.

Status: 2026 November

In the news

Appears in 9 Agent Age stories; the curve shows weekly occurrence in the news magazine.

New York Times and three other US papers sue Brave over AI scraping and paywall bypassNov 2, 2026 · Jog
Meta delays new business terms after publisher pushback over AI scrapingOct 19, 2026 · Jog
Apple must face lawsuit over alleged YouTube video scrapingOct 13, 2026 · Jog
UMG and Sony file new suit against Suno; Suno admits scraping YouTube audioOct 10, 2026 · Jog
EDPB guidelines on web scraping for training generative AI modelsOct 7, 2026 · Jog
Unredacted filings: Microsoft executive called AI scraping "the largest theft of labor in human history"Oct 6, 2026 · Jog
ByteDance fails to escape YouTubers' copyright suit over AI trainingOct 5, 2026 · Jog
Delhi High Court refuses immediate stay in ANI v OpenAI copyright caseSep 15, 2026 · Jog

Also known as

web scraping · webscraping · scraping · web scraper · web scraping · data scraping · web data extraction · web data extraction

Sources

This entry was machine-edited and has undergone machine expert review; the “In the news” section updates automatically from Agent Age stories.