algorithmic discrimination

basicwidespread

Algorithmic discrimination occurs when an automated system treats people unfairly without justification, for example due to their gender, age, origin, or place of residence.

In plain terms

Imagine a company that hired almost only men as engineers for twenty years, then trains a program to screen resumes based on this data. The program is not evil; it simply diligently learns the pattern: the 'good candidate' is male. It is like a student who prepared from a biased textbook and recites the bias flawlessly.

Algorithmic discrimination occurs when a system's decision or recommendation places someone in a worse position due to a protected characteristic (gender, age, ethnic origin, disability, religion, etc.). This can happen in credit assessments, hiring, insurance premiums, social benefits, or ad display.

It differs from human discrimination in that it is harder to detect and spreads much faster. A biased clerk affects a few dozen people daily, while a biased system can impact hundreds of thousands simultaneously, all while the machine appears objective.

Legally, it does not matter whether the disadvantage was caused by a human or a machine. Responsibility lies primarily with the organization using the system, but the developer also has obligations. Lack of intent is not an excuse.

How it works

Biased training data: the system learns from past decisions, and if those included prejudice (for example, fewer women received promotions), it adopts this as a rule.
Surrogate features: even if gender or origin are removed from the data, the system can infer them from other data (postal code, first name, school, shopping habits). This often leads to indirect discrimination: a seemingly neutral criterion in practice affects a protected group.
Incomplete coverage: if there is little data about a group, the system works less accurately for them. Several studies show, for example, that facial analysis and recognition systems made more mistakes with darker-skinned women.
Self-reinforcing loop: the system's decisions generate new data that feeds back into training. If someone is once rejected, no 'good borrower' data is generated about them, so the bias can strengthen over time.
Discovery and repair: a bias can be found by comparing the system's decisions and errors group by group. The data, the model or the decision process can then be changed, and human review can be built in.

An everyday example

You apply for an online loan and receive rejection within seconds, while a friend with similar income from another district is accepted. If the system classified you into a higher-risk group based on your residence, and this actually affects an ethnic or social group, it could be algorithmic discrimination. If the decision was made fully automatically without human involvement, in the EU you have the right to request human review and an understandable explanation, and to challenge the decision.

Who is affected

jobseeker, new graduateYour resume is increasingly screened by software or an AI agent, and if biased, you may be rejected without a human ever seeing your application. → Work & HR

Banking and insurance customerCredit assessments and insurance pricing often rely on automated scoring, which can indirectly disadvantage certain groups.

entrepreneur, executiveIf your company uses a biased system, the legal and reputational liability is primarily yours, even if the software was purchased from an external vendor.

lawyer, citizenThe requirement for equal treatment also applies to machine decisions, so you must know when and how to challenge an automated decision. → Law

A common misconception

MythThe machine only calculates, so it cannot be biased.

In realityThe system learns from data collected by humans and goals set by humans, so it also learns past inequalities. Moreover, bias is harder to detect due to the appearance of neutrality.

Limits and risks

It is rarely provable from the outside: a single rejection does not reveal whether discrimination occurred; this requires comparing many cases across groups.
Deleting protected data (gender, origin) alone is not a solution, because the system can infer them from other data.
There are multiple mathematical measures of fairness, which generally cannot be met simultaneously, so it is always necessary to weigh what each means in a given situation.
Not every distinction is discrimination: the law prohibits unjustified disadvantage linked to protected characteristics, but not objectively justifiable differentiation.

Similar, but not the same

algorithmic bias – Bias is a systematic deviation in the system's operation, i.e., a technical-statistical phenomenon. Discrimination is its harmful consequence for people, legally assessable. Not every bias leads to discrimination.
Profiling – Profiling is the automated evaluation of personal characteristics, which is not illegal in itself but can lead to discrimination if the classification affects protected groups.
personalization – Personalization adapts the offer to your behavior, whereas discrimination disadvantages you due to belonging to a group. The line is not sharp: personalization can become discrimination if it practically affects a protected group.

Related concepts

Broader

trustworthy artificial intelligence, AI ethics, equal treatment

Narrower

indirect discrimination, Proxy variable

On the edge

algorithmic bias, fairness, Explainability, automated decision-making, high-risk AI system, price discrimination

A brief history

Batya Friedman and Helen Nissenbaum described bias in computer systems in a 1996 paper. The topic moved to the centre of public attention in 2016, when ProPublica reported that the COMPAS risk assessment program used in US courts more often wrongly classified Black defendants as high risk. The program's developer disputed this conclusion, and the debate showed that fairness can be measured in several ways. The EU's Artificial Intelligence Act (AI Act), adopted in 2024, requires high-risk systems to be examined for biases that lead to discrimination, and those biases to be reduced.

Where things stand

By the end of 2026, the phenomenon is well-documented, and regulation addresses it: in the EU, the AI Act, the GDPR, and equal treatment rules together provide a framework. The obligations of the AI Act are coming into force gradually: rules on prohibited practices have been applicable since February 2025, but the application of requirements for high-risk systems was postponed by the EU in 2026, making them mandatory no earlier than the end of 2027. Administrative and judicial practice is still taking shape.

Status: 2026 November

In the news

Appears in 1 Agent Age stories; the curve shows weekly occurrence in the news magazine.

EEOC seeks consent decree in AI discrimination caseSep 21, 2026 · Work & HR

Also known as

algorithmic discrimination · algorithmic discrimination · machine discrimination · AI discrimination · algorithmic discrimination

Sources

ProPublica: Machine Bias (2016)

This entry was machine-edited and has undergone machine expert review; the “In the news” section updates automatically from Agent Age stories.