Explainable AI (XAI)

advancedspreading

A collection of methods that make it understandable to humans why an AI system made a particular decision or gave a specific answer.

In plain terms

Most modern AI systems work like a very experienced but laconic expert: they give the result but not the reason. Explainable AI aims to reveal the "why": which data, criteria, or patterns led to the specific decision.

Think of a doctor who not only prescribes medication but also explains which finding led to that decision. This does not make you a doctor, but you can ask questions, disagree, and notice if something was misunderstood. We expect the same from machines when they decide on loans, jobs, or diagnoses.

How is it different from traditional software? In an old-style program, rules were written by humans, so they can be traced. A learning system, however, forms its own internal connections from millions or billions of numbers that no one can read directly. Explainability tries to bridge this gap.

How it works

The simplest path is an inherently transparent model: for example, a decision tree or scoring sheet where every step is traceable. These are well understood but often weaker in complex tasks (such as image or text recognition).
For complex "black box" models, they prepare post-hoc explanations: they estimate how much each input data point (for example, income, age, debt) shifted the result in one direction or the other.
A common method is the "what if" (counterfactual) explanation: the system shows the smallest change that would have led to a different decision, for example, "if your monthly income were 40,000 Hungarian forints higher, you would get the loan".
For images and texts, they highlight what the model "attended to": which spot on the X-ray or which words in the letter it based its judgment on. These highlights are also only approximations.
With large language models, researchers also examine internal operations, meaning which parts of the network represent which concepts. This field is still in the research phase.

An everyday example

You apply for an online loan, and the bank's system rejects it within a few minutes. In an explainable system, you see not only "no", but also that your short employment history and existing credit limit weighed most heavily in the decision. This way you know what you should change, and also if the bank worked from incorrect data.

Who is affected

bank customer, investorCredit assessment and risk classification are increasingly automated. In the EU, if a decision with significant impact on you is made solely by a machine, you have the right to request meaningful information about the basis of the decision and to ask for human review. → Money

patient, doctorA doctor can responsibly accept or reject a diagnosis-assisting system's suggestion only if they see what it is based on. → Health

jobseeker, new graduateIf a machine filters your resume, without an explanation it remains unclear whether you were rejected due to a genuine criterion or a hidden bias. → Work & HR

developerExplainability tools can identify when a model gives a correct answer for the wrong reason, which is also a basis for regulatory compliance. → Code

A common misconception

MythIf the AI justifies its answer, that justification shows exactly what happened inside the machine.

In realityThe explanation is usually a simplified approximation, not a description of the actual calculation. A chatbot's justification in its own words can also be a post-hoc formulated, plausible text that does not necessarily match the real reasons.

Limits and risks

The explanation can be convincing yet inaccurate, so a well-sounding justification alone does not prove that the decision is correct or fair.
Sometimes a trade-off has to be made between interpretability and performance: for processing images and text, the most accurate models are the hardest to interpret. With tabular data (for example, in credit scoring), however, a simple, transparent model is often just as good.
Different explanatory methods can give different reasons for the same decision, and there is no single standard for what counts as a "good" explanation.
The explanation shows what influenced the decision but not whether that factor was justified; this still requires human judgment.

Similar, but not the same

transparency – Transparency means that for those affected (users, authorities), it is documented and accessible how the system was built, what data it learned from, and what it is capable of. Explainability is narrower: it means that the reason for a specific decision is understandable.
Interpretability – Many use the two terms as synonyms. Where they are distinguished, interpretability refers to understanding the model's internal workings, while explainability is satisfied with providing a human-useable justification for the decision.
human oversight – Human oversight requires that a human checks and can override the machine, while explainability provides the information needed for this.

Related concepts

Broader

trustworthy AI, AI ethics, Responsible AI

Narrower

mechanistic interpretability, Counterfactual explanation, feature importance

On the edge

fekete doboz, transparency, algorithmic bias, human oversight, EU AI-rendelet

A brief history

The demand for justification of machine decisions already appeared in the expert systems of the 1970s and 1980s. The concept in its current sense, and the abbreviation XAI, became widely known through the Explainable AI program launched in 2016 by DARPA, the US defense research agency, when the opacity of deep learning models had become a practical problem.

Where things stand

Explainability tools are widely used in financial and healthcare systems. The EU AI Regulation imposes transparency requirements on high-risk systems and grants affected individuals the right to request an explanation for decisions made about them. However, these rules are not yet applicable: the original start date of August 2026 was postponed by the EU, with most high-risk systems now scheduled for December 2027. The reliable explanation of the internal workings of large language models and agents remains an open research question.

Status: 2026 November

Also known as

Explainable AI · explainable artificial intelligence · explainable AI · explainable AI · explainable artificial intelligence · XAI · explainability · Explainability

Sources

This entry was machine-edited and has undergone machine expert review; the “In the news” section updates automatically from Agent Age stories.