pipeline
A pipeline is a series of linked, automated work steps where the output of each step becomes the input of the next.
In plain terms
The word pipeline literally means a line of pipes, but a better analogy is a factory assembly line. At one end of the line, you put in the raw material, something happens to it at every station, and at the end, the finished product comes out. No one needs to hand anything over manually between stations.
In the software world, the same happens with data, program code, or text. One step, for example, collects the data, the next cleans it, the third analyzes it, and the fourth creates a report from it. Once set up, it runs through the same steps repeatedly without human intervention.
In AI, you encounter it most frequently in two places. One is model training: data collection, preparation, training, and verification. The other is agent work, where one or more agents perform subtasks in a predefined sequence.
An agent working freely decides for itself what to do next. In a pipeline, however, the sequence is fixed in advance, making it more predictable and easier to audit, but less flexible.
How it works
- The task is broken down into smaller, well-defined steps. Ideally, each step does just one thing.
- They determine the sequence and the format in which each step passes the result to the next.
- Something starts the sequence: a schedule (for example, every day at dawn), an event (a new file arrived, the developer changed the code) or a person pressing a button.
- The steps run in sequence while the system logs what happened. If any step reports an error, the sequence typically stops to prevent the faulty result from proceeding.
- At the end, the final result appears: a report, an updated website, a trained model, or work completed by an agent.
An everyday example
Every morning you receive a newsletter summarizing the most important articles from the previous day. Behind this, a pipeline is likely working: one step collects the articles, another filters out duplicates, an AI model summarizes them, and the final step sends you the email.
Who is affected
A common misconception
Limits and risks
- The error cascades: if an early step yields a bad result and no one notices, every subsequent step builds on that flawed foundation.
- Rigid: if the task changes mid-process or an unexpected situation arises, the fixed sequence cannot adapt on its own.
- Automation can lull you into complacency: just because the command line ran without error messages does not mean the result is correct, so checkpoints are needed.
- In AI steps, the same input does not always produce the same output, so a pipeline's result may differ from run to run.
Similar, but not the same
- munkafolyamat (workflow) – Workflow is a broader concept: it can include human decisions, approvals, and rollbacks. A pipeline is typically an automatic, pre-defined sequence of steps where data moves in one direction, even if it branches.
- AI-agent – An agent selects its next step to achieve a goal, whereas in a pipeline, the sequence of steps is predefined.
- sales pipeline – In business, the same word refers to the stage-by-stage tracking of ongoing deals, from initial interest to contract signing. It is not a software processing pipeline.
- processzor-pipeline – In hardware, a pipeline means the processor breaks down the execution of an instruction into parts and works on different parts of multiple instructions simultaneously. This is the machine's internal operation, not a sequence of data or task processing steps.
Related concepts
automation, munkafolyamat, Software architecture
adatpipeline, CI/CD, training pipeline
orchestration, AI-agent, multi-agent system, MLOps, RAG
A brief history
The software meaning of the word largely originates from the Unix operating system. As early as 1964, Douglas McIlroy suggested in an internal memo that programs should be connectable like garden hose segments. Ken Thompson implemented the "pipe" solution in Unix in 1973, allowing the output of one program to be directly connected to the input of another. From there, the term spread to data processing, software releases, and then AI systems. In hardware, the term "pipeline" for processor instruction processing has been used since the 1960s.
Where things stand
By the end of 2026, a pipeline is a fundamental concept in software development and data processing, found in most serious systems. The novelty is that individual steps are increasingly performed by AI models and agents, while fixed sequences are often combined with more freely operating agents.
In the news
- Roche partners with Chai Discovery to deploy Chai-3 model across its pipelinesNov 5, 2026 · Health
- AI is rewriting entry-level work, putting the junior talent pipeline and career ladder at riskNov 2, 2026 · Work & HR
- Circle, Polygon, Instinct, Hecto and Kakao Pay advance stablecoin payments for AI agentsOct 26, 2026 · Money
- AI threatens the junior lawyer pipeline as first AI-trained law graduates enter the workforceOct 14, 2026 · Jog
- AI erodes entry-level jobs, leaving HR to rebuild the talent pipelineOct 12, 2026 · Work & HR
- AI code review: Atlassian splits it between two agents, JetBrains proposes a three-tier reviewOct 12, 2026 · Code
- Analysis: AI-enabled drug discovery pipeline is growing but remains largely early-stageOct 8, 2026 · Health
- AI coding could break the junior engineer pipeline, LeadDev says; Jensen Huang weighs inOct 5, 2026 · Code
Also known as
pipeline · pipe-line · pipeline · processing pipeline · processing chain · adatpipeline · data pipeline · CI/CD pipeline
Sources
This entry was machine-edited and has undergone machine expert review; the “In the news” section updates automatically from Agent Age stories.