Pipelining
How CPUs process multiple instructions simultaneously
Introduction
In the previous chapter, we learned that every instruction goes through a fetch-decode-execute cycle. If the CPU processes one instruction at a time — waiting for each to complete before starting the next — most of its components would sit idle for large portions of each cycle. This would be terribly inefficient, like having only one worker on an entire assembly line.
Pipelining solves this problem by allowing multiple instructions to be in different stages of the instruction cycle at the same time. While one instruction is being executed, the next is being decoded, and the one after that is being fetched. This dramatically increases the throughput — the number of instructions completed per unit of time — without requiring a faster clock.
Modern CPUs use pipelines that are 14 to 20 stages deep (or even more in some designs). Pipelining is one of the most important innovations in computer architecture, and understanding it is essential to understanding how modern processors achieve their incredible performance. However, pipelines also introduce challenges — hazards, stalls, and the need for complex branch prediction logic.
How It Works
A CPU pipeline works like a car assembly line. Instead of one team building an entire car from start to finish, each station performs a specific task: one installs the engine, another fits the doors, a third paints the body. Every station works simultaneously on a different car, so a finished car rolls off the line every few minutes instead of every few days.
Household Object Analogy
Think of doing laundry. Without pipelining, you would wash one load, wait for it to finish, dry it, wait for that to finish, then fold it. With pipelining, while one load is drying, the next is washing, and you are folding the previous load. All three stages happen at the same time — the laundry throughput triples, even though each individual step takes the same amount of time.
Deeper Dive
A classic five-stage RISC pipeline consists of five stages: IF (Instruction Fetch), ID (Instruction Decode), EX (Execute), MEM (Memory Access), and WB (Write Back). Each stage corresponds to a different part of the CPU: the fetch unit, the decoder, the ALU, the cache/memory controller, and the register file.
In cycle 1, instruction 1 is fetched. In cycle 2, instruction 1 is decoded while instruction 2 is fetched. By cycle 5, instruction 1 completes its write-back, and instructions 2, 3, 4, and 5 are all in various stages of processing. From that point on, one instruction completes every cycle — a five-fold improvement over the non-pipelined approach.
Pipeline Hazards
Pipelines are not without problems. A hazard is a situation that prevents the next instruction in the pipeline from executing during its designated clock cycle. There are three types: structural hazards (two instructions need the same hardware resource), data hazards (an instruction depends on the result of a previous one), and control hazards (caused by branches and jumps).
Data hazards are the most common. For example, if instruction A computes a value in R1 and instruction B needs that value, instruction B cannot execute until A finishes. Solutions include forwarding (also called bypassing) — routing the result directly from the ALU output to the next instruction's input — and stalling (inserting a bubble in the pipeline) when forwarding is not enough.
Branch Prediction
Branch prediction is a technique used to handle control hazards. When the CPU encounters a conditional branch (like an "if" statement), it does not know which instruction will come next until the condition is evaluated. Instead of stalling the pipeline until the condition is resolved, the CPU predicts whether the branch will be taken and fetches instructions from the predicted path.
If the prediction is correct, the pipeline continues without interruption. If it is wrong, the CPU must flush the incorrectly fetched instructions and start fetching from the correct address — a penalty of several cycles. Modern branch predictors use sophisticated history tables and pattern matching to achieve accuracy rates of 95% or higher. The best predictors can handle complex patterns like nested loops with remarkable reliability.
Superscalar and Out-of-Order Execution
A superscalar CPU has multiple execution units — multiple ALUs, multiple floating-point units, multiple memory ports — allowing it to execute more than one instruction per clock cycle. A superscalar processor might have three ALUs and two FPUs, meaning it can potentially execute five instructions simultaneously if there are no dependencies.
Out-of-order execution takes this further by allowing the CPU to reorder instructions dynamically. If instruction 3 depends on instruction 2 but instruction 4 is independent, the CPU can execute instruction 4 ahead of instruction 3. This keeps the execution units busy even when there are data dependencies in the program flow. The CPU then reorders the results to maintain the illusion of sequential execution.
Throughput vs. Latency
Pipelining improves throughput — the number of instructions completed per second — but does not reduce latency — the time it takes to complete a single instruction. In fact, pipelining may slightly increase latency because of the overhead of the pipeline registers between stages. However, the throughput gain is so large (up to five times for a five-stage pipeline) that the small latency increase is well worth it.
This trade-off is fundamental to CPU design. For interactive applications, latency matters most. For batch processing and server workloads, throughput is king. Modern CPUs balance both: deep pipelines for high throughput, with sophisticated hazard detection to minimise stalls and keep latency under control.
Key Insight
Pipelining is the reason a 3 GHz CPU from 2024 can be dramatically faster than a 3 GHz CPU from 2004 — even at the same clock speed. Improvements in pipeline depth, branch prediction, superscalar width, and out-of-order execution have enabled far more work per clock cycle, even when frequency stopped scaling.
Advanced
At a deeper level, pipelining involves rules and patterns that engineers use worldwide. Pipeline follows standards so different brands and devices can still work together. That is why your phone, school laptop, and game console can all connect to the same network or use the same apps.
Stage does not happen in a straight line. Systems often use backup paths, error checking, and retries so information arrives correctly. When something fails, smart Hazard design helps the system recover instead of shutting down completely.
Scientists and engineers keep improving these systems every year — making them faster, safer, and more energy-efficient. The ideas you learn in this chapter are the same building blocks used in real data centers, robots, apps, and websites around the world.
Vocabulary Table
| Term | Definition |
|---|---|
| Pipeline | A technique where multiple instructions overlap in execution by being in different stages of the instruction cycle simultaneously |
| Stage | One discrete step in the pipeline, such as Fetch, Decode, Execute, Memory Access, or Write Back |
| Hazard | A condition that prevents the next instruction from executing in its designated clock cycle within the pipeline |
| Stall | A pipeline bubble inserted when a hazard prevents an instruction from advancing to the next stage |
| Branch Prediction | A technique that guesses the outcome of a conditional branch to avoid stalling the pipeline |
| Superscalar | A CPU design that includes multiple execution units to execute more than one instruction per cycle |
| Throughput | The number of instructions completed per unit of time, the primary benefit of pipelining |
| Latency | The time taken to complete a single instruction from start to finish |
| Forwarding | A technique that routes results directly from one pipeline stage to another to resolve data hazards |
| Out-of-Order Execution | A technique where the CPU reorders instructions dynamically to keep execution units busy |
Fun Facts
The first commercial CPU with pipelining was the IBM System/360 Model 91 (1966). It used a sophisticated pipeline with out-of-order execution and register renaming — concepts that are still state of the art today, nearly 60 years later.
The Pentium 4 (2000) had an extremely deep 20-stage pipeline, later extended to 31 stages in the Prescott revision. This allowed very high clock speeds (up to 3.8 GHz) but also meant branch mispredictions were very costly — flushing up to 31 stages of work.
Apple's M1 and M2 chips use a wide superscalar design with a 9-wide decode and 14-wide issue — meaning they can fetch, decode, and issue up to 14 instructions per cycle to various execution units. This wide design contributes to their exceptional performance per watt.
Spectre and Meltdown, the famous security vulnerabilities discovered in 2018, exploited a feature called "speculative execution" — where the CPU executes instructions ahead of time based on branch predictions. These vulnerabilities allowed malicious programs to read protected kernel memory through timing side channels.
Some GPUs have pipelines that are thousands of stages deep, but for a different reason: they process massive numbers of pixels or vertices in parallel. GPU pipelines are optimised for throughput on highly parallel workloads, while CPU pipelines are optimised for low-latency single-thread performance.
Interactive Diagram
Launch the interactive diagram to see Pipelining in action.
Open Interactive DiagramThe interactive diagram for this chapter demonstrates Pipelining. It shows a pipeline diagram showing multiple instructions at different stages simultaneously.
What to explore:
- toggle between non-pipelined and pipelined execution; watch instructions overlap; see the speed improvement
- pipelining allows the CPU to work on multiple instructions at once by overlapping their fetch, decode, and execute stages
Knowledge Check
1. What is the primary benefit of pipelining in a CPU?
Answer: It increases throughput by allowing multiple instructions to overlap in execution
2. What happens when a branch is mispredicted in a pipelined CPU?
Answer: The CPU must flush the pipeline and fetch from the correct address, losing several cycles
3. How does a superscalar CPU differ from a standard pipelined CPU?
Answer: It has multiple execution units and can execute more than one instruction per cycle