CPU Cache

Introduction

Modern CPUs can execute instructions blazingly fast — but they are often held back by how slowly data arrives from main memory. Reading from RAM takes around 100 nanoseconds, while a CPU register can be read in 0.3 nanoseconds. This enormous speed gap, known as the memory wall, means the CPU spends most of its time waiting for data unless there is a way to bridge the difference.

The solution is CPU cache — small, ultra-fast memory banks built directly onto the processor die. Cache stores copies of frequently accessed data from main memory so the CPU can retrieve it in just 1 to 20 nanoseconds. Modern CPUs use a hierarchy of cache levels: L1, L2, and L3, each with increasing size but decreasing speed.

How It Works

When the CPU needs to read data, it first checks the L1 cache — the smallest and fastest cache, typically 32 KB to 128 KB per core, divided into L1 instruction cache (L1i) and L1 data cache (L1d). If the data is found, it is a cache hit and the data is returned in 3-5 CPU cycles. If not, the CPU checks the L2 cache (256 KB to 1 MB per core, 10-20 cycles), and then the L3 cache (8 MB to 64 MB shared across all cores, 30-50 cycles). If all caches miss, the CPU must fetch from main memory, which costs 200+ cycles.

Cache exploits two fundamental properties of programs: temporal locality and spatial locality. Temporal locality means that if a memory location is accessed, it is likely to be accessed again soon (think of a loop counter). Spatial locality means that if a memory location is accessed, nearby locations are also likely to be accessed soon (think of iterating through an array). Cache loads data in blocks called cache lines, typically 64 bytes, to exploit spatial locality.

Cache is built from SRAM (Static Random Access Memory), which is faster but much more expensive and power-hungry than the DRAM (Dynamic RAM) used for main memory. SRAM uses 6 transistors per bit (vs. 1 transistor + 1 capacitor for DRAM) and does not need periodic refreshing, making it ideal for cache but impractical for large capacities.

Deeper Dive

Different cache organizations exist. A direct-mapped cache maps each memory address to exactly one cache line — simple and fast but prone to conflict misses. A fully associative cache allows any memory address to be stored in any cache line — flexible but complex and slow. Most modern CPUs use N-way set-associative caches, which strike a balance by dividing the cache into sets of N lines each, where a memory address maps to a specific set but can occupy any of the N lines within that set.

When the cache is full and a new line must be loaded, a replacement policy decides which existing line to evict. The most common is LRU (Least Recently Used), which evicts the cache line that hasn't been accessed for the longest time. Modern CPUs use approximation algorithms like pseudo-LRU or RRIP (Re-Reference Interval Prediction) to achieve near-LRU performance with less hardware overhead.

Cache coherency is a critical challenge in multi-core processors. When one core modifies data in its private L1 or L2 cache, other cores' caches may still hold the old value. Cache coherency protocols like MESI (Modified, Exclusive, Shared, Invalid) and MOESI ensure that all cores see a consistent view of memory. The L3 cache often serves as a snoop filter or last-level cache (LLC) to reduce coherency traffic.

Advanced

Cache latency is measured in CPU clock cycles. An L1 hit typically costs 3-5 cycles, L2 costs 10-20 cycles, and L3 costs 30-60 cycles. Main memory access costs 200-400 cycles. The cache miss penalty is the time wasted when data is not found in cache — minimizing misses is one of the most important optimizations in both hardware design and software development.

The inclusive vs. exclusive cache design determines whether data in L1/L2 is also duplicated in L3. Inclusive caches (common in Intel processors) guarantee that any line in L1 or L2 is also in L3, simplifying coherency at the cost of capacity. Exclusive caches (used in AMD's Zen architecture) ensure that data appears only in one cache level, maximizing effective capacity but requiring more complex data movement.

Some modern CPUs feature hardware prefetching, where the cache controller predicts future memory accesses and loads data before the CPU requests it. The prefetcher detects patterns like sequential access (streaming through arrays) or strided access (accessing every Nth element) and proactively fetches cache lines. Prefetching can dramatically reduce cache miss rates but must be tuned carefully to avoid polluting the cache with unused data.

Vocabulary Table

Fun Facts

Interactive Diagram

Explore the L1, L2, and L3 cache hierarchy and see how data moves between cache levels.

Open Interactive Diagram

---

The interactive diagram for this chapter demonstrates CPU Cache. It shows the cache hierarchy (L1, L2, L3) with data moving between cache levels and RAM.

Knowledge Check

Chapter Navigation