Merging Memory and Compute

New Approaches Promise Power Savings But Bring Confusion
Near-memory and in-memory compute are techniques for reducing computing power—especially for AI. But they mean different things to different companies. Understanding the differences is important for understanding how some AI chips work.
Bryon Moyer
Bryon Moyer

In high-performance computing (HPC) and artificial intelligence (AI), data movement has surpassed computing as the main energy consumer. Moving a 64-bit word from off-chip DRAM to a CPU can consume a thousand times the energy of performing a compute operation on that word. For AI workloads, traditional von Neumann approaches can dedicate as much as 90% of energy to fetching data and delivering it to the compute engine. Reducing energy is important for both data centers and battery-powered edge equipment. Developers therefore seek creative new approaches to reducing the number of memory fetches.

Among these approaches are ideas variously called near-memory and in-memory compute, although no formal definition exists for either term. Near memory refers to moving memory close to the compute function or vice versa. In memory has many embodiments, such as adding compute resources to a memory array, adding logic or other computation circuits to a memory bit cell, and having a memory element perform the computation.

Many of these techniques started in universities or other research labs, but they are beginning to show up in commercial products. Ambient Scientific and Upmem are approaching production of near-memory chips. Samsung is pursuing a near-memory proof-of-concept, and Untether has a near-memory chip in production. Mythic is sampling a flash-based in-memory chip, and TetraMem has announced an in-memory chip that employs resistive RAM (RRAM). Practical challenges blocking commercial production include differences between DRAM and logic processes, analog-process variability, and digital/analog conversion.

Free Logic Newsletter

Get the latest analysis of new developments in microprocessors and other semiconductor products.


Subscribers can view the full article in the TechInsights Platform.

 

You must be a subscriber to access the Microprocessor Report

If you are not a subscriber, you can purchase a subscription or you can purchase an article. Enter your email below to contact us about access.

The authoritative information platform to the semiconductor industry.

Discover why TechInsights stands as the semiconductor industry's most trusted source for actionable, in-depth intelligence.