Or avoid shuttling the data back and forth and go with a unified architecture like the M1. Memcpy is terrible and yet we keep trying to build bigger and better pipes to make it faster with diminishing returns.
> Or avoid shuttling the data back and forth and go with a unified architecture like the M1.
AMD has been shipping unified architectures since August 2011. Nobody cares.
The problem is that while a small low power CPU and a small low power GPU living on the same die and sharing the same memory controller will happily share memory, a big honkin' CPU and a big honkin' GPU don't like to share a memory controller. Some of the big advancements in absolute performance in recent years stems from making the memory controller closer and closer to the relevant processor. In recent AMD systems, the pins on the CPU are directly wired to the pins on the RAM DIMMs. Relatively old mother boards shipped when DDRx memory was the standard, and were still compatible with DDR(x+1) CPUs and RAM when those became the new standard; the motherboard has no RAM logic. If you wanted a unified architecture, you're either going to make the CPU talk to RAM through the GPU, or the GPU talk to RAM through the CPU, or go back to the early 2000s and put a north bridge on the motherboard.
In addition to all that, the bottlenecks are different. The bottleneck on a GPU is always, 100% of the time, memory bandwidth. Latency is mostly irrelevant. The bottleneck on a CPU is usually (95%? 99%?) the memory latency. There's no such thing as RAM that will perform well for both a CPU and a GPU.
I strongly suspect that this is Nvidia's next big move. A giant CPU/GPU hybrid with unified memory. Who doesn't want threadripper scale Nvidia/ARM cores?
Yes, unified memory being the key here. FAFAIK GPUs still use a hub-and-spoke model, where all processing is done in the hub but data is stored on the rim. They achieve their speedup by increasing the number of spokes (data channels). You could do better by moving the data processing to the spokes as well, and using the hub only for orchestration and synchronization.
At least, it's computationally better. But it won't allow scaling the processing power and memory size independently, so it might require a different commercial model.
