I was recently able to get a workload of mine optimized enough on Zen 5 to hit a sustained 6.0 IPC/core (3.0 / thread) at 5.1 GHz. Seeing the a > 99.8% branch prediction rate and a > 99.99% L2 cache hit rate retiring > 1T instructions every 3 seconds feels amazing.
Zen5 is incredible when you're able to make the most of it. I’m super excited about Zen6.
For someone who has never touched workload optimization, can you share a little about how you measure this? Does IPC mean inter-process communication in this context?
I know nothing about workload optimization, and I'd like to know more - right now I feel like the Good Burger gif. "Yeah, I know some of these words".
I've never done workload optimisation myself, but have looked into it just-in-case. It's a really interesting field and unfortunately, you need to know about both your compiler and target CPU/GPU to get the big gains. I believe most modern CPUs have registers for cache use. You have to do some guesswork to get IPC numbers, since core frequency can vary.
Increasing instructions-per-clock is all about minimising program branching (essentially 'if' statements). Because a CPU core can execute instructions faster than main memory can fetch em. It's a fun game to look at an 'if' statement and figure out how you could instead make it an arithmetic operation :).
Maximising cache hits is all about how you structure and access data. For example, if a cache entry is n bytes long, you want to ensure your struct is smaller than n bytes. Having very consistent access patterns can also help (e.g. arrays-of-structs vs structs-of-arrays).
This field is super deep, it's very fun to learn about!
Though you should always ask what your compiler/optimizer is doing for you. Research into how to turn your ifs into arithmetic is ongoing. If the compiler can optimize your simple ifs into the complex arithmetic then you should go with the simple if and let the compiler do that. (unfortunately in many cases the compiler cannot be sure because of some special case - even though odds are you don't care about it)
Thank you. Does workload optimisation usually refer to the CPU/GPU alone, or would it include disk or network IO? Or would that be more accurately called something else, like profiling?
in the context of CPU performance / architechture `IPC` pretty much always means "Instructions per clock". That's a measure of the internal parallelism a given CPU core is achieving. Most of the time code is not able to get very close to the theoretical maximums a core can achieve for very long, so the numbers the OP is quoting are incredibly impressive.
I'm not too familiar with the CPU world of performance measurement, but in GPU land, and I suspect for CPUs too, there are a slew of hardware "performance counters" which are registers that increment every time the event they measure happens.
So there could be a "instructions completed" counter and a "cycle" counter, and before starting the benchmark, you record the current value of both counters, then after the benchmark, record the final values, and compute the Instructions Per Cycle (IPC) as the difference in instructions completed over the difference in cycle count.
As for optimization, at a high level it's about minimizing the amount of time any part of the CPU is waiting for other parts of the CPU. The specifics require a lot of background knowledge about how modern CPUs work, more than can fit in a post, but if you're interested, topics to read about include:
### CPU cache hierarchy
CPUs store copies of data from RAM in smaller, faster memory physically closer to where the computation happens, so it's available more quickly. The CPU decides what values to store in the cache, and gives only limited control to the program, so an optimal program needs to be careful to not make the CPU make bad caching decisions (including for synchronizing the cache between multiple threads of execution).
### Instruction pipelining and out-of-order execution (a.k.a. "superscalar" execution)
Modern CPUs operate like an assembly line. A new instruction can start executing before the previous instruction(s) finishes. CPUs also have redundant hardware, so multiple instructions can be in progress at the same step of the pipeline. But there are limitations; sometimes the input to one instruction depends on the output of the previous instruction, so the whole pipeline stalls until the result is ready. An optimal program orders its operations to avoid these stalls as much as possible.
### Branch predicition
When the code to execute depends on the result of a computation, like in an `if` statement, we call it a branch. Branches can stall the pipeline, because the CPU doesn't know what instructions to execute next until the current result is ready. However, to mitigate this, modern CPUs predict which code-path will be taken when a branch is reached and begin executing the associated instructions immediately. If the prediction is right, the pipeline stall is avoided, but if it's wrong the pipeline state has to be restored to what it was before the wrong branch started executing, which is even more expensive than a stall. Usually, the CPU predicts branches correctly, so branch prediction is a net gain. Optimal programs need to understand how the CPU makes these predictions and make their branches as predictable as possible.
### Single Instruction, Multiple Data (SIMD)
Some programs do the same operations to each item of a set of data. CPUs have so-called SIMD instructions to accelerate this by, unsurprisingly, performing the same operation to multiple items at once. For example, they could add 4, 8, or even up to 64 pairs of numbers at once, depending on the instruction set and the range of the inputs. Suitable programs are optimized by arranging their inputs and operations so that SIMD instructions can be used — either directly or by being written in a way that a compiler can translate individual operations to SIMD operations.
Thanks. Is there any convenient tool you'd recommend for watching those registers?
I've said in another comment that any optimisation I'd be doing would be looking at the entire stack, and CPU/GPU optimisation would only be as a learning exercise. I'm a tiny bit familiar with caching and instruction pipelines thanks to college, and branch prediction thanks to Spectre-class bugs. I want to find some time to dive deeper, now.
Their successors had more tangible performance improvements in everyday applications, so yes they were quickly forgotten. Also, it was a reduction in max core count from 10 to 8, intel changes sockets too often, and 14nm+++++.
It’s a backtracking search program looking for integer solutions to a specific problem. I’ve tuned a series of bloom filters to fill my L2 cache so that I rarely have to touch main memory (this alone took my IPC from 0.1-0.3 to 3.0 per thread). Without SMT it’s 4.6 IPC/core.
I think it’s only able to exceed 4.0/thread with SMT off because of a uops cache? From what I’ve read the Zen5 front end only had a 4-wide instruction decode per thread.
Zen5 is incredible when you're able to make the most of it. I’m super excited about Zen6.