The Ampere Altra Max Review: Pushing it to 128 Cores per Socket
by Andrei Frumusanu on October 7, 2021 8:00 AM EST- Posted in
- Servers
- Arm
- Neoverse N1
- Ampere
- Altra Max
128 Cores Mesh Setup & Memory Subsystem
Starting off the testing, one thing that is extremely intriguing about Ampere’s implementation of their Altra designs is the fact that they’re achieving more than 64 cores whilst still using Arm’s CMN-600 mesh network IP. In our more recent coverage earlier this year of Arm’s newer upcoming CMN-700 mesh network, we wrote about the fundamental structure of the CMN mesh and its building blocks, such as RN-F, HN-F, and components such as CALs.
In a typical deployment, a mesh consists of cross-points (XPs) to whose RN-F (Fully coherent request node) connect to either directly a CPU, or a CAL (component aggregation layer) which can house two CPUs.
Our initial confusion last year with the Quicksilver designs was that 80 cores was more cores than what the CMN would actually support when configured with the maximum mesh size and two cores per CAL per XP – at least officially. Ampere back then was generally coy about talking about the mesh setup, but more recent discussions with Arm and Ampere, the companies have divulged that it’s also possible to house the CPUs inside of a DSU (DynamiQ Shared Unit), the same cluster design that we find in mobile Arm SoCs with Cortex CPUs.
Ampere has since confirmed the mesh setup in regards to the CPUs: instead of housing cores directly to the mesh via RN-Fs, or even via a CAL on each XP, they are employing two DSUs, each with two Neoverse-N1 cores, connected to a CAL, connected to a single XP. That means each mesh cross-point houses four cores, vastly reducing the needed mesh size to be able get to such core numbers – this is both valid for the Quicksilver 80-core designs and the new Mystique 128-core designs. The only differences with the Mystique design is that Ampere has now simply increased the mesh size (we still don’t have official confirmation on the exact setup here).
From a topology perspective, the Altra Max is still a massive monolithic 128-core chip, with competitive core-to-core latencies within the same socket. Our better understanding of the use of DSUs in the design now also explains the more unique low-latency figures of 26ns which only happens between two core pairs – these would presumably be two sibling cores housed within a single DSU, and coherency and communications between the two doesn’t have to go out into the mesh, which incurs higher latencies.
We had discussed Ampere’s quite high inter-socket latencies in our review of the Altra last year, as a fresh reminder, this is because the design doesn’t have a single coherency protocol that spans from the mesh network to the remote mesh of the other socket – instead having to have to go through an intermediary cache-coherency protocol translation for inter-socket communication, CCIX in this case. In particular this wasn’t very efficient for when two cores within a socket have to work on a remote socket cache line – the communication between cores in DSU is very efficient here, however between cores in a mesh it means doing a round-trip to the remote socket, resulting in pretty awful latencies.
The good news for the new Altra Max design is that Ampere was able to vastly improve the inter-socket communication overhead by optimising the CCIX stack part of things. The results are that socket-to-socket core latencies have gone down from ~350ns to ~240ns, and the aforementioned core-to-core within a socket with a remote cache line from ~650ns to ~450ns – still pretty bad, but undoubtedly a large improvement.
Latencies within a socket can be up at the extremes, simply due to the larger mesh. Ampere has boosted the mesh frequency from 1800MHz to 2000MHz in this generation, so there is a slight boost there as well as associated bandwidth.
Looking at the memory latencies of the new part, comparing the Q80-33 to the M128-30 results at 64KB page size, of course the first thing that is noticeable is the fact that the new Altra Max system now only has 16MB of SLC, or system level cache, half of the 32MB of the Quicksilver design. This was one of the compromises the company decided to make when increasing the core count and mesh in the Mystique design.
L3/SLC latencies are also slightly up from 30 to 33.6ns, some of that is the 10% slower CPU clock, but most of it is because the larger mesh with more wire distance and more cross-points for data to travel across.
One thing that we hadn’t covered in our initial review was the chip running regular 4K pages – the most surprising aspect here is not the fact that things look a bit different due to the 4K pages themselves, but rather because the prefetchers now behave totally differently. In our first review we believed that Ampere had intentionally disabled the prefetchers due to the sheer core count of the system, but looking at the 4K page results here they appear to be in line with what we saw in behaviour in Amazon’s Graviton2. Notably the area/region prefetcher no longer pulls in whole pages in patterns which have strong region locality, such as the “R per R page” pattern (Random cache lines within a page followed by random pages traversal). Ampere confirmed that this was not an intentional configuration at 64KB pages, though we didn’t have an exact explanation for it. I theorise it’s maybe a microarchitectural aspect of the N1 cores trying to avoid increased cache pressure at larger page sizes.
This weird behaviour also explains the discrepancy in scores between Graviton2 and Altra in SPEC’s 507.cactuBSSN_r, which is actually due to the prefetchers working or not between 64/4KB pages.
It’s still possible to run the chip in either monolithic, hemisphere, or quadrant modes, segmenting the memory accesses between the various memory controller channels on the chip, as well as the SLC. Unfortunately, at 128 cores and only 16MB of SLC, the quadrant mode results in only 4MB of SLC, which is quite minuscule for a desktop machine, much less a server system. Each core still has 1MB of L2, however as we’ll see later in the tests, there are real-world implications of such tiny SLC sizes.
In terms of DRAM bandwidth, the Altra system on paper is equal to AMD’s EPYC Rome or Milan, or Intel’s newest Ice Lake-SP parts, due to all of them running 8-channel DDR4-3200. Ampere’s advantage comes from the fact that it is able to detect streaming memory workloads and automatically transform them into non-temporal writes, avoiding an extra memory read due to RFO (read for ownership) operations that “normal” designs have to go through. Intel’s newest Ice Lake-SP design has a somewhat similar optimisation, though working more on a cache-line basis and seemingly not able to extract as much bandwidth efficiency as the Arm design. AMD currently lacks any such optimisation and software has to have explicit usage of non-temporal writes to be able to fully extract the most out of the memory subsystem – which isn’t as optimal as a generic workload agnostic optimisation that Ampere or Intel currently employ.
Between the Q80-33 and M128-30, we’re seeing bandwidth curves that roughly match – up to a certain core count. The new M128-30 naturally goes further to 128 cores, but the resulting aggregate bandwidth also goes further down due to resource contention on the SoC – something very important to keep in mind as we explore more detailed workload results on the next pages.
At lower core count load, we’re seeing the M128-30 bandwidth exceed that of the Q80-33 even though it’s at lower CPU frequencies, again this is likely due to the fact that the mesh is now running 11% faster in frequency on the new design. AMD’s EPYC Milan still has access to the most per-core bandwidth in low thread situations.
Facebook
Twitter
RSS
60 Comments
View All Comments
Wilco1 - Sunday, October 10, 2021 - link
> You need only note the cases where Max significantly underperforms, relative to its 80-core sibling, to see where the cache reduction is likely an issue.There are regressions in 4 of the 10 integer benchmarks - only mcf is significant. However if you look closely, Altra Max still beats/equals the 8380 in 3 out of those 4. Clearly a 40MB L3 is not large enough for these benchmarks, so would you also call that a "major liability"? EPYC beats all by a huge margin in these 4, so clearly 256MB L3 works well, but it's also way too expensive for a monolithic die.
> The reason why there are so many different benchmarks is that you can't just seize on the aggregate numbers to tell the whole story.
No. The aggregate result averages out the extremes and is a better prediction for average performance. For example Altra Max is slower than Altra on gcc_r and far behind EPYC. However in LLVM compilation Altra Max beats Altra by ~20% and is pretty much equal to the 7763. So in real world tests EPYC's huge caches don't help nearly as much as the gcc_r subtest suggests.
mode_13h - Sunday, October 10, 2021 - link
> There are regressions in 4 of the 10 integer benchmarks - only mcf is significant.When you have 45% more core x GHz, *any* regression is significant! By that token, we should also be marking the xz test as underperforming, since it's only a ~20% improvement.
It's also convenient to seize on specint, when it suffers regressions on 7 of the 12 specfp tests.
> Altra Max still beats/equals the 8380 in 3 out of those 4.
> Clearly a 40MB L3 is not large enough for these benchmarks,
> so would you also call that a "major liability"?
This seems like a rather disingenuous point. To say anything about the 8380's cache, we'd need to see a comparison against other Ice Lake CPUs with a different core-to-cache ratio.
> No. The aggregate result averages out the extremes and is a better
> prediction for average performance.
It's a flawed inference to conclude that "the average workload" will match an unweighted average of a set of intentionally disparate workloads.
Furthermore, people less & less buy hardware for "the average workload".
> in LLVM compilation Altra Max beats Altra by ~20% and is pretty much equal to the 7763.
You can't just cherry-pick the best results of each memory configuration. If you're going to deal in aggregates, then you need to aggregate the results per-configuration.
> So in real world tests EPYC's huge caches don't help nearly as much as the gcc_r subtest suggests.
As a matter of fact, the monolithic vs. quadrant results would argue the opposite, in your chosen example of LLVM. Furthermore, what qualifies LLVM compilation as more "real world" than the gcc test?
schujj07 - Monday, October 11, 2021 - link
"The Altra Max wins the more useful critical-jOPS benchmark by over 30%"What are you talking about? In both critical-jOPS & max-jOPS the 2s 7763 is on the top of the chart. We cannot try to extrapolate possible performance on the Altra Max due to "Unfortunately, trading in one issue with another, we ran into other issues on the 2-socket test scenario where the test ran into issues at large thread counts. The 2S Q80-33 figures here only stresses 130 cores, while I wasn’t able at all to get 2S M128-30 figures at reasonable core counts, so I completely omitted results here."
Per-core performance matters a lot. There are A LOT of programs, especially databases, that are licensed on a per core metric. This means I need 8 cores of Altra Max to equal the performance I get from an Epyc 4c that will kill my licensing cost. Those added cores could easily double the license cost and those license are often times MUCH more expensive than the server itself. It is obvious you don't work in industry as this is common knowledge.
Overall the Altra Max is interesting but nothing more than that. It won't be a player in industry until the per core performance is at least double what it currently is and there is enterprise software able to take advantage of it. Basically Altra Max is like IBM Power and that is niche at best.
Wilco1 - Tuesday, October 12, 2021 - link
Altra Max is still at the top for 1S critical-jOPS - that's not invalidated by missing 2S results.If you worked in the industry, you would know that per-core licenses have a multiplier based on CPU type to level out performance differences. In cases where per-core performance really matters and you completely disable SMT (for example for high-frequency trading), you would not consider these many-core servers at all but get 8 or 16 core CPUs with significantly higher bandwidth, cache and power per core.
It seems you misunderstand the target market completely. You probably also call Graviton 2 a niche eventhough it is already a significant percentage of AWS and growing fast. And that with just 64 cores and far lower per-core performance than Altra...
schujj07 - Tuesday, October 12, 2021 - link
How about we do some math instead. Compared to the 1S 80 core, the 1S Max gets 42% better performance, THP disabled, or 30% better performance THP enabled for 60% more cores in your beloved critical-jOPS. Compare that to the Epyc 7763 which gets 105% better THP disabled and 102% THP enabled. Even the older 80C only adds 62% despite doubling its cores. Based on that alone best case scenario is the 2S Altra Max ties the Epyc 2S in critical-jOPS. Sure it is beating the 1S 7763 but it barely beats the 2S 7443 a 24c/48t CPU.I do work in industry as a VMware Admin. Unless you are running Oracle, most of these will be run on systems up to 32c/64t to max out your VMware license. If you have specific needs you can get the higher frequency parts that also are up to 32c AMD or 28c Intel. The difference in costs for Windows DataCenter for the core additional core licenses is saved by reducing the number of physical hosts. What software has "per-core licenses have a multiplier based on CPU type to level out performance differences?" That sounds like they are going to charge your X for Xeon Scalable Gen 1 but Y for Gen 2 and Z for AMD. That doesn't happen. MS SQL Server charges per core with a base license of 4 cores. Now if I need 8 cores on the Altra Max to equal the performance of an Epyc at 4 cores I have doubled my license cost.
Overall ARM with under 10% total market share IS a niche player. They need to get to the same per core performance & have software available for it to be an actual alternative. Until that happens companies will play around with it but nothing serious in the data center environment.
mode_13h - Tuesday, October 12, 2021 - link
> You probably also call Graviton 2 a nicheDon't put words in people's mouths. If you want to know whether @schujj07 considers it a niche, you can certainly ask.
Kangal - Thursday, October 7, 2021 - link
This is basically a 3GHz Cortex-A76 (Neoverse N1), running in a 128-core tandem, and built with a more efficient/expensive Monolithic Socket based on TSMC's 7nm node. Sounds neat.I enjoyed seeing the older generation which was basically a 2GHz Cortex-A73, running in 64-core tandem, and built on TSMC's 16nm node. Was quiet value-for-money, at least in its time.
Seems like this new version is giving Intel's Core-i, decent competition in the single-threaded work. Since Intel is having some issues with their own node, and can't clock too high. Whilst AMD has a clear advantage here. When it comes to total/multi-threaded performance, ARM wins through sheer grunt of all those extra cores. Overall, it is a competitive choice for today and the next few years.
What will be interesting is when they bump it up to the Cortex-A78 (Neoverse V1) and use something like TSMC's 5nm node which should bring it to full-parity on the single-threaded performance against Intel. Or to the next best thing, ARM v9, using the Cortex-X2 (Neoverse N2) on the same TSMC 5nm node. But I share my previous concerns that the first-generation of (USA) ARM v9 is going to be quiet disappointing, but I'm optimistic about the (European) second-generation. I think then we should see more tangible benefits, when combined with the TSMC 3nm node, which should bring it on parity to AMD's cores on the single-threaded characteristic. Exciting times ahead. And yes, I know I am over-simplifying things here.
SarahKerrigan - Thursday, October 7, 2021 - link
Previous Ampere parts weren't 64-core, 2GHz, or Cortex-A73. They were a custom (and bad) core, 32 per socket, at 3.3GHz.Neoverse V1 is based on the Cortex-X1, not the Cortex-A78. Neoverse N2 is based on the Cortex-A710, not the Cortex-X2.
Kangal - Friday, October 8, 2021 - link
Sorry, by "older generation" I was talking about the Amazon Graviton one, not the previous Ampere Version.The proper upgrade from the Cortex-A76 is the Cortex-A78.
The Cortex-A78 is the base micro-architecture, with the Cortex-X1 being a slightly modified derivative of it, and the Neoverse-V1 is a further slightly modified version of that. That's why I worded it in that way. Whilst ARM claims a divergence between the Cortex-A710, Cortex-X2, and Neoverse-N2... I think we will end up seeing them much more closer in-common than different.
SarahKerrigan - Friday, October 8, 2021 - link
The Graviton1 was 16 Cortex-A72 at 2.3GHz.