ARM Challenging Intel in the Server Market: An Overview
by Johan De Gelas on December 16, 2014 10:00 AM ESTCavium Thunder-X
A few months ago, we talked briefly with the people of Cavium. Cavium is specialized in designing MIPS SoCs that enable intelligent networking, communications, storage, video, and security applications. The picture below sums it all up: present and future.
Cavium's "Thunder Project" started from Cavium's existing Octeon III network SoC, the CN78xx. Cavium's bread and butter has been integrating high speed network capabilities in SoCs, so you will be able to choose between SoCs that have 100 Gbit Ethernet and 10GBit Ethernet. PCI-Express roots and multiple SATA ports are all integrated. There is no doubt that Cavium can design a highly integrated feature-rich SoC, but what about the processing core?
The MIPS cores inside the Octeon are much simpler – dual-issue in-order – but also much smaller and need very little power compared to a typical server core. Four (28nm) MIPS cores can fit in the space of one (32nm) Sandy Bridge core.
Replace the MIPS decoders with ARMv8 decoders and you are almost there. However, while the Cavium Thunder-X is definitely not made to run SAP, server workloads are bit more demanding than network processing, so Cavium needed to beef up the Octeon cores. The new Thunder-X cores are still dual-issue, but they're now out-of-order instead of in-order, and the pipeline length has been increased from eight to nine stages to allow for higher clocks. Each core has a 78KB L1 Instruction cache and a 32KB data cache.
The 37-way 78KB L1 I cache is certainly odd, but it might be more than just "network processor heritage". Our own testing and a few academic studies have shown that scale-out workloads such as memcached have a higher than normal (meaning the typical SPECIntRate2006 characterization) I-cache miss rate. The reason is that these applications run a lot of kernel code, and more specifically the code of the network stack. As a result, the software footprint is much higher than expected.
Another reason why we believe Cavium has done it's homework is the fact that more die area is spent on cores (up to 48) than on large caches; an L3 cache is nowhere to be found. The Thunder-X has only one centralized relatively low latency 16MB L2 cache running at full core speed. A lot of academic studies have confirmed that a large L3 cache is a waste of transistors for scale-out workloads. Besides the most used instructions that reside in the I-cache, there is a huge amount of less frequently used kernel code that does not fit in an L3 cache. In other words, an L3 cache just adds more latency to requests that missed the L1 cache and that will end up in the DRAM anyway. That is also the reason why Cavium made sure that a beefy memory controller is available: the Thunder-X comes with four DDR3/4 72-bit memory controllers and it currently supports the fastest DRAM available for servers: DDR4-2133.
On the flip side, having 48 cores with a relatively small 32KB D-cache that access one centralized 16MB L2 cache also means that the Thunder-X is less suited for some "traditional" server workloads such as SQL databases. So a Thunder-X core is simpler and probably quite a bit weaker than an ARM Cortex-A57 in some ways, let alone an X-Gene core. The fact that the Thunder-X spends a lot less transistors on cache than on cores clearly indicates that it is targeting other workloads. Single-threaded performance is likely to be lower than that of the AMD Seattle and X-Gene, but it could be close enough: the Thunder-X will run at 2.5GHz, courtesy of Global Foundries' 28nm process technology. Cavium is claiming that even the top SKU will keep the TDP below 100W.
There is more. The Thunder-X uses Cavium's proprietary Coherent Processor Interconnect (CCPI) and can thus work in a dual socket NUMA configuration. As a result, a Thunder-X based server can have up to 96 cores and is capable of supporting 1TB of memory, 512GB per socket. Multiple 10/40GBE, PCIe Root Complex, and SATA controllers are integrated in the SoC. Depending on SKU, TCP/IP Sec offload and SSL accelerators are also integrated.
The recent launch of Cavium's Thunder-X SKUs make it clear that Cavium is trying to compete with the venerable Xeon E5 in some niche but large markets:
- ThunderX_CP: For cloud compute workloads such as public and private clouds, web caching, web serving, search, and social media data analytics.
- ThunderX_ST: For cloud storage, big data, and distributed databases.
- TunderX_NT: For telecom/NFV server and embedded networking applications.
- ThunderX_SC: For secure computing applications
Considering Cavium's background and expertise, it is pretty obvious that ThunderX_NT and SC should be very capable challengers to the Xeon E5 (and Xeon-D), but only a thorough review will tell how well the ThunderX_CP will do. One of the strongest points of Calxeda was the highly integrated fabric that lowered the total power consumption and network latency of such a server cluster. Just like AMD/Seamicro, Cavium is well positioned to make sure that the Thunder-X based server clusters also have this high level of network/compute integration.
78 Comments
View All Comments
JohanAnandtech - Tuesday, December 16, 2014 - link
Did you miss this page?http://www.anandtech.com/show/8776/arm-challinging...
The software ecosystem is developing...there is no indication that this will stop soon.
Kevin G - Wednesday, December 17, 2014 - link
The LAMP stack is there and can easily give ARM a foot hold. Scaling up they'll need vendors like Oracle to port key applications. ARM will also need to enhance there RAS to be production capable with that software.Samus - Tuesday, December 16, 2014 - link
Johan,You need to review the compatibility of the Xeon E3's. They actually work in just about any Intel 80 or 90-series board. I have an E3-1230v3 in an Asus ITX H87 on the PC I'm currently typing on.
A C220 chipset is NOT required.
JohanAnandtech - Tuesday, December 16, 2014 - link
you are right :-).By "Xeon E3 needs C220" I meant that you need to add that part to calculate the power consumption per node. And the E3 needs it to support ECC RAM.
eanazag - Tuesday, December 16, 2014 - link
Ubuntu's ARM version OS is a big deal. I believe the fact that MS had been dragging on with supporting RT was in fact to have something to port to the server side. Even though RT is mostly a dud at first, it could still be sensible and sell in a server config.I'm waiting for AMD to finally sell the ARM chip in the channel so I can throw a mobo with it together. If it has 10GbE I would be all over it.
rootheday3 - Tuesday, December 16, 2014 - link
Intel also has Rangeley soc which includes crypto block for comms usagewintermute000 - Tuesday, December 16, 2014 - link
"What if I need massive amounts of memory but moderate processing power? The Xeon E3 only supports 32GB."Thousands of techs labbing away @ home nod sagely in agreement. Right now our choices are to scale horizontally or live with loud jet-engine ex-enterprise gear, because I can't get 64gb of RAM into a whitebox.
wintermute000 - Tuesday, December 16, 2014 - link
Clarification: a whitebox that I can afford i.e. not a Xeon E5. lolbeginner99 - Wednesday, December 17, 2014 - link
What kind of servers use tons of RAM and little processing power? Right, memcached and similar stuff. But let's be honest. That is still a niche market given the total server market. Most servers are just standard multipurpose servers running some company internal low-traffic (web) application. They don't need memcached. Memcached is for huge internet deployments and let's be honest that in itself is niche.I work in a 10'000 people company and I would bet you $1000 we have 0 memcached servers. I don't really know except for the lack of performance in core apps and the questionable competency of our IT.
bobbozzo - Wednesday, December 17, 2014 - link
VM servers.And ZFS-filesystem storage (NAS/SAN) servers. e.g. FreeNAS. Add much more RAM if using DeDup.