Overview

So let's summarize what we have seen so far. First we look at system power.


At the system level, the power savings of the Balanced power plan are pretty disappointing. Especially when benchmarking with two threads, we would have expected better power savings. Of course, we tested with only one CPU. The power savings should be better as you add more CPUs to the server. On the X3470 power savings are better, but the reason is not so much "SpeedStep" but more the fact that this turns off Turbo Boost. The most spectacular power savings cannot be seen in this picture: the automatic power and clock gating. As we have shown in this article, power and clock gating happen in both power plans and are responsible for some very significant savings, especially on the "Nehalem Lynnfield" Xeons. SpeedStep and PowerNow! are no longer very impressive for the following reasons:

  • Clock gating and "deep sleep" core C-states already save a lot of power
  • They are limited to frequency scaling; voltage scaling is only possible if all cores are running at a low P-state

In the case of Intel, frequency scaling is demoted to a pretty insignificant role: it is more important to power gate cores than to clock them down to a lower P-state.


For AMD, P-state changes are still important, but their effect is dubious: performance is up to 20% lower. Even the specific AMD driver in Windows 2008 (amdppm.sys) is not capable of working optimally under low load. Cores frequently stay at 800MHz too long or only get to 1400MHz before a thread is moved to another core. Ideally our two thread benchmark should get at least two CPU cores to quickly ramp up to 2.6GHz and stay there, but that's not what happened in our tests.

We'll make a simple performance/watt calculation by multiplying the performance numbers by 1000 and dividing them by our power numbers.


Look at the green bars of the AMD Opteron processors: performance/watt is clearly lower when running in Balanced mode. This is of course not the case when we run four threads on the quad-core and six threads on the six-core Opteron. In that case the operating system gets very few chances to drop the power. Still, it is important to note that PowerNow! results in a significant performance loss, a performance loss that is not justified by the meager power savings. The result is that the six-core Opteron in Performance mode offers the best performance/watt ratio when we focus on the Opteron family. It is a shame that the Windows 2008 CPU driver does not adapt better: the six-core Opteron is pretty competitive in performance/watt.

The situation is a lot more complex in the Intel Xeon family. With low thread counts, the Xeon is capable of using Turbo Boost. From a "system power" perspective, power is only a bit higher, while performance goes up by a third. But with low TDPs, there is little wiggle room, and Turbo Boost is quickly put out of action. With more than two threads, the L3426 never clocks above 1.86GHz. With "normal" Xeons, there is no significant difference between the two power plans.

More Performance Please! Initial Conclusions
Comments Locked

35 Comments

View All Comments

  • JohanAnandtech - Tuesday, January 19, 2010 - link

    Well, Oracle has a few downsides when it comes to this kind of testing. It is not very popular in the smaller and medium business AFAIK (our main target), and we still haven't worked out why it performs much worse on Linux than on Windows. So chosing Oracle is a sure way to make the projecttime explode...IMHO.
  • ChristopherRice - Thursday, January 21, 2010 - link

    Works worse on Linux then windows? You have a setup issue likely with the kernel parameters or within oracle itself. I actually don't know of any enterprise location that uses oracle on windows anymore. "Generally all Rhel4/Rhel5/Sun".
  • TeXWiller - Monday, January 18, 2010 - link

    The 34xx series supports four quad rank modules, giving today a maximum supported amount of 32GB per CPU (and board). The 24GB limit is that of the three channel controller with unbuffered memory modules.
  • pablo906 - Monday, January 18, 2010 - link

    I love Johan's articles. I think this has some implications in how virtualization solutions may be the most cost effective. When you're running at 75% capacity on every server I think the AMD solution could have possibly become more attractive. I think I'm going to have to do some independent testin in my datacenter with this.

    I'd like to mention that focusing on VMWare is a disservice to Vt technology as a whole. It would be like not having benchmarked the K6-3+ just because P2's and Celerons were the mainstream and SS7 boards weren't quite up to par. There are situations, primarily virtualizing Linux, where Citrix XenServer is a better solution. Also many people who are buying Server '08 licenses are getting Hyper-V licenses bundled in for "free."

    I've known several IT Directors in very large Health Care organization who are deploying a mixed Hyper-V XenServer environment because of the "integration" between the two. Many of the people I've talked to at events around the country are using this model for at least part of the Virtualization deployments. I believe it would be important to publish to the industry what kind of performance you can expect from deployments.

    You can do some really interesting HomeBrew SAN deployments with OpenFiler or OpeniSCSI that can compete with the performance of EMC, Clarion, NetApp, LeftHand, etc. NFS deployments I've found can bring you better performance and manageability. I would love to see some articles about the strengths and weaknesses of the storage subsystem used and how it affects each type of deployment. I would absolutely be willing to devote some datacenter time and experience with helping put something like this together.

    I think this article really lends itself well into tieing with the Virtualization talks and I would love to see more comments on what you think this means to someone with a small, medium, and large datacenter.
  • maveric7911 - Tuesday, January 19, 2010 - link

    I'd personally prefer to see kvm over xenserver. Even redhat is ditching xen for kvm. In the environments I work in, xen is actually being decommissioned for VMware.
  • JohanAnandtech - Tuesday, January 19, 2010 - link

    I can see the theoretical reasons why some people are excited about KVM, but I still don't see the practical ones. Who is using this in production? Getting Xen, VMware or Hyper-V do their job is pretty easy, KVM does not seem to be even close to being beta. It is hard to get working, and it nowhere near to Xen when it comes to reliabilty. Admitted, those are our first impressions, but we are no virtualization rookies.

    Why do you prefer KVM?
  • VJ - Wednesday, January 20, 2010 - link

    "It is hard to get working, and it nowhere near to Xen when it comes to reliabilty. "

    I found Xen (separate kernel boot at the time) more difficult to work with than KVM (kernel module) so I'm thinking that the particular (host) platform you're using (windows?) may be geared towards one platform.

    If you had to set it up yourself then that may explain reliability issues you've had?

    On Fedora linux, it shouldn't be more difficult than Xen.
  • Toadster - Monday, January 18, 2010 - link

    One of the new technologies released with Xeon 5500 (Nehalem) is Intel Intelligent Power Node Manager which controls P/T states within the server CPU. This is a good article on existing P/C states, but will you guys be doing a review of newer control technologies as well?

    http://communities.intel.com/community/openportit/...">http://communities.intel.com/community/...r-intel-...
  • JohanAnandtech - Tuesday, January 19, 2010 - link

    I don't think it is "newer". Going to C6 for idle cores is less than a year old remember :-).

    It seems to be a sort of manager which monitors the electrical input (PDU based?) and then lowers the p-states to keep the power at certain level. Did I miss something? (quickly glanced)

    I think personally that HP is more onto something by capping the power inside their server management software. But I still have to evaluate both. We will look into that.
  • n0nsense - Monday, January 18, 2010 - link

    May be i missed something in the article, but from what I see at home C2Q (and C2D) can manage frequencies per core.
    i'm not sure it is possible under Windows, but in Linux it just works this way. You can actually see each core at its own frequency.
    Moreover, you can select for each core which frequency it should run.

Log in

Don't have an account? Sign up now