AnandTech Home IT Portal Home Increase Font Size Decrease Font Size Change Page Size
VMmark Scores Investigated: should VMmark be part of your hardware decisions?
VMmark Scores Investigated: should VMmark be part of your hardware decisions?
Date: May 8th, 2009
Topic: IT Computing
Manufacturer: VMware
Author: Johan De Gelas
Buy the Cisco CSACSE-4.0-SWUP ver. 4.0 Secure
Blank
 MacMall $3,269.99
 ClubMac $2,899.99
 OnSale.com $2,899.99
 
 

Introduction

While the SPEC Virtualization Committee is slowly but steadily developing a new virtualization benchmark called "SPECvirt_sc2009", VMmark remains the only industry standard benchmark to compare hardware for virtualization as all the tier-one hardware vendors have published results for this benchmark. Why would you, the IT professional, care about yet another benchmark? Some professionals will probably point towards the lack of impact (and interest) that TPC and other industry standard benchmarks make when it comes to purchasing hardware decisions, as "performance is only a small part of the decision". We could not agree more when it comes to many industry benchmarks, but a virtualization benchmark is a special case.

We have said this before: virtualization is the killer application of the first decade of the 21st century. More than half of the servers sold today will end up being host for quite a few virtual machines. Even better, it is a market that is expanding even in these times of economic crisis. From the 591 IT professionals that took the Data Center Decisions 2008 Purchasing Intentions survey, only 2% indicated that the budget for virtualization related purchases would decrease, while 56% indicated that this budget would increase. That means scoring well in the VMmark benchmark is starting to get crucial for the server hardware vendors.

Higher virtualization performance means higher consolidation ratio, and thus translates in most cases in immediate cost savings: you need fewer servers, which allows you to save on energy, hardware costs, and manpower to maintain and install those servers. Performance is a much more important factor in the decision process when it comes to virtualizing the datacenter. So while a higher TPC or SPECjbb score will seldom result in happier users and cost savings, a higher VMmark score should almost always do that… in theory.

Understanding the VMmark Score   Next Page

 
  Index

Tools Share
Find lowest prices Find the lowest prices
Digg   del.icio.us   E-mail  
Print This Article Print this article  

23 Comments - Last by JohanAnandtech, 188 days ago
Username:
Password:
You're right by mlambert, 197 days ago
No one pays attention to this. It's all about whats available from HP in terms of blades & enclosures when it's time for your 3yr hardware refresh.

Right now it's BL2's the 495's. This is how corp IT works.

Reply
different approach needed by tynopik, 197 days ago
one problem with VMmark is that it tries to reduce a complex combination of variables to ONE NUMBER

that's great if everyone has the SAME WORKLOAD, but they don't

how about more focused benchmarks that stress ONE particular area of vm performance?

then people can looks at the numbers that most impact them

also it would be interesting to see where the gains are coming from

is nehalem better at virtualization simply because it's a faster cpu? or are the vm-specific enhancements making a difference? inquiring minds want to know

Reply
Exceptional Article by lopri, 196 days ago
Yet again from Johan. Johan never disappoints! I have just had a quick read, but I will take a thorough read later. Thank you much and I'd like the follow-up articles very much, too.

Reply
RE: Exceptional Article by JohanAnandtech, 195 days ago
Thanks for the compliment and especially the confidence!

- Johan

Reply
common sense by duploxxx, 196 days ago
Finally some good article to breach this new vmmark scores.

Although it is clear that the new Nehalem based system is better then current shanghai this is mostly due to the 3 mem controllers (which in the end provides more mem/cpu) and faster memory. The HT feature is the main VMmark whoop score cause here, it is already stated by many Vmware performance representatives that people should take care about the HT core as a real core in production, if you do this the performance will get bad just as previous HT, although the ESX sw is no much more aware of this feature (esx 3.5u4 and esx4), but is seems like the vmmark is not able to see the difference since there is not enough load on the system.

All other features are now equal while shanghai switching time was way better then harpertown the nehalem is more or less equal, also the ept/npt or rapid V or whatever you want to call it is now implemented.

so a final vmmark performance score you stated around 16-17 sounds very reasonable.

the performance enhancements in esx4 are not really for HT rather the core coherency features like vmware wants to call this, iommu which will be first introduced by amd istanbul and most important the paravirtualized scsi driver and off course more cpu/vm and a lot of memory, scheduling improvement.....etc....

Perhaps you should contact there are aware of this VMmark real world difference and are working on a new version.

Reply
How many of us are running more than 50 to 100 VMs? by noxipoo, 196 days ago

Plenty of companies have old servers that doesn't need much. NT4 to 2000 servers can easily range to those numbers at most corporations.

Reply
RE: How many of us are running more than 50 to 100 VMs? by JohanAnandtech, 196 days ago
Possible, but still an exception. Windows 2000 and NT4 servers have become a minority, probably less than 5% of the installed base.

And you are probably not too concerned about CPU performance when consolidating those servers.

Reply
Belgian persuasion by tshen83, 196 days ago
So,I take it that you want to discredit VMmark as being a relevant benchmark for virtualization?

You know VMmark isn't the only benchmark that says Nehalem is twice as efficient in performance/watt than Shanghais right?

In the last paragraph, you said "give us a few more days". To do what? To selectively choose a few benchmarks that show that Shanghai is a better CPU for virtualization workloads? Good luck with that.

Sometimes I want to find a ruler and measure just how deep you stuck your head up AMD's rear end. Sometimes I also wonder why a Belgian is so freaking adamant about AMD. Anand got too cheap I guess to outsource an important job to Belgium I guess.

Reply
RE: Belgian persuasion by whatthehey, 196 days ago
Or perhaps he just has some information on a new virtualization benchmark suite, which may or may not show Shanghai in a better light.

I think it's pretty easy to conclude that the two year old design of VMmark is aging and not as relevant as when it first came out (if it was even truly relevant then). So let's wait a few more days, eh?

Sometimes I want to pull out a ruler to measure just how far up Intel's ass tshen83 has shoved his head so that he can't even consider any viewpoint that doesn't state that Intel is unequivocally the best. Seriously, look at any AMD or Intel article, and he's there espousing the virtues of Intel and trashing everything AMD does. It's not all black and white, dude... except when you get paid by Intel to do what you do, of course.

Reply
RE: Belgian persuasion by Viditor, 194 days ago
My own guess is that tshen83 has become a 100% Intel ass, the 2 things have merged in this reality...:)

I would like to see how things compare on the larger boxes though. There are an awful lot of 4 and 8 way VM machines going out there right now...

Reply
Comments Page 1 of 3

Free Forrester Risk Management Report
Demystifying Enterprise Risk Management. Download Free With Registration.
DOWNLOAD vWire Today - FREE TRIAL
Take Control of Your Virtual Infrastructure. Manage VI Data & Prevent Problems.
Report Unlicensed Business Software Use
Earn Up to $1 Million by Reporting Unlicensed Software Use. Fill Out Our Form!
Download Microsoft Visual Studio ® Team System
Streamline Dev processes, Reduce time to market. Try Microsoft Visual Studio Team System, FREE!
Supermicro Barebone Servers
We Carry Everything Supermicro. Low Price, Top Service, FREE Shipping, and more.




Latest news by
DailyTech

 November 20, 2009

Blank
Blank
Blank
Blank
Blank
Blank
Blank
Blank
Blank

 November 19, 2009

Blank
Blank
Blank
Blank
Blank
Blank
Blank
Blank




pipeboost
Copyright © 1997-2009 AnandTech, Inc. All rights reserved. Terms, Conditions and Privacy Information.
Click Here for Advertising Information