Benchmark Overview

2017 CPU

For our review, we are implementing our fresh CPU testing benchmark suite, using new scripts developed specifically for this testing. This means that with a fresh OS install, we can configure the OS to be more consistent, install the new benchmarks, maintain software version consistency without random updates and start running the tests in under 5 minutes. After that it's a one button press to start an 8-10hr test (with a high-performance core) with nearly 100 relevant data points in the benchmarks given below. The tests cover a wide range of segments, some of which will be familiar but some of the tests are new to benchmarking in general, but still highly relevant for the markets they come from.

Our new CPU tests go through six main areas. We cover the Web (we've got an un-updateable version of Chrome 56), general system tests (opening tricky PDFs, emulation, brain simulation, AI, 2D image to 3D model conversion), rendering (ray tracing, modeling), encoding (compression, AES, h264 and HEVC), office based tests (PCMark and others), and our legacy tests, throwbacks from another generation of bad code but interesting to compare.

All of our benchmark results can also be found in our benchmark engine, Bench.

A side note on OS preparation. As we're using Windows 10, there's a large opportunity for something to come in and disrupt our testing. So our default strategy is multiple: disable the ability to update as much as possible, disable Windows Defender, uninstall OneDrive, disable Cortana as much as possible, implement the high performance mode in the power options, and disable the internal platform clock which can drift away from being accurate if the base frequency drifts (and thus the timing ends up inaccurate).

New Tests

SpecWPC v2.1 – A lot of the industry turn to SPEC to produce standard benchmarks suitable for various markets. The latest version of the workstation focused benchmark suite was released this year, and tackles six main areas with over 30 different benchmarks. These include compute, visualization, medical, oil and gas, finance, and typical workstation areas. For consistency we run all the tests (except IOMeter) on Windows 10, using an RX 460 graphics card at 1080p resolution with an MX200 SSD.

PCMark10 – We had several requests to include PCMark10 in our new testing suite. Normally we wait until a new benchmark has most of the problems ironed out, however our initial performance scaling metrics show that PCMark10 is basically there already. The extended suite covers ‘Essential, Productivity and Creativity’ benchmarks such as GIMP, Blender, video editing, conferencing, complex spreadsheets and other tests. We use the subtest values as well as the gaming physics result.

Agisoft PhotoScan 1.3.3 – Again, requests to use a more updated version of Photoscan were also coming through the inbox. Over the older version, Photoscan includes various throughput enhancements to each of the core points of the algorithm. Agisoft also gave us a new larger set of more detailed test images to generate our 3D models, giving a longer benchmark (but results are not comparable to the old data). We’ve run this benchmark on about a dozen CPUs ready for this review.

Office / Professional Tests

PCMark8
Chromium Compile (v56)

Rendering Tests

Corona 1.3
Blender 2.78
LuxMark v3.1 CPU C++
LuxMark v3.1 CPU OpenCL
POV-Ray 3.7.1b4
Cinebench R15 ST
Cinebench R15 MT

Encoding Tests

7-Zip 9.2
WinRAR 5.40
AES Encoding (TrueCrypt 7.2)
HandBrake v1.0.2 x264 LQ
HandBrake v1.0.2 x264-HQ
HandBrake v1.0.2 HEVC-4K

System Tests

PDF Opening
FCAT
3DPM v2.1
Dolphin v5.0
DigiCortex v1.20
Agisoft PhotoScan v1.0

Legacy Tests

3DPM v1 ST / MT
x264 HD 3 Pass 1, Pass 2
Cinebench R11.5 ST / MT
Cinebench R10 ST / MT

A Note on CPU Gaming Tests (Repeat from Page 1)

I know a lot of our readers are gamers, and are interested in seeing how well (or poorly) these massive multi-core chips perform in the latest titles at the highest resolutions. Apologies to disappoint, but I am going to tackle the more traditional consumer tasks in a second review, and which will mean that gaming will be left for that review. For the users that have followed my reviews (and Twitter) of late, I am still having substantial issues with my X299 test beds on the gaming results, with Skylake-X massively underperforming where I would expect a much higher result.

After having to dedicate recent time to business trips (Hot Chips, IFA) as well as other releases (Threadripper), I managed to sit down in the two weeks between trips to figure what exactly what was going on. I ended up throwing out the two X299 pre-launch engineering samples I was using for the Skylake-X testing, and I received a new retail motherboard only a few days before this review.  This still has some issues that I spent time trying to debug, which I think are related to how turbo is implemented, which could either be Intel related or BIOS specific.

To cause insult to injury to everyone who wants to see this data, I have jumped on a plane to travel half-way around the world for a business trip during the week of this launch, which leaves the current results inconclusive. I have reached out to the two other motherboard vendors that I haven’t received boards from; just in case the issue I seem to be having is vendor specific. If I ever find out what this issue is, then I will write it up, along with a full Skylake-X gaming suite. It will have to wait to mid-late October, due to other content (and more pre-booked event travel).

Test Bed and Setup Benchmarking Performance: SPECwpc v2.1
Comments Locked

152 Comments

View All Comments

  • Spunjji - Monday, September 25, 2017 - link

    Super relevant, because they indicate how badly thermally limited the CPU is - which is hella good info to have if you're, say, considering delidding a $1999 processor because the manufacturer used toothpaste under the IHS.
  • tricomp - Monday, September 25, 2017 - link

    Poor AMD...
    No chance they are going to supply (even more) cpu's demand after posting this article..
    I am trying to purchase at list 7 systems for my customers in my country but there's nowhere I can find them beasts here..
  • iwod - Monday, September 25, 2017 - link

    I wish someone could do an article on that too. GF doesn't seems to be the limitation here. GF, should in theory more then enough capacity in their Fab 8 for AMD. Unless GF have some other big customers, otherwise AMD should really be bumping out as much unit as possible.
  • Atom11 - Monday, September 25, 2017 - link

    Can we please see one test (!), if you could possible manage, that shows the advantage of AVX-512 in compare to AVX2 when doing:

    1.) matrix multiply
    2.) FFT
    3.) convolution
  • ZeDestructor - Monday, September 25, 2017 - link

    Give us a comparison to AVX1 and SSE4 too!
  • Gothmoth - Monday, September 25, 2017 - link

    threadripper delivers 80+% of the perfromance for less than 50% of the price.... you don´t have to be a genius to see what the better deal is (price germany: TR 1950x = 950 euro, 7890xe =2300 euro)
  • Spunjji - Monday, September 25, 2017 - link

    Don't let that stop them equivocating about how companies who need that power yet somehow have no need for ECC don't care about cost because something something software TCO blah blah.
  • spdragoo - Monday, September 25, 2017 - link

    I'm trying really, really hard to think of a company that, at some point or another, doesn't say, "Equipment X may outperform Equipment Y, but the extra cost to buy Equipment X is too much, we'll just make-do with Y instead." Especially since 100% of companies have a limit on their budgets.

    What's that, you say? Multi-billion dollar corporations don't have to worry about the money they spend? Someone apparently didn't pay attention in their Econ 200 class, or their Introduction to Accounting coursework.

    By definition, every business has a *finite* amount of money they can spend, based on a) how much money they collect from their customers, b) how much they can recoup on the sale of assets (tangible or intangible), & c) how much they can get from "other sources" (mostly bank loans or by selling stock shares, or sometimes government grants, but you might find the occasional situation where a generous benefactor just bequeaths money to a company...but I doubt you'll even see that happen to 1% of the companies out there -- & no, venture capitalists pouring money into a company is *not* a situation where they "give the money away", they're getting something for their money, usually stock shares or guarantees of repayment of the loans). Of that money, some of it is earmarked for employee compensation (not just the executives, but the office drones & lower-level employees that do 99% of the actual work), some of it goes towards taxes, some of it pays for rental payments, some for loan payments, some for utilities (telephone, Internet, electricity, gas, water, etc.), some of it may get set aside for "emergencies", some gets earmarked for dividends to the shareholders, etc. That means that a (relatively) small portion is set aside for "equipment replacement". Now, if the company is lucky, the lion's share of that budget is for IT-related equipment...but that covers more than just the office drones' machines, that covers everything: server racks, storage services, cloud vendor payments, etc.

    And that is where the price comes into play. For probably 90% of office users out there, not only is Threadripper an overpowered product, so are these products. Heck, we're in the middle of an upgrade from Windows 7 to Windows 10, & they're taking the opportunity to replace our old Sandy Bridge i5 machines with Skylake i7 machines. Sure, they're running faster now...but the main reason they're running faster is because we went from 32-bit Windows to 64-bit Windows, so our PCs now have 8GB of RAM instead of 4GB. That helps with our workload...which primarily revolves around MS Office & using browsers to access & modify a number of massive databases. Having an 8C/16T CPU, let alone a 16C/32T CPU, wouldn't provide any boost for us, since the primary slowdown is on the server side.

    These are going to be expensive systems for specialized purposes...& those individual companies are going to look at their budgets very closely, as well as the performance benchmarks, before deciding to purchase these systems. Sure, they may hold the performance crown...but not by that big of a margin, & especially when compared to the margin that gives them the "most expensive price" crown.
  • BrokenCrayons - Monday, September 25, 2017 - link

    Human labor is more expensive than hardware. The 20% additional performance for $1000 more can be earned back quickly by the increased productivity of your workforce (assuming your management staff is effective enough to keep the employees gainfully employed of course and that's certainly not always the case).
  • vladx - Tuesday, September 26, 2017 - link

    Indeed the difference in price is pretty much negligible in a professional setting.

Log in

Don't have an account? Sign up now