backtop


Print 30 comment(s) - last by aalaardb.. on Dec 12 at 4:38 PM

SPEC has introduced a new power-efficiency measurement standard for CPU architectures. The winner? The consumer

SPEC, the Standard Performance Evaluation Corporation, is a non-profit corporation whose goal is to provide "fair, impartial and meaningful benchmarks for computers."

With the latest excitement and fuss over AMD's K10 power usage figures, SPEC conveniently announced a new set of benchmarks to compare power efficiency of CPUs based on an independent and unbiased standard.  The standard holds the promise of a level playing field for CPU manufacturers to test the quality of their products on. 

SPEC's new solution is dubbed the SPECpower_ssj2008.  The test, which is to be run on a server, takes the chip through a series of graduated descending workloads, starting at 100% usage and dropping by 10% per set period of time until idle is reached.  The total processor throughputs for each segment are first summed and then divided by the sum of the average power consumed for each segment.  The final product is a performance measure called "ssj_opps/Watt."

The test is promising, as it takes into account that workloads on a server vary greatly, and calculates efficiency based on a sensible averaging approach.  With AMD championing their new ACP standard, and Intel still advertising its modified TDP standard, finding understandable and accurate comparative power-performance figures has become a rather confusing business for the average customer.

A unified power-performance standard could certainly change that.  It provides Intel and AMD with a chance to truly put their processors' power prowess to the test in an unbiased benchmark -- if they are willing.

This test is geared towards measuring power-efficiency across a wide pattern of usage, and probably would not be very applicable to providing information on the specific cooling solutions needed for a particular processor.  For cooling evaluation, traditional TDP measurements would be of greatest use, as they test the processor power under a steady 100% real load scenario.

In terms of power-efficiency, though, SPEC's new system hopes to bring honesty, integrity, a renewed sense of competition to the processor market.  At a lowly cost of $1,600 for the software, Intel and AMD should be eager to put this independent benchmarking suite to work.


Comments     Threshold


This article is over a month old, voting and posting comments is disabled

Not very useful because...
By aalaardb on 12/11/2007 7:05:58 PM , Rating: 2
Well for 2 reasons really. First and foremost, this compares apples to oranges.
quote:
It provides Intel and AMD with a chance to truly put their processors' power prowess to the test in an unbiased benchmark

SPECpower_ssj2008 compares system to system, not CPU to CPU. In the link provided by the other poster, there's just 1 AMD system, running at 2.0ghz, and with 16gb of DDR2-667 memory. This particular system also has a 400gb hard drive (which I imagine takes more energy than a, say, 80gb hard drive), 2 700 watt PSUs rated at whatever efficiency I don't know, etc etc. It's then compared with several different Intel systems, and for this arguement I picked the one with the highest rating (and incidently we are comparing systems with a 4 month gap in hardware availability - this Intel system isn't even available yet).

This Intel runs at 3.0ghz, has 16gb of a different type and speed of memory (even a different number of memory modules), 1 1200 watt PSU, and an 80gb hard drive. Now, when you have this different memory you will have different energy requirements and different performance. Same with each and every component. No, I'm not saying let's test the CPUs all on the same MB with the same memory - that would be impossible. I'm saying now that we are comparing systems, comparing CPUs is not possible.

The second reason this is not very useful is it is comparing apple/cranberry cider to peach/mango smoothie. What I mean by that is not only are we measuring the power as in energy requirements, but also the power as in performance. The resulting SPEC number is performance/energy. Again comparing AMD's only against Intel's best, Intel beats AMD just barely by using 269 watts at 100% load against 276. But then Intel gets a bonus because it does 3x the work for those watts, giving it over 3x the score. If we had a test system and swapped in one Intel CPU for another and then compared performance/watt it would be a much fairer comparison.

So basically the only use I see out of these numbers is helping a system administrator pick between Intel by server manufacturer X and Intel by server manufacterer Y. In this case I see the system administrator as already having decided between AMD and Intel - though it could be used to help decide this - and wanting to find the best performance per watt per server cost.

The average consumer is not "the winner" as this benchmark of servers has little to do with the average consumer.
This won't solve the CPU TDP arguement either. CPU TDP is CPU's energy used at load, load to be determined. This standard is system's energy used as a function of system's performance.




RE: Not very useful because...
By JumpingJack on 12/11/2007 11:36:10 PM , Rating: 2
Of course it is useful, it is the first step to initiating a industry acceptable standard for establish power efficiency independent of this TDP (not a consumption), APC, and odd bickering where both claim the best in class energy efficient solution.

More in detail...there are a number of inaccuracies in your post as well as flaws (more like mis-understandings of SPEC) in your argument.

Point 1 ...
Your first point it is that is not apples to apples (or as you say, apples to organes), this is more or less a broad dismissal of SPEC in general as no spec score produces a apples to oranges comparision, be it SPEC2006, JBB2005, etc. what have you nor is it specific to any one component of the system.

This is because SPEC is not a consumer level benchmark suite, it is used extensively in the server space to compare platforms which enterprise uses to make procurement decisions, as such it is not CPU specific rather CPU inspecific and more aptly called a platform test. This is why exact details of not only HW but SW and compiler options are required so that the performance promise can be readily replicated and indicative of intended use for the platform.

Nonetheless, this new power bench will most certainly be adopted as the industry standard as many of the key players have purchased licenses (HP was #3, Intel was #13, and AMD was #39 I believe) and submitted the initial round of scores.

Now the inaccuracies in your post on point 1 -- the 1 AMD system running is there because AMD submitted this score, this is a new bench so it has yet to be populated with data, nonetheless .. it is not 2.0 GHz, it is running at 2.4 Ghz, http://www.spec.org/power_ssj2008/results/res2007q... you can see it in the config table.

In terms of efficiency of the PSU as you try to argue, it does not matter... this is a platform level bench, measuring platform level power efficiency, thus, it does -- in fact -- measure what is actually important -- what you pay at the pump, all items accounted for -- power at the socket, which is what determines the electric bill. As the bench database evolves there is now a compelling drive to minimize power on all components by any vendor submitting scores for relative comparision -- this can only be a good thing.

Next, you make a point of Intel running at 3.0 Ghz.. actually, the database has more Intel systems running AMD just happens to have the dual core HE part which is holding second to last place... but again, as this is platform level, a person interested in the best performance/watt will want to see all the platforms -- again, fulfilling it's function. Barcelona data will slowly appear, this will change the landscape to be sure.
quote:
T I'm saying now that we are comparing systems, comparing CPUs is not possible
Precisely, that is the intent of the benchmark.

quote:

(and incidently we are comparing systems with a 4 month gap in hardware availability - this Intel system isn't even available yet).
another inaccuracy, Intel launched 45 nm quad cores several weeks ago, HP who is the manufacturer of the PLATFORM is still in development and lists HW availability of Jan. 08 (not 4 months from now).
http://www.spec.org/power_ssj2008/results/res2007q...
In fact, spec has strict reporting rules to avoid this very scenario that you are suggesting... to see it in action, look at the treatment of a barcelona score when it became apparent that it would not ship within the 90 day of submission guideline:
http://www.spec.org/cpu2006/results/res2007q3/cpu2...

Point 2 ...
quote:
Again comparing AMD's only against Intel's best, Intel beats AMD just barely by using 269 watts at 100% load against 276.

No look at the database... there are also dual core 2 socket systems. You can compare against what ever you want... since it is platform level, the comparison will yield a perf/watt for the system an enterprise customer may buy:
http://www.spec.org/power_ssj2008/results/power_ss...

It is quite remarkable that Intel's best is a quad consuming less overall than AMD's energy efficient dual core :) ... nonetheless, there are quite a few different platforms to look through, if you wish choose the one with the closest price points (though in enterprise, price is secondary to performance and power).

quote:
But then Intel gets a bonus because it does 3x the work for those watts, giving it over 3x the score. If we had a test system and swapped in one Intel CPU for another and then compared performance/watt it would be a much fairer comparison

This is the advantage of quad core... swapping out CPUs until you get the answer you like is not 'fair'. Perahps you would prefer this comparision:
http://www.spec.org/power_ssj2008/results/res2007q...
A single two socket netburst core... pretty pitiful. Nonetheless, it would appear that you are labeling the SPEC bench as 'useless' because you don't like the results.

quote:
So basically the only use I see out of these numbers is helping a system administrator pick between Intel by server manufacturer X and Intel by server manufacterer Y. In this case I see the system administrator as already having decided between AMD and Intel - though it could be used to help decide this - and wanting to find the best performance per watt per server cost.

No... it helps the system administrator decide what X platform he wants if maximizing energy efficiency is his gig. Regardless of what processor is in it or who makes it, among other criteria.

Now I don't necessarily disagree that there this incarnation has it's draw backs.... but your 'points' above are not the reasons that should be used.

- One may argue that the numerator (performance) is not fairly distributed to represent a true working set.

- A second argument may be made that the code used to load the CPUs favors one CPU arch over another (Java has always been an Intel strength), the counter to that is that this is more representative of actual code base than FP or INT from the SPEC2006 suite (I do not know, I have not seen an analysis of what is most common).

There are a whole lists of problems, and most certainly there will be strong objections to it's usage as I doubt AMD will fair well in this benchmark across the board. In fact, from an AMD perspective, if I were AMD I may even want to try to establish my own standard before this one takes off ... :) hmmmmmm....


RE: Not very useful because...
By aalaardb on 12/12/2007 12:54:47 AM , Rating: 2
quote:
This is because SPEC is not a consumer level benchmark suite

Then you agree with me when I say that the byline assertion "SPEC has introduced a new power-efficiency measurement standard for CPU architectures. The winner? The consumer" is false.

Yup, I misquoted 2.4 as 2.0. I compared a 2.4 AMD to a 3.0 Intel not because I felt like giving Intel an advantage, but because I had to arbitrarily pick one to show that the configuration is different, and picking it arbitrarily shows they are all different. I just happened to pick a 3.0, for the given reason.

I never said that specific Intel server was due out in 4 months, I said there was a 4 month gap between availability. And there is, the AMD server is available now; it has been since September. September - January = 4 months. I originally mentioned availability not only because one might argue one shouldn't compare different generations, but also because if one can buy a certain server now and one can't buy another yet, then that may influence one's purchasing decision. Especially if it's a Christmas present.

quote:
Precisely, that is the intent of the benchmark
Yes, comparing system to system is the intent of the benchmark. However, the article said it was the intent of the benchmark to compare CPU to CPU, which it is not.

quote:
Again comparing AMD's only against Intel's best, Intel beats AMD just barely by using 269 watts at 100% load against 276.

You said no to this, but I fail to see how it is wrong. It's true that at 100% load the AMD server uses 276 and the best (as defined by SPEC) Intel server uses 269.

quote:
we had a test system and swapped in one Intel CPU for another and then compared performance/watt it would be a much fairer comparison

I don't think you got the point of this sentence. If we had an Intel MB with a fixed set of components, then compared Intel CPU A to Intel CPU B to Intel CPU C, swapping as we go, then that is fairer. In this Intel test system we have constructed, it is fair to compare performance/watt. But it's not fair to compare cross platform performance/watt. It is fair to compare cross platform performance, and it is fair to compare cross platform wattage, but not together.

What's wrong with comparing performance between systems on different platforms? Not much. Same thing for wattage. What's wrong with comparing power in performance AND power in consumption between systems on different platforms? The fact that one system's components may require more energy. See the 400gb / 80gb hard drive disparity. At the same speed I'd guess that the 400gb drive takes more energy to spin. Other disparities include the number of memory modules - 4x 4gb modules will take less energy than 8x 2gb modules, even if they are the same speed and overall size. (CPU to CPU comparisons have their own disparity, that of the NorthBridge, but hopefully that will fade).

I'm not arguing this won't be used by company and system administrator alike. I'm arguing it's not very useful for consumers. I'd go further and say it's of limited use to system administrators for both the reasons I gave and the reasons you gave.

In the end, I agree with you about AMD's standing regarding this benchmark. But I'm not a fanboy of either side, I used an Athlon 64 and am using a Core 2 Duo. I wrote then deleted that disclaimer in the last post because it was too long. Maybe I left a false impression in that regards.


RE: Not very useful because...
By aalaardb on 12/12/2007 1:32:24 AM , Rating: 2
Oh, I almost forgot.
quote:
In terms of efficiency of the PSU as you try to argue, it does not matter.


Yes, the PSU difference is relevant and quite possibly even prominent. To see why, let's take an Intel CPU, put it on a MB, and add some ram to it. To get this up and running we don't need a hard drive or keyboard or anything, but we do need a PSU. Company A takes this server configuration and adds an 80% efficient PSU, and Company B takes the same but adds a 90% efficient PSU. These 2 servers have the same performance, in fact quite obviously the architechure has the same capabilities in energy, but B will have a better SPEC score than A. Same thing if we happen to be comparing AMD and Intel. So actually that confirms that this shouldn't be used to compare apples to oranges, and confirms that this score isn't exact on the characteristics of an architechure and so won't mean much to the average consumer.


RE: Not very useful because...
By JumpingJack on 12/12/2007 2:57:07 AM , Rating: 1
You see the problem is you are not holding the bench in the correct prespective.... put a power meter between the plug and the computer and measure power... this represents what you will pay for the power bill. All components, even PSU efficiency accounted.

It does not matter what the overall PSU efficiency is, this is a 'DESIGN' consideration by the OEM who makes the box... if they shove a 25% efficient PSU in the box... so be it... they will pay the price in lost sales.

Same thing with RAM, if RAM is pulling too much power, the OEM pulls it out... sacrifying perfomrance, both cases reflected in the metric.

It is platform level...

An enterprise procurement will not care about whose processer is in the box in so much as they want a low power, high performing, stable system. This power metric gives them one part of the procurement equation.

You are taking a platform and trying to drill down to a single component... this is futile.


RE: Not very useful because...
By JumpingJack on 12/12/2007 3:20:34 AM , Rating: 1
On last thing... it is late but want to make sure it gets in...

quote:
Yes, comparing system to system is the intent of the benchmark. However, the article said it was the intent of the benchmark to compare CPU to CPU, which it is not.


This clarifies the point of your original post much better... yes, the article does indeed allude to a CPU level comparision for this new bench when it is the farthest thing from it.


RE: Not very useful because...
By aalaardb on 12/12/2007 4:38:15 PM , Rating: 2
Yes, I had thought because that was quoted right off the bat that that would be the understood intent of the post. You can question how much use it will be to a system administrator, but of use to the average consumer because it compares CPU to CPU... well no. All else is just details.


RE: Not very useful because...
By JumpingJack on 12/12/2007 3:10:11 AM , Rating: 2
"Then you agree with me when I say that the byline assertion "SPEC has introduced a new power-efficiency measurement standard for CPU architectures. The winner? The consumer" is false.
"

I will go back and go over the entire post, this will be an interesting discussion I am sure... but on this point...

Yes and no, I both agree and disagree -- I will divide this into two parts, first your statement then a general comment on the article.

This is the first step toward establishing a standard that provides a consistent way of measuring and reporting power. Without that, then the market is flooded with PR hype from both sides -- AMD and Intel. You have Intel who measures TDP but gives some ambiguity and does not define what 'worst case' app they use, furtermore Intel does not even define any kind of 'consumption' metric to go by... AMD invents this ACP metric, which is questionable in it's methods for reporting the result... stating == "we account for margin based on historical learning from the AMD manufacturing process" ... paraphrased.

In the end, both Intel and AMD statements are bogus from a consumption point of view because the CPU is a component (a sizable percentage granted) piece of the equation.

Now, general comment about the DT article ... meh ... I would not call it false but I would not have chosen those words either. Also, they label AMD APC and Intel TDP as 'standards' they are not. Intel has not APC equivalent to state consumption either, so it is pointless to even discuss that part.

Finally, I think the strength of my rebuttal was more a knee-jerk reaction to a more or less 'negative sentiment' about a power benchmark. I don't disagree that for a consumer level utility, this is useless... I will not be watching out for reviews and making my purchasing decisions based on whether or not this bench ran. It is simply not that important to me..... what is important I believe is that the industry moves to a standard way to test and compare at the server/platform/enterprise level.... Especially this way... why?

- It brings into focus all the platform parts that waste power, not just the CPU.

- It directs attention for all manufactures to improve their power profiles from memory to PSU to chipset to CPU.

With that comes a halo effect, which ultimately will benefit me at the consumer level.

Jack


RE: Not very useful because...
By JumpingJack on 12/11/2007 11:51:52 PM , Rating: 2
Side note... :) ... I am of the opinion thought that this benchmark is unfair to AMD in that it does not exploit and quantifiably include some of AMD's architectural key strengths.... I can see a AMD/Intel spat brewing on this one.


"Well, we didn't have anyone in line that got shot waiting for our system." -- Nintendo of America Vice President Perrin Kaplan

Related Articles
AMD Clarifies ACP, TDP Enigma
December 10, 2007, 6:45 PM
Intel Prices "Penryn" Xeons
July 17, 2007, 4:06 PM













botimage
Copyright 2014 DailyTech LLC. - RSS Feed | Advertise | About Us | Ethics | FAQ | Terms, Conditions & Privacy Information | Kristopher Kubicki