The authors of the white paper set out to examine the changes to data center costs. “Unfortunately, there has been little systematic, transparent and peer-reviewed work documenting the aggregate trends in IT equipment that are driving changes in total data center costs,” the report states.

To achieve these objectives – or at the very least to start the process towards achieving them – the authors asked the following key questions:

1. What kind of data would be needed to accurately characterise trends in performance per watt, performance per server cost and power use per server cost?
2. Can changes in these parameters be measured in a credible, accurate and representative way using publicly available data?
3. If so, how have these parameters changed over the past 10 years and what can we say about how they are likely to change in the next decade?

“Data centers are at the heart of the global economy. In the mid- 1990s, the cost of these large computing facilities was dominated by the cost of the IT equipment they housed, but no longer. As the electrical power used by IT equipment per dollar of equipment cost has increased, the annual facility costs associated with powering and cooling IT equipment has in some cases grown to equal the annual capital costs of the IT equipment itself.

“The trend towards ever more electricity-intensive IT equipment continues, which means that direct IT equipment acquisition costs will be a less important determinant of the economics of computing services in the future.

“Consider Figure 1, which shows the importance of different data center cost components as a function of power use per thousand dollars of server cost.

“If power per server cost continues to increase, the indirect powerrelated infrastructure costs will soon exceed the annualised direct cost of purchasing the IT equipment in the data center.

“The 2008/09 server data applies to the servers in Figure 2. Capital and operating cost components have been derived using equations specified in Appendix A of the full document.

“Ken Brill of the Uptime Institute has called these trends ‘the economic breakdown of Moore’s Law’, highlighting the growing importance of power-related indirect costs to the overall economics of IT.



Figure 1: Power-related costs grow as power per server cost grows

“The industry has, in general, assumed that the cost reductions and growth in computing speed related to Moore’s Law would continue unabated for years to come, and this may be true at the level of individual server systems. Unfortunately, far too little attention has been paid to the true total costs for data center facilities, in which the power-related indirect costs threaten to slow the cost reductions from Moore’s Law.

“These trends have important implications for the design, construction and operation of data centers. The companies delivering cloud computing services have been aware of these economic trends for years, although the sophistication of their responses to them has varied.

“Most other companies that own data centers, for which computing is not their core business, have significantly lagged behind the vertically organised large-scale computing providers in addressing these issues.

“There are technical solutions for improving data center efficiency, but the most important and most neglected solutions relate to institutional changes that can help companies focus on reducing the total costs of computing services. The first steps, of course, are to measure costs in a comprehensive way, eliminate institutional impediments and reward those who successfully reduce these costs. “This article assesses trends in servers to help explain the driving forces affecting data center costs. It develops and documents detailed examples from available data, estimating costs and correcting them for inflation, and explaining the implications of the results.

AN EXAMPLE OF WHAT IS BEING MEASURED

DATA AND METHODS – General issues Our purpose is to develop ”peer-reviewed, consistent comparisons for performance, costs and energy use over time”. By peer-reviewed we mean that a broad section of knowledgeable industry observers (identified by name in the acknowledgements section to this report) have examined the assumptions, data and analyses, and found them credible. By consistent we mean that measurements of these parameters are conducted in a fashion that allows for meaningful comparisons over time. To understand these trends for server equipment, we ­ rst need to de­ ne system boundaries. Servers can be analysed at the CPU level, the system level, or the applications level. The applications level is closest to the tasks performed by users, but data at that level is the hardest to measure and generalise. Data is abundant at the CPU level, but CPU measurements are su‑ ciently removed from actual computing tasks that they are of limited usefulness. System level data is in the middle in terms of both data availability and relevance to actual computing tasks. In practice, the system level data is the most likely to be both available and relevant.

“Figure 2 summarises some of the key technical findings of this study for the examples investigated here. With one exception, performance per server and performance per thousand dollars of purchase cost double every two years or so, which tracks the typical doubling time for transistors on a chip predicted by the most recent incarnation of Moore’s Law.



Figure 2: Summary of trends for servers, expressed as doubling time in years

“Power used per thousand dollars of server acquisition cost is the most important driver of power and cooling costs in data centers, because the total cost to purchase electricity and almost all of the facility costs are directly related to the power use of IT equipment.

“The power-related capital costs for cooling, backup power and power distribution are substantial (approximately $25,000 per kW of IT power use), and together with the electricity costs account for almost half of total annualised costs in typical data centers.

“Power used per thousand dollars of server cost can be broken down into two components: performance per dollar of server cost; and performance per watt. Performance per dollar of server cost has, in all cases examined here, increased more rapidly than performance per watt in recent years, and this trend leads to increases in the power use per server cost.

“The result is that the indirect costs for cooling and power distribution (which are directly related to the power use per dollar of server acquisition cost) begin to offset the performance-related benefits of Moore’s Law.

“A purchaser of servers who does not assess the total cost for purchasing new servers, but instead focuses solely on performance per dollar of server acquisition cost, will invariably overestimate the benefits from buying more computing power. This mismatch between costs and benefits is the primary cause of institutional changes in most data center operations, which traditionally have separate budgets for IT and facilities departments.

“IT departments generally do not pay the electric bill or the costs to build cooling or power distribution capacity, so they do not demand high-efficiency servers, because the costs for inefficiency come out of someone else’s budget.

“Cloud computing providers have been ahead of the rest of industry in fixing these misplaced incentives, which is one economic advantage they hold compared with in-house corporate data center operators.

“There are some indications that the industry’s focus on reducing power use of servers since 2006 has been paying off, although more research is needed to confirm this finding.

“Three of our case studies (the DL360, the DL380 and the LBNL cluster computing examples) show slowing growth in recent years for power use, resulting in longer doubling times for power use per real server cost than in the other examples.

“Longer bars on the chart mean slower growth. Doubling time has been calculated using instantaneous exponential growth rates as described in the text of the document.”


This article first appeared in DatacenterDynamicsFOCUS magazine
Related White Paper: Assessing Power Trends in Servers Over Time- The full paper - registration required
Related Event: DatacenterDynamics London 10th, 11th, 2009