Data centers may be more reliable, but failures due to malicious attacks are increasing. Their cost is also rising, says Michael Kassner
Some cost accountants would cringe at his methodology, but after a 2013 DDoS attack on Amazon, Network World journalist Brandon Butler took a simple route to come up with an attention-grabbing headline: “Amazon.com suffers outage - nearly $5M down the drain?”
Did Amazon really lose this much money? Or did it lose more? Butler worked backward from the company’s reported quarterly earnings: “Amazon.com’s latest (2013) earnings report shows the company makes about $10.8 billion per quarter, or about $118 million per day and $4.9 million per hour.” The DDoS outage lasted nearly an hour, hence the almost $5 million figure.
That is a truly staggering amount to lose in one hour of unplanned maliciously-caused downtime. And Butler’s methodology seems logical on the surface. But could we get a more accurate idea of the actual cost?
The Ponemon way of estimating
If the Ponemon Institute is known for anything, it is the company’s diligence in providing accurate accounting of issues on the company’s radar - in particular security issues. Its areas of interest happen to include the cost of data center outages, which it covers in a regular report series.
The executive summary of the latest, January 2016, report says: “Previously published in 2010 and 2013, the purpose of this third study is to continue to analyze the cost behavior of unplanned data center outages. According to our new study, the average cost of a data center outage has steadily increased from $505,502 in 2010 to $740,357 today (or a 38 percent net change).”
To reach those conclusions the Ponemon researchers surveyed organizations in various industry sectors (63 data centers) that experienced an unplanned data center outage during 2015. Survey participants held positions in the following categories:
- Facility management
- Data center management
- IT operations and security management
- IT compliance and audit
The Ponemon researchers used something called activity-based costing to come up with their results.
Harold Averkamp at AccountingCoach.com describes activity-based costing as follows: “Activity-based costing assigns manufacturing overhead costs to products in a more logical manner than the traditional approach of simply allocating costs on the basis of machine hours. Activity-based costing first assigns costs to the activities that are the real cause of the overhead. It then assigns the cost of those activities only to the products that are actually demanding the activities.”
Following Averkamp’s definition, Ponemon analysts came up with nine core process-related activities that drive expenditures associated with a company’s response to a data outage (see Box). It’s a detailed list, and includes lost opportunity costs.
The research report goes into some excruciating detail, and significant real information can be gleaned from the survey’s key findings. For example, the maximum cost of a data center outage has more than doubled since Ponemon Institute started keeping track, from $1 million in 2010 to more than $2.4 million in 2016.
Overall outage costs
Source: Ponemon Institute
“Both mean and median costs increased since 2010 with net changes of 38 and 24 percent respectively,” says the report. “Even though the minimum data center outage cost decreased between 2013 and 2016, this statistic increased significantly over six years, with a net change of 58 percent.”
The report also found that costs varied according to the kind of interruption, with more complexity equalling more cost.
“The cost associated with business disruption, which includes reputation damages and customer churn, represents the most expensive cost category,” states the report. The least expensive costs, the report says involve “the engagement of third parties such as consultants to aid in the resolution of the incident.”
The Ponemon report looked at 16 different industries, and the financial services sector took top honors with nearly a million dollars in costs per outage. The public sector had the lowest cost per outage at just under $500,000 per outage.
Primary causes of outages
Source: Ponemon Institute
Next, the Ponemon team looked at the primary cause of outages. UPS system failure topped the list, with 25 percent of the companies surveyed citing it. Twenty-two percent selected accidental or human error and cyber attack as the primary root causes of the outage.
Something of note is that all root causes, except cyber crime, are becoming less of an issue, whereas cybercrime represents more than a 160 percent increase since 2010.
One more tidbit from the key findings: complete unplanned outages, on average, last 66 minutes longer than partial outages.
The Ponemon researchers did not determine the cost of an outage per hour; deciding to look at the price per outage and per minute, and how those numbers have changed over the three survey periods. The cost per outage results are considerably less than that reported for the Amazon incident, but an average of $9,000 per minute or $540,000 per hour is still significant enough to make any CFO take note.
DDoS is not going away
Data centers can only increase in importance, according to the Ponemon analysts, due in large part to cloud computing (30 percent CAGR between 2013 and 2018) and the IoT market (expected to reach 1.7 trillion dollars by 2020).
“These developments mean more data is flowing across the internet and through data centers—and more opportunities for businesses to use technology to grow revenue and improve business performance,” write the report’s authors. “The data center will be central to leveraging those opportunities.”
An interesting point made by the report is how costs continue to rise and the reasons for data center downtime today are mostly not that different from six years ago. The one exception is the rapid and apparently unstoppable growth in cyber attacks.
The report authors are concerned about this very large increase in cyber attack outages, and they make a stark warning that the problem is not going away soon.
Components of cost:
- Detection cost
Activities associated with the initial discovery and subsequent investigation of an outage incident.
- Containment cost
Activities and associated costs that allow a company to prevent an outage from spreading, worsening, or causing greater disruption.
- Recovery cost
Activities and associated costs related to bringing the organization’s networks and core systems back to normal operation.
- Ex-post response cost
All after-the-fact incidental costs associated with business disruption and recovery.
- Equipment cost
The cost of equipment, new purchases, repairs, and refurbishment.
- IT productivity loss
The lost time and expenses associated with IT personnel downtime.
- USER productivity loss
The lost time and expenses associated with end-user downtime.
- Third-party cost
The cost of contractors, consultants, auditors, and other specialists engaged to help resolve unplanned outages.
- Lost revenues
Total revenue loss from customers and potential customers because of their inability to access core systems during the outage.
- Business disruption
Total economic loss of the outage, including reputational damages, customer churn, and lost business opportunities.
This article appeared in the April issue of DatacenterDynamics magazine.