A major Fastly outage that brought down much of the Internet was blamed on a software bug triggered by a valid customer configuration change.
The June 8 outage brought down Amazon.com, Twitch, Reddit, the UK government's website portfolio, and a whole bunch more sites and services.
Fastly said it identified the issue in one minute, and had restored 95 percent of its network within 49 minutes. However, some customers saw longer outages due to increased origin load and lower Cache Hit Ratio.
Content delivery network Fastly runs an Edge cloud between companies' data centers and the end user, reducing latency, protecting from DDoS attacks, and helping them handle traffic spikes.
Content Downtime Network
It appears that Fastly inadvertently introduced a booby-trap in its system, by deploying a version of its software which could be disabled by a cuctomer: "On May 12, we began a software deployment that introduced a bug that could be triggered by a specific customer configuration under specific circumstances," SVP of engineering and infrastructure Nick Rockwell said in an explanatory post.
"Early June 8, a customer pushed a valid configuration change that included the specific circumstances that triggered the bug, which caused 85 percent of our network to return errors."
The outage began at 9:47 UTC, with services starting to recover from 10:36. At 12:35 it was fully mitigated, and by 17:25 a fix for the bug began to be rolled out
"We are conducting a complete post mortem of the processes and practices we followed during this incident," Rockwell said.
"We’ll figure out why we didn’t detect the bug during our software quality assurance and testing processes."
He added that the company "should have anticipated" the issue, and apologized to Fastly customers.
Investors, however, do not appear phased. If anything, it had the reverse effect, with company shares surging - perhaps due to the attention the outage garnered, or because it revealed just how many customers Fastly has. Even its rival Amazon, which has its own CloudFront CDN is a Fastly customer.
At time of writing, shares have risen about 10 percent, adding a little under $600 million to its valuation. Prior to the outage-related spike, the company's share price had been on a steady decline, down 42 percent year to date.
Part of the drop was due to general investor sentiment in growth stocks cooling, but was exacerbated by the company's May announcement that its CFO was leaving.
