This article is from 2019. For information about an apparent outage on 5 November 2021, click here

Customers of US bank Wells Fargo have been left unable to access ATMs, or their online and mobile banking accounts due to a widespread outage.

The company's systems are believed to have been brought offline after an issue in one of the company's data centers in Shoreview, Minnesota. It is thought the outage is unrelated to a separate incident on February 1.

This story has been updated.

Differing stories

Local media KULR-8 reports that an employee tipped them off to a possible reason for the unexpected outage - a fire.

"The literally fried the servers in Shoreview. Like [we had] to put out flames," an employee conversation shared by the tipster states. "And they don't have [back-up] servers, they shut them down."

Another message said: "The power went out. Generators didn't start. 2 burned. Rest have smoke damage and can't be started."

DCD has yet to independently confirm the story, although multiple Minnesotans on social media mentioned the fire prior to the publication of KULR-8's report.

Update:

A Wells Fargo employee told DCD under condition of anonymity: As of 7:30 EST- All 3 Wells Fargo Auto locations are down too (Chandler, AZ; Irving, TX; and Raleigh, NC). We heard as well that there was a fire at the Shoreview MN data center. There is no ETA as to when it will be fixed."

In a statement, Wells Fargo said: "We’re experiencing system issues due to a power shutdown at one of our facilities, initiated after smoke was detected following routine maintenance. We’re working to restore services as soon as possible. We apologize for the inconvenience."

However, conflicting reports about the true cause of the data center outage exist. The local Lake Johanna fire department tweeted: "There was NOT a fire at the Wells Fargo facility in Shoreview, MN. They did have an activation of a fire suppression system that was triggered due to dust from construction."

Unconfirmed comments on Reddit, made prior to the fire department's statement, corroborated the fire suppression allegation: "Fire suppression went off in one of their main Data Centres from some utility work this morning. No power to any of the network or compute equipment and some failovers did not work as expected."

"The last I heard (15-20 min ago) was that everything minus core network gear was manually being unplugged from any PDUs to help the control the initial power-on. No ETA on when that will be finished, let alone when things will be back up."

Fire suppression system issues have taken data centers out before - including an ING Bank facility in Romania, where the noise of escaping gas was loud enough to damage spinning hard drives.

Original article continues:

Little information is publicly available on Wells Fargo's Shoreview data center. A 2010 planning document notes: "Currently, the Wells Fargo Campus includes a 400,000 square foot Operations Center, a retail bank, and a Data Center of 158,120 square feet.

"The addition would be one story with 16,580 square feet, of which 13,300 square feet would be fully enclosed to house the HVAC and electrical equipment. The addition will increase space for HVAC and electrical equipment needed to support the data processing equipment and to provide power if there is a power outage.

"It will increase the footprint of the building by approximately 10 percent. Security enhancements are proposed for the Data Center with an extended security fence and alteration of service drives throughout the site."

As of 2015, the company had closed some 100 small data centers as part of a consolidation initiative, that continues into 2020. Wells Fargo has also previously talked about its efforts to move to the cloud.

It is not clear why the failure of one data center caused services to go down, instead of failing over to another data center.

If you have further information about the incident, let us know confidentially at [email protected]

Update 2:

Wells Fargo said in a statement: "We want our customers to know that this is a contained issue affecting one of our facilities, and not due to any cybersecurity event. We apologize for the inconvenience caused by these system issues, and any Wells Fargo fees incurred as a result of these issues will be reversed."

Update 3:

In a Friday statement, the bank said: "As a result of the process to restore systems yesterday, some transactions and balances were not visible in online banking or ATMs. The transactions were processed normally, and customers can use their accounts with confidence. This issue has now been corrected, and all transactions are now visible. We are experiencing higher than normal volumes so there still may be delays in online banking and contact center response times."

At time of writing, 12:35 EST, numerous customers on social media claim that services remain unavailable.