The South Korean government will be questioning the heads of South Korea's two major Internet companies, as well as the owner of a data center which caught fire over the weekend, bringing lengthy service outages which hit all sectors of Korean society.
Fire broke out on Saturday in a building in Panyo owned by SK Group, which houses data centers for Kakao and Naver, Korea's two leading Internet companies. The fire was extinguished by late Saturday night, but outages at Kakao continued throughout Sunday, hitting services including Kakao's KakaoTalk messaging platform, which is used by the vast majority of South Korea's population and is regarded as "practically national communications infrastructure."
South Korea's President Yoon called for an investigation into the causes, as well as plans to prevent a repetition of the outage. In the meantime, he has ordered government departments to help in the aftermath of the disastrous outage, and called for the CEOs of Kakao, Naver, and SK Group to explain themselves.
Batteries blamed
The fire began at 3:30pm on Saturday, reportedly in a battery room in the third-floor basement of the SK Group building at 46 Pangyo-ro. The recently-constructed facility houses data centers for Naver and Kakao; it was built jointly by IBM and SK Group, and opened in 2016. The facility has two buildings with a total floor area of 66,900 sqm (72,100 sq ft or 20,250 yeong), and is touted by SK as running the latest eco-friendly technology.
The power to the building was cut off immediately and the fire was extinguished in around eight hours. No casualties were reported.
The outage had a catastrophic impact on Kakao services, affecting finance and transportation within South Korea. Services affected include mobile payment system Kakao Pay, transport app Kakao T, Kakao Games, the Daum website portal, and Melon music service.
Most of these outages were caused or exacerbated by the fundamental failure of the KakaoTalk messenger service. KakaoTalk, is actively used by between 43 million and 47 million of the country's 51 million population - and most other Kakao services require a KakaoTalk ID as login.
One Kakao service escaped the outage: Kakao Bank uses a data center in Sangam, owned by LG.
Poor planning
Multiple sources have criticized Kakao's disaster recovery preparation "If Kakao protected its data with a hot site, or a backup site, the service disruption would have been quickly resolved," said a local IT expert quoted in The Korea Times. KakaoTalk has suffered several multiple malfunctions in the last few years.
Kakao officials said they tried to implement data recovery, but weren't able to complete it because they weren't expecting the authorities to take the precaution of shutting down power to the burning building.
"We started the data replicating process after the blaze, but we were thwarted by the unexpected power shutdown," a Kakao official said, according to The Korea Times. "All our data are replicated, but it takes a lot of time because we have so much data."
Kakao vice president Yang Hyun-Seo admitted the company wasn't prepared, according to Korea JoongAng Daily: “We weren’t prepared enough for a crash of an entire server system from a fire. There is some difficulty as it is the first time in the history of IT in which 32,000 servers were shut down," he said, after visiting the site. “It is taking a lot of time to transfer traffic to additional servers.”
A statement on Saturday from Kakao's joint CEOs Namkoong Whon and Hong Eun-Taek promised swift action, and claimed the company did have a backup system for emergencies, with data divided between data centers in the company.
On Sunday, a Kakao press release announced an "emergency response committee", led by Hong Eun-Taek, head of the Kakao Community Center. It will have three sub-groups, investigating the cause, disaster countermeasures, and (a likely contentious issue) compensation. The committee will consult external experts.
On compensation, Kakao says: "In the next week, we plan to establish a channel to report damage and start receiving reports of damage. Based on the reported information, the company plans to discuss the subject and scope of compensation."
Other reports say that Kakao's services were delivered from the SK Group facility, and one other owned by KT (Korea Telecom) in Mok-dong, Seoul. The company is building its own new facility in Ansan, 30 kilometers south of Seoul, and says it hopes to open it next year (2023).
In the absence of Kakao services, updates on the outage were delivered on Facebook and Twitter.
Kakao's share price dropped from 51KRW to 48KRW over the weekend.
Naver seems to have suffered somewhat less, as it has its own data center in Chunchon. The company reported disruption to its shopping service Shopping Live, and restored services in a few hours from an off-site backup. Its share price seems unaffected
President Yoon weighs in
A statement from President Yoon Suk-Yeol on Sunday said “I feel a heavy responsibility for the inconvenience and damages that the people are going through”. He ordered the Minister of Science and ICT to take charge of crisis management for the Kakao meltdown, with other government departments assisting.
“Not only do we need to look into the exact cause as well as the damages, but we need to come up with a system that would handle such a situation as well as preventive measures,” Yoon said, according to JoongAng Daily.
Science and ICT Minister Lee Jong-ho visited the scene and said: “The government is looking at the situation seriously as [the outage] has raised concerns about the possibility of our society and economy being paralyzed."
In another statement, he said: "As the minister in charge (of ICT), I express my sincere regrets for the great inconvenience suffered by the people because of this fire. The government takes service disruptions very seriously and will consider systemic support measures in the event of such service failures."
South Korea's two main political parties, the ruling People Power Party (PPP) and the main opposition Democratic Party of Korea (DPK) have jointly called for a meeting with the three companies' leaders, SK Group Chairman Chey Tae-Won, Kakao founder Kim Beom-Su, and Naver founder Lee Hae-Jin. As well as the founders, the three CEOs Park Sung-ha of SK, Hong Eun-Taek of Kakaoa, and Choi Soo-Yeon of Naver have also been summoned.
Leaders of both parties called out Kakao's "negligence", and threatened to enact laws regulating Kakao, according to The Korea Times.
Given that the fire happened in a third-party facility, the compensation claims will likely be leveled at SK Group.
SK Group, the family-owned tech conglomerate which owns the data center building, is South Korea's second largest "chaebol" (family-owned conglomerate) after Samsung.
Footnote: With Kakao claiming 32,000 servers were involved, this may well be the industry's largest-ever data center fire disaster. The OVHcloud SBG2 facility in Strasbourg France, which burnt down in early 2021 reportedly held around 30,000 servers.
This precedent may not bode well for Kakao customers and South Korean politicians hoping for swift answers and action. Some eighteen months after that fire. OVHcloud has still not discussed the cause of the event, and a class action over the company's compensation offer is still unresolved.