LinkedIn’s site crashed the day Sonu Nayyar started work as VP of production operations and IT at the social networking platform. The very next day, it crashed again.

“It was a bit of a shock to learn that this kind of major failure was a regular occurrence,” he told DCD. And it wasn’t that long ago either: Nayyar joined LinkedIn seven years ago, in 2010.

How did he and his team tackle the growing pains of the rapidly-expanding company, which was acquired by Microsoft for $26.2 billion in June last year? We met him at DCD>Zettastructure Singapore, and he is presenting at DCD>Converged Hong Kong. The tale he tells is far more real and gritty than most startup success stories. 

A new data center each year

A dramatic turnaround would have made a compelling story, but in reality the rapid growth of the company put an incredible continuing strain on existing network and compute resources, he told us.

“Between 2012 to 2015, we were growing so fast that we opened a new data center each year… by the time we opened one data center, we had to immediately start work on the next,” he said. ”The needs of our network are incredible – bigger than anything we’ve ever known. And it’s just getting bigger. We knew we needed to completely rethink how we did things.

“Like any startup, there was a laundry list of challenges that we could tackle, and there was a lot of focus on introducing new features that could grow our business at the time or in the future,” he explained. “There were real trade-offs that sometimes need to be made between technical resources and innovation.”

Setting a challenge

The first few data centers were essentially cookie-cutter images of each other, which wasn’t ideal as each one was built with a siloed design that reflected the organization’s structure at that time. Nayyar  felt that the team was focusing “too much” on smaller projects instead of optimizing the infrastructure.

Fortunately, those three data centers offered the necessary stability to start thinking about future growth and to make other improvements within the company’s technology stack. To get people out of their silo and make the organization more comfortable with risk, Nayyar created a cross-functional team with experts in the various disciplines. He also set out an audacious goal to spur the team to greater heights.

For instance, the team designing the next data center were asked to come up with a new architecture, ten times the size of the current design, with the potential of scaling to 100,000 servers. To top that, they were asked to do this with greater cost efficiency, and more sustainably, than ever before.

“Sometimes, talent isn’t enough if it’s not working towards the correct goal,” Nayyar told us. Having a new goal for the organization gave the team a renewed sense of purpose and helped focus the team’s efforts in the right way, he says.

Owning the equipment

The cross-functional team was given the autonomy to make the requisite design decisions, and came up with a next-generation architecture that wasn’t constrained by silos. The new vision was presented to senior executives and the board approved renewed investment in infrastructure that will be rolled out over five years.

The needs of our network are incredible – bigger than anything we’ve ever known

“A cornerstone of our strategy is ‘multi-colocation’. The idea is to ensure total reliability by being able to serve our applications from multiple data center sites, not just one or two,” said Nayyar. “We also determined through analysis that leasing and owning our own equipment was the best option.”

Nayyar believes the best decision the team made was to switch its software stack to become more open. That gave it more hardware equipment options. Interestingly, it also made it easier to recruit top talents, too.

“One of the big changes we made a long time ago was switching to a software stack that gave us flexibility to choose from a variety of hardware vendors. Not being locked into a single supplier gave us massive financial leverage,” he said.

Nayyar says he will share more about LinkedIn’s data center journey at his upcoming keynote at DCD>Converged Hong Kong, which will be held at the Hong Kong Convention & Exhibition Center in Hong Kong in November.

“We’ve built a worldwide presence at LinkedIn in only a few years, and have a lot of lessons learned that I’m excited to share with attendees,” said Nayyar. “I’m even more excited to talk about what we see on the horizon for future innovation in the data center, and can’t wait to share that with the attendees at DCD>Converged Hong Kong.”