Google sings as the music streamer moves its complete works from data center to cloud
Online music streaming platform Spotify is moving its core infrastructure to Google’s cloud. The new host has hailed this as a major win in its competition with Microsoft Azure and Amazon Web Services (AWS).
Spotify will be moving from its current infrastructure, which includes its own and rented data centers mixed in with a bit of AWS cloud, to be based entirely on the Google Cloud Platform. The move involves a complete restructuring of Spotify’s operations.
Spotify’s decision to move out of the datacenter business has come at the end of a period of comparative evaluation in which Google came out top. The migration will be handled in two streams: services and data.
Former Spotify headquarters in Stockholm
Source: Wikimedia Commons / Erik Stattin
Services are being moved first because they comprise a number of micro-services which can be easily transferred to Google’s Cloud Storage Compute Engine. It will enable each micro-service to employ the right amount of resources on the fly.
This will be followed by the heavy lifting operations - moving Spotify’s massive store of music and operational data to Google Cloud Datastore and Bigtable. The move will also provide the services modules with input/output and local solid state disk (SSD) storage.
“What really tipped the scales towards Google for us, however, has been our experience with Google’s data platform and tools,” said Spotify’s Nicholas Harteau, vice president for engineering and infrastructure. “Good infrastructure isn’t just about keeping things up and running, it’s about making all of our teams more efficient and more effective, and Google’s data stack does that for us in spades. “
The company aims to keep all services running during the migration period. It is also investigating a completely new technology stack based on Google products. At present, Spotify is using a number of products from Hadoop, MapReduce, Hive and a series of home-grown dashboard tools.
Google’s BigQuery and Cloud Dataproc will allow Spotify to perform in-depth, interactive analysis more frequently. The same tools will be used to guide product development, test features and add more-intelligent user-facing features.
Spotify is also using Cloud Pub/Sub, Google’s global service for messaging and streaming data, to gather and forward hundreds of thousands of messages per second to its ecosystem. The Cloud Dataflow data processing service will power Spotify’s ETL (extract, transform, load) workloads so that a single cloud-based managed service can provide both batch and stream processing.
Guillaume Leygues, lead sales engineer for Google Cloud Platform, said, “It’s not every day you move a 75 million-plus user company from a home-grown infrastructure to the cloud but if you use Spotify, more and more of your musical experience will be delivered by Google Cloud Platform over the coming weeks and months.”
Earlier this month, another major content streaming service completed its move into the cloud. Netflix replaced its data center network with AWS - but the company continues to run its own content delivery network.