It’s long been known that the fastest data transfer rate can be physical media delivered by courier. Amazon has taken this to an extreme, by announcing Snowmobile, an 18-wheeler semi-trailer truck that will carry 100 petabyte (100PB) in a shipping container from the customer premises for upload to the Amazon Web Services (AWS) cloud.
The Snowmobile truck, which comes with optional security guards, was announced by AWS chief executive Andy Jassy at the company’s annual re:Invent conference in Las Vegas. It extends last year’s announcement of Snowball, a 50TB storage unit in a rugged, tamperproof box, with which businesses can send large chunks of data to the AWS cloud by post. Jassy also announced Snowball Edge, a 100TB upgrade to Snowball which can be clustered and rackmounted.
Many companies have more than exabyte (1000PB) in their data centers, and they are clamoring for faster ways to move this data, said Jassy, before he was joined on stage by a 45-ft shipping container on a truck.
Sending an exabyte of data to AWS using a 10Gbps network pipe would take some 26 years, but this drops to less than six months with a fleet of 10 Snowmobiles, he explained.
Like the original Snowball, the Snowmobile shipping container itself is ruggedized and temper-resistant. Powering up the Snowmobile and its built-in climate-controlled equipment requires around 350kW of AC power, though AWS says it can arrange for a generator if there is insufficient capacity on site.
Once in place, the Snowmobile is hooked up to the data center using fiber optic cables, where it appears as a local, NSF-mounted volume, explained AWS chief evangelist Jeff Barr in a blog entry, which also included a demonstration of Snowmobile at work - built in Lego.
Existing backup and archiving tools can be used to shunt data over to the Snowmobile, and when it arrives at an Amazon facility, it can be ingested into Amazon Simple Storage Service (S3) or Amazon Glacier.
“Each Snowmobile includes a network cable connected to a high-speed switch capable of supporting 1 Tbps of data transfer spread across multiple 40 Gbps connections. Assuming that your existing network can transfer data at that rate, you can fill a Snowmobile in about 10 days,” wrote Barr.
Multiple mechanisms ensure that the data on the Snowmobile is protected, including chain-of-custody tracking, video surveillance, GPS tracking, and cellular or satellite connectivity back to AWS.
A security vehicle escort will be arranged when the Snowmobile is in transit, and dedicated security guards can also be arranged when it is on-premises, he said.
Snowmobile is already in use by satellite imagery firm DigitalGlobe, to move 100PB to AWS. “Like many large enterprises, we are in the process of migrating IT operations from our data centers to AWS,” says DigitalGlobe VP of infrastructure Jay Littlepage in a case study quoted in Barr’s blog. “Our geospatial big data platform, GBDX, has been based in AWS since inception.”
The company has an on-site archive of high-resolution satellite imagery, covering some six billion square kilometers of the Earth’s surface, gathered over 16 years, which has been slowly moving to AWS, but the process is too slow for Littlepage: “Our constellation of satellites generate more earth imagery each year (10 PB) than we have been able to migrate by these methods.”
The Snowmobile will take DigitalGlobe’s entire archive to an Amazon Glacier vault, from where the company will be able to provide time-lapse imagery of Earth’s surface, and let customers access massive data sets, using AWS’ elastic computing platform, said Littlepage: ”Without Snowmobile, we would not have been able to transfer our extremely large volume of data in such a short time or create new business opportunities for our customers.”
AWS also announced a new data transfer appliance in the form of the Snowball Edge, essentially an upgraded Snowball that can transport up to 100TB of data. Multiple Snowball Edge devices can be clustered together to create a storage pool, and can be mounted onto racks.
Finally, built in processing capabilities allows the Snowball Edge to execute AWS Lambda functions and process data locally.