At every company Mike Leber has worked for in his 15 years as a software engineer, he has ended up building a network. “I was always the person who’s like Prometheus, trying to bring the fire,” he recalls. “Way back in 1989, it was... difficult to even get connection to the internet because it was largely non-commercial.”
He would save his colleagues from having to transfer files onto floppy disks by connecting their machines to the network. His internet expertise came from doing that work.
Today, Leber is president of
Hurricane Electric, a company he founded in his garage in the early 1990s, which reportedly operates the world’s largest IPv6 network and one of the world’s largest IPv4 networks.
In September, Hurricane celebrated the 15-year anniversary of its founding and in November, the company opened the second phase of its 208,000 sq ft colocation data center in Fremont, California – Leber’s hometown and his company’s birthplace.
TO INFINITY AND BEYOND
Hurricane got its start because of Leber’s presence at the public internet’s dawn and the ‘extra-curricular’ networking jobs he did at the companies that had originally employed him to write software.
By 1994, Leber had a full-service consultancy, with numerous staff setting up servers for various clients. The Mosaic web browser was created around the same time and became the most widely used browser, spurring growth of internet use by the general populace.
The developments caused a lightbulb to go off in Leber’s head: “As opposed to setting up servers for other people, we could actually set up servers for ourselves and provide web service to clients.”
Leber and his team brought the idea to life and the company quickly grew to the point of needing to expand into MAE-West – a major internet exchange point in San Jose, California – and interconnecting with networks there.
“We had periods where we had 20% growth a month,” Leber says, recalling the notorious dotcom boom. “It was quite crazy.” Hurricane Electric ended up expanding by about 2,000-3,000 sq ft in MAE-West before it opened its first 46,000 sq ft data center in Fremont in 2000 – the Fremont 1.
Expansion announced in November was of the company’s second Fremont data center, bought five years later and brought online in 2006.
Phase 1 of Fremont 2 – a former Apple manufacturing facility – contains 11 similarly designed computer rooms, each housing about 80 cabinets arranged in six rows.
THE RIGHT WAY TO BUILD
Hurricane decided on a very different design for the 3MW second phase: one 24,000 sq ft computer room with long rows of more than 80 cabinets each. The blue uncaged cabinets are provided by the company.
“This is the right way to build,” says Hurricane Electric’s director of IPv6 strategy, Martin Levy, explaining the decision to move away from multiple-room design, which had proven more costly to build. “You’ve got the cost of the walls and you’ve got a completely different cooling mechanism. Scalewise, we couldn’t build this big enough if we kept putting all the walls up there.”
Free cooling was a big part of the equation – something the company would not have been able to take advantage of had it continued building out in suites.
“Even though the sun’s out, it’s a cool day in Fremont,” Levy said during an interview in early November. “There are no compressors running today… and although we’re not full here, the reality is we can use a large number of days of free cooling: drop the compressors off, filter the air, bring it in and out, and that makes for a significantly better economy on running the cooling system.”
The facility is designed to separate hot and cold aisles, uses variable-frequency drives in its rooftop McQuay Maverick II HVAC system and Eaton’s 9395 UPS units, capable of operating at an efficiency rate of 99% when power coming in is in an ideal state.
According to Eaton’s Bob Lyding, Hurricane was the first company in California to deploy the product that has since enjoyed popularity among data center engineers in the state.
If the incoming power is not ideal, the system switches to double-conversion mode in 1.2 milliseconds, Lyding says. Leber confirmed the claim, explaining that he had personally examined test documentation before agreeing to buy the equipment.
Operators can remotely measure power going to air conditioning separately from power used to run the IT equipment, as well as every electrical outlet in every cabinet. “We can monitor any customer,” Levy says. “They can see how much power they’re using on a cabinet-by-cabinet or outlet-byoutlet basis.”
Humidity and temperature are measured at HVAC units above (instead of a raised floor, the data center has an overhead cooling system). The ‘environmentals’ are not measured at rack level, which Levy says was unnecessary given the design. “By having a hot-cold aisle – and having that enforced on our customers – we can find out consistently what’s going on.
“There’s also another subtlety: we do a fixed power-per-cabinet (limit) for our customers. There are either 15- or 20-amp feeds into the cabinets and that’s the limit. If a customer comes along and wants a high-power server deployment, they normally end up using many cabinets.
“We don’t get hot-spots inside our data center, so we know what the heat load is going to be when it’s fully built,” says Levy.
Consistent power and heat load across the data center floor allows for efficient design of the cooling system, since the operator does not have to invest in being able to take a lot of heat out of any particular customer’s equipment.
“And, more importantly, a customer – when they’re installed – is not going to be next to somebody who’s dumping a lot of heat,” Levy says. “That means our design, and therefore our cost structure, is far more efficient than having to build for each customer.”
WORLD HUNGER FOR BANDWIDTHGiven its start as a web service provider, Hurricane’s networking business is bigger than its colocation business.
The company operates one of the world’s 10 largest IPv4 networks, with 4,500-plus BGP sessions with more than 1,250 different networks, at more than 30 exchange points in the US, Europe and Asia. Leber claims that Hurricane’s IPv6 network is the largest of its kind in the world.
His claim is based on the number of IPv6 networks Hurricane connects with and the number of IPv6 customer prefixes it announces. In early November, the network passed the mark of interconnecting with 600 IPv6 networks.
About mid-November, Hurricane announced plans to add new points of presence at Equinix data centers in Tokyo, Hong Kong and Zurich, and two more cities in Eastern European markets are in the pipeline, Levy says.
The biggest demand for bandwidth in Europe, according to Levy, has been Frankfurt and Amsterdam feeding into Eastern Europe. That growth is spurred by a large amount of new dark fibre being built.
The company’s decision to go into Stockholm in June was based not only on demand in that market but also to handle all the new bandwidth that is coming in from Russia and other former Soviet republics.
“We install bandwidth on a regular basis,” he says, adding that the global recession has not affected the company’s internet bandwidth business. “We just installed another 10GB between London and New York, and put another 10GB into the DE-CIX exchange in Frankfurt.”
There is also no slowing of growth in the US markets, Levy says, citing California cities including Los Angeles, San Francisco and Palo Alto, where there is demand for trans- Pacific connectivity with Asian markets. “That’s a much more mature market, but we still see an enormous amount of need for bandwidth there.”
Similar trends are observed in New York and Ashburn, Virginia, and in Miami, Florida.
Growth in demand for bandwidth coming in from Latin America comes in spurts.
The most recent expansions into the three Equinix facilities in Asia and Europe were driven by demand for IPv6 connectivity.
“IPv6 is the next-generation internet protocol,” Leber says. “It’s like the whole internet all over again for the backbone.”
IPv6 – THE WAY OF THE FUTURE
The company started its IPv6 business in 2001 and has been heavily promoting the networking protocol ever since. Today, almost one decade later, Hurricane is starting to reap the benefits of its long-time research and development investment in IPv6.
Leber predicts that IPv4 addresses are going to run out around 2012, at which point all new network-connected devices are going to have to be IPv6-compatible.
“So, there is going to be a whole cottage industry of consultants all over again, just like there was for Y2K, with people running around, educating about… IPv6-readiness,” says Leber.
“Basically, if you’re a backbone, or if you’re somebody who actually sells internet service… at some point IPv6 is going to be a drop-dead item – it won’t be just a checklist item. If you don’t have it by, say, 2012-2014, basically you won’t exist as a normal business.”
Leber’s only word of caution to data center operators regarding the looming shift to IPv6 is to make sure their backbone providers are compatible with the next-generation protocol.
“Fibre is already IPv6-compatible,” he says with a chuckle. “Electricity is already IPv6- compatible.”
This article first appeared in DatacenterDynamics FOCUS magazine
