The Case For Medium Voltage Data Centers

When the electrical engineer sits down to design the service entrance for a new commercial building the first choice is usually a 480 volt system. Only when the building is very large or high-rise in nature, and voltage drop or maximum bus capacity considerations require it, does the engineer opt for a medium voltage service entrance. Then, more often than not, a 15 kV class of equipment is selected, especially if that matches the utility voltage.

Data centers and mission-critical facilities with their huge and highly concentrated loads are good candidates for medium voltage service, yet they are often served with low voltage for a variety of reasons. One of the strongest incentives the mission-critical engineer has to use a 480 volt service is that most Uninterruptible Power Supply systems employed in data center applications cannot use a higher voltage directly. Also, since computer rooms typically have very dense and compact loads their electrical distribution feeder lengths are not long enough to require higher voltages.

Another set of reasons for selection of a low voltage service is the perception that medium voltage switchgear is inherently more costly to purchase, requires more space to install, and is less reliable than low voltage switchboards. And perhaps the top reason for going with low voltage is the perception that medium voltage gear is more dangerous to operate. Below we will attempt to explain why these perceptions are false, and why the choice of 4160 volts as an electrical system voltage may also be the green choice.

When less is more (and green)

Recent advancements in medium voltage breaker technologies have changed the norms for medium voltage switchgear design. The most noteworthy is the introduction of compact, solenoid-actuated vacuum breakers with on board capacitor-stored operating energy and integrated electronic trip units. These new breakers have a look and feel similar to low voltage breakers. However, they have far fewer moving parts and have proven to be more robust in torture testing than standard spring mechanism operated breakers. This has increased their reliability considerably. It substantially reduced their initial cost and maintenance costs as well.

New medium voltage switchgear architecture is starting to enter the market to house these breakers, most notably a front accessible design with reduced depth. Such gear can now handle equivalent power in about 3/4 the footprint of equivalent low voltage switchboards. The smaller footprint results in switchgear that costs less per kVA than low voltage equipment.
The biggest advantage of medium voltage distribution, of course, is the reduction of feeder ampacity. Data centers with 480 volt service and distribution virtually float on copper. Moving to 4160 volts allows dramatic improvements. Compare, for instance, a 75 foot long 2500 kVA 3-phase service entrance ductbank, which may be one of many in a large data center. At 480 volts, in a typical installation, it requires eight 4-inch conduits, 6 cubic yards of concrete, 2720 feet of 750kCM wire, and 64 terminations. At 4160 volts the service would typically be 3-wire in lieu of 4-wire, and it will take just one 4-inch conduit, 2 cubic yards of concrete, 255 feet of 750kCM MV conductor, and 6 terminations.

Data center construction can require dozens of such feeders. If they are installed in underground ductbanks that crowd each other, the risk of mutual heating can occur. Extreme care must be taken during design and installation to avoid ductbank heating issues that can destroy not only the feeders, but the switchboards they serve, as well. If that 75 foot long service feeder is one of many in close proximity, it may need nine or ten conduits to reduce heating.

Changing to a medium voltage feeder not only reduces the conduit, wire and termination count, it also reduces the worry over cross-ductbank heating. At the end of even a cursory comparison it can be seen that more power can be delivered more reliably with less cost and smaller equipment at higher voltages. Besides these obvious benefits in the reduction of building materials such as copper and PVC, though, there is also an environmental benefit from not having to mine or manufacture as much of these products in the first place. There may even be LEEDS points in this choice. But is it safe?

Safety pays

In light of the recent code NFPA 70E for arc flash, some engineers (including CCG) are re-evaluating their designs for enhanced electrical safety. CCG’s analysis revealed some surprises. Intuition tells us that the higher the voltage the more dangerous the equipment. However, the contrary may be true regarding arc flash hazard.

Using IEEE 1584, Guide for Performing Arc-Flash Hazard Calculations, as the method for calculating arc flash levels, CCG developed a comparison of two designs to determine the differences in arc flash incident energy and the required category of personal protection equipment, or PPE. (See the accompanying chart.) Each of the two designs consisted of a standard 2.5 MVA pad-mounted utility transformer connected to a 34.5 kV, 750 MVA utility distribution feeder with appropriate primary protection. The secondary of transformer #1 was 480 volts and served a 4000 amp main switchboard with an insulated case main breaker. Transformer #2 has a 4160 volt secondary serving a 600 amp main switchgear lineup with a solenoid-actuated vacuum bottle main breaker.

In both cases the main breakers were fitted with electronic trip units with adjustable long time and short time elements, but, as is typical in many facilities with critical loads, instantaneous elements were turned off to provide for selectivity with branch feeders. Both the low voltage and medium voltage equipment were given a 1000 HP motor as a load, and the trip devices were adjusted to provide for maximum feeder selectivity and motor starting capability. As recommended by the standard, we calculated arc flash energies for maximum arc currents, and for arc currents where fault path resistance may reduce them to 85% of maximum. The results were eye opening.

The worst case classification for the 480 volt switchboard was PPE Category "Dangerous," not to be worked on live. The 4160 volt switchgear, on the other hand, had a PPE Category 1. The IEEE 1584 calculations for the maximum incident energy indicates the low voltage fault energy to be more than an order of magnitude greater than the medium voltage fault energy. The main reason for this difference lies in the way IEEE 1584 calculates arc currents for equipment 1kV and below versus higher voltages. Low voltage arcs tend to be much less than the full three-phase bolted fault current. This often puts breaker trip elements into the long time range instead of the short time range, allowing the arc to exist for a longer period of time. Conversely, medium voltage arc currents are seen to be much closer to the bolted fault level, and trip elements stay in the short time range for almost all types of arcing faults.

Take a look at the relative arc flash boundary distances; 297 inches (nearly 25 feet!) for the worst case 480 volt arc flash, and 128 inches for the 4160 volt switchgear. A strict interpretation of the arc flash code might require that if the arc flash level is calculated and posted on the equipment, then the arc flash boundary must be observed in ALL situations, equipment doors open OR closed, unless the equipment is classified as "arc resistant." If this strict interpretation is enforced, it would not be legal to even enter a main electrical room while it was energized.

Therefore, for a temporary tradeoff of selectivity for greater safety, breakers should be outfitted with a "maintenance mode" which automatically engages the instantaneous trip settings on the main in response to an appropriate external signal. With their instantaneous trips active, both LV and MV breakers have clearing times in hundredths of seconds instead of tenths of seconds. In that condition the LV switchboard will now require PPE Category 1 (fire retardant clothes), while the MV gear is classified PPE Category 0 (normal work clothing). But notice that even in the maintenance mode the incident energy at the LV board is five times higher than at the MV gear, and the arc flash boundary at the 480 volt board is still greater than 4 feet.

The main point of this discussion is: The smaller, more reliable, less expensive, and greener choice, that is the medium voltage equipment, is also the safest in all circumstances.

4160 Volts: the new 480 Volts

It’s a short estimating exercise to evaluate how many transformers can be paid for with the difference in cost between low voltage and medium voltage feeders. However, if 4160 volts is the chosen medium voltage there are many types of equipment that can be served at that voltage without the need for huge unit substations and step-down transformers. Engine-generators and large chillers are readily available in that voltage, and with little cost premium over 480 volt models. There are also choices for medium voltage UPS systems. Certain Diesel UPS systems with flywheel energy storage can be purchased at medium voltages, as are some off-line type static UPS systems with battery storage. Diesel UPS and off-line static UPS systems both run at significantly higher operating efficiencies than typical on-line static UPS systems. For instance, an off-line static UPS using a 4160 volt thyristor bypass switch as the normal power path, an intermittent duty inverter, and a quarter-sized step-up transformer can achieve electrical efficiency as high as 99%. With no transformation in the normal operating mode power path the losses between the utility grid and the UPS output bus can be less than 3%.

Critical power at 4160 volts can be very economically extended into the computer room, especially if "A" and "B" redundant circuits are required. Larger power distribution units (PDU), say 500 to 1000 kVA, with 4160 volt primary step-down transformers make sense here. In lieu of large 480 volt switchboards distributing critical power to several small PDU’s, the output of a medium voltage UPS system can directly feed a few medium voltage PDU’s with no need for distribution switchgear within the computer room. With a properly configured UPS system and medium voltage PDU’s it should be possible to contain all the electrical losses along the critical power path from the utility grid to the computer rack within 5% to 6%. That number usually falls between 10% and 15% for typical critical power systems in existence today.

With the potential for millions of dollars in equipment and infrastructure capital cost savings in a typical data center and hundreds of thousands less yearly in operating expenses, plus decidedly safer electrical equipment rooms and a greener design, the engineer of large data center projects should seriously consider 4160 volt electrical systems. Furthermore, manufacturers and vendors of mission-critical equipment should stand ready to support those efforts with medium voltage options for their products.  

CHART 1 – ARC FLASH CALCULATIONS AT SWITCHBOARD/SWITCHGEAR BUS

Mike Mosman, PE, is Vice President and Chief Technical Officer for CCG Facilities Integration Incorporated in Baltimore, MD, and John Mayan, PE, is one of their senior engineers expert with computerized short circuit and coordination studies.