Traditional colocation providers are well-versed in physical security. The norms of constructing a resilient building and restricting access both to the building and individual customer cages are well established.

But as facilities get smarter and operators evolve to become hybrid or private cloud providers, the security landscape changes.

As those cybersecurity risks change, the relationship and responsibilities around security between the operator and the customer also need to change.

More IT, more risk for colo providers As security becomes more of a concern for organizations of all shapes and sizes, colocation providers sit in the unenviable position of needing to not only manage physical security of a data center portfolio alongside the core IT of their own organization, but also secure a growing selection of software and services being offered to customers.

Many companies might have a large remit in terms of what needs securing, but few CISOs and security leaders will be bound to as many customers in terms of uptime requirements and SLAs as colo CISOs.

In September 2020, Equinix suffered a ransomware attack that didn't affect customers. But other colo and hosting providers haven’t been so lucky in recent years. A ransomware attack on CyrusOne in 2019 led to a number of customers – mostly serviced by the company’s New York Data Center – being affected.

The same year, QuickBooks cloud hosting firm iNSYNQ was also hit with a MegaCortex ransomware attack in July. The company said it was a “carefully planned ransomware attack” on one of its primary data centers, affecting more than 50 percent of its customer base. The malware entered its network via a phishing email and spread rapidly through its network, including some backups.

2019 also saw hosting firm A2 Hosting go down for more than two weeks after a ransomware attack encrypted some of their Windows hosting servers and virtual private servers. A compromised RDP connection infected A2’s data center in Singapore before spreading to its US facilities as well as

some customer backups. Full service wasn’t resumed for more than a month.

A bad year for ransomware attacks on hosting providers, 2019 also saw ASP.NET hosting provider SmarterASP.NET as well as cloud hosting provider Dataresolution.net hit. In late 2020, Managed.com suffered an attack that bought customer sites offline.

Montreal-based service provider Web Hosting Canada suffered a lengthy outage in August 2021 it blamed on unauthorized activity by an undisclosed third-party service provider.

“No organization from a CSO perspective is there to eliminate all risks,” explains Michael Montoya, CSO, Equinix. “But our role is to help balance risk for the company; understand our risk and mitigate that risk as much as possible.”

“From a data center perspective and product perspective, we drive security across protecting the physical elements of our HVACs, our PDUs, our UPS devices, all of our power distribution, access control into our IBX facilities,” he adds.

“Then we have to protect our core critical IT assets that run our financial systems and core business infrastructure, and we have to identify our key suppliers and make sure that our data is protected within those suppliers.”

This article appeared in Issue 42 of the DCD>Magazine. Subscribe for free today

IoT and OT: increasingly integrated, increasingly targeted

The broad collection of industrial control systems – often grouped together as what’s known as Operational Technology (OT) – are relatively simple in operation, but key to ensuring systems such as HVACs function normally. Their simplicity, though, can often be an advantage for attackers.

OT is often viewed separately to traditional IT systems, meaning it can lack the same controls, maintenance, and security as more standard hardware and applications despite sitting on the same network. This means they can be both easy targets to compromise if connected to the Internet and vulnerable to attack if connected to a compromised IT network.

“Hackers attacking back-office systems, such as building automation and building management systems are common,” says William Schultz, VP of technology at Netrality Data Centers. “Hackers will use monitoring systems as a backdoor to access the broader network in order to circumvent front-end layer security. These back-office systems are generally not as well protected.”

Recent years have seen large-scale OT-based attacks increase. A 2020 survey of OT executives by Fortinet found just eight percent of respondents had seen no intrusions in the previous 12 months, yet 65 percent had seen three or more incidents. Another survey from Honeywell noted that three-quarters of facility managers were worried about the security of OT systems and improving security posture was a priority over the next 12-18 months.

Montoya notes that there are around 13 threat actor groups that are actively building tools and technology responsible for OT-related attacks. “Unfortunately in the industry there's been this perception with OT environments that it's air-gapped,” he says. “There's been this, in my opinion, very false sense of security that’s put OT environments years behind IT security.

“But if you look at the latest breaches that just happened with Colonial Pipeline in the United States, with the large meat provider GBS, or with the Florida Water System recently, a lot of organizations are finally waking up with some of these more visible breaches that are happening.

“That's been a big focus for us for years; we’ve spent tremendous efforts on doing the right level of segmentation on a physical side as well as to control access to those systems and facilities, and then ensuring that that is very well tied into our data lake so if we do see some anomaly, we can triangulate that against some of our IT assets that they may touch and how do we sort of understand more if there's a threat environment happening inside of our facility space.”

At the same time, as more Internet of Things (IoT) devices make their way into data centers, a new playground for potential attackers to compromise opens up. New sensors might make data centers

much smarter when it comes to monitoring operations, but it creates added complexity and potential vulnerability as each device can potentially be a new point of failure or route in for an attacker.

“When it comes to those IoT types of things, you want to try and isolate those things as much as possible,” notes Flexential’s VP of cyber security Will Bass. “You don't want those devices being on the same network as customer data traffic for instance.”

Managing those Industrial IoT (IioT) systems starts to look increasingly similar to managing a traditional IT stack, requiring constant security monitoring, regular patching cycles, restricted access controls, and the ability to respond to any unusual activity quickly.

“IoT devices, such as CCTV cameras and HVAC systems, are often the targeted entry point due to vulnerable security within deployed systems,” explains Michael Carr, head of strategic development at UK IT firm Six Degrees. “This often leads to access into the corporate networking environment.”

Separation of IoT and building systems from both core IT and customer environments – no matter what kinds of services an operator might be providing – is key, as is robust monitoring and access management. Regular penetration testing and patch management processes should also be adopted.

“At our data centers, all supporting infrastructure is both physically and logically separated from customer environments,” Carr says. “Physical security controls – including door access, CCTV, and HVAC systems – operate on separate networks within facilities, and all segmented control networks

and systems are monitored through event collection into a SIEM platform analyzed 24x7 by our SOC facility.”

New services mean new security challenges for colo providers

Colo companies are increasingly offering software and service solutions that blur the lines between traditional colocation and cloud.

“You still have customers that come in and just want to buy data center space,” says Bass. “But we're also having more customers come in and want some colo space, some private cloud, some help with disaster recovery.

“You definitely see that merging and changing for data center companies,” Bass continues. “Protecting the HVAC is definitely much different than having a VMware stack that has customer data on it, and we have to have the right processes and alerting and monitoring in place.”

As colos evolve their offerings, the cybersecurity focus has to change too. Software development requires constant consideration around security, but even more so when applications and services being developed are being consumed externally.

Colo providers need to ensure they are adopting the latest advice and methodologies around securely developing applications, such as OWASP top ten or NIST’s Secure Software Development Framework, to ensure they offer resilient products.

“As we move more to the software element, we have put a lot of focus into ensuring that we've got the right security controls around our software fabric or metal service, starting with how we do development overall of our fabric solutions,” says Montoya.

“We're running a very strict automated CI/CD pipeline; we work very closely with our product organization to control that instrumentation and ensure that we have visibility across that pipeline so that before it hits production we are able to sign off and ensure that all of the right security gates are made.

“Starting from the threat modeling, all the way to the build, into the actual scanning of code as well as anything in production that we need to manage once it gets into our production facilities.”

Colos becoming clouds means new security responsibilities

Major vulnerabilities in IaaS providers’ cloud stacks are rare, while companies leaving themselves accidentally exposed due to configuration errors are nearly daily occurrences.

Exposed AWS S3 Buckets leaking information have been a common configuration faux pas for a number of years, but AWS will always reaffirm its platform is secure. Such cloud compromises are usually rooted in human errors; something that cloud providers often offer a service to help with, but would never take the blame for.

Cloud providers have spent years informing customers about the cloud security shared responsibility model and the notion that they will secure the hardware and underlying software, but everything to do with configuration, access, and monitoring of data and applications remain firmly in the customers’ hands.

Where the traditional roles and responsibilities of colo operator and customer have long been well understood, those old lines have become blurred as more colo providers offer cloud services. And as yet there isn’t an equivalent shared responsibility model for the new cloudy colo firms for who owns what risks.

“As organizations seek to take advantage of colocation services, we find that there isn’t always a clear delineation for which entity is responsible for network security,” says Mike O’Malley, SVP of technical advisory firm SenecaGlobal. “Companies often incorrectly assume that the colocation provider is handling all aspects of cybersecurity, protecting their servers, applications, and

digital assets in a sort of electronic vault.

“Colocation providers that clearly communicate to clients how they protect the physical colocation premise and network infrastructure – and what security protections for applications and data need to be handled by the client – are in a better position to protect the entire ecosystem.”

Equinix’s Montoya acknowledges that no such shared responsibility model exists for the new world of cloud and service-based colocation, and that the industry as a whole probably has to get better at educating both customers and operators on who owns what risks.

“There's a lot of work we need to do as colo providers to really help people understand where those demarcations are, and how we play in the overall shared inherited risk model,” he says. “I think as a community there's a lot more dialog that needs to happen and collaboration around thinking about inherited risk and shared security overall.

“This is an incredible opportunity for us as a community to create more standardization, so that we all are speaking the same language, and we're all able to build support around a very sort of common approach to how we're dealing with shared security.”

Quite what that shared responsibility model between colo and customers could look like in the future hybrid world is still up for debate, but for now the onus is still very much on the customer to do their homework.

“That responsibility aspect is definitely different from someone that's in our colo than it is someone that's in our private cloud,” adds Flexential’s Bass, “But it's figuring out where do those responsibilities stop [that is difficult]; every company needs to do that risk assessment.”

Changing customers, changing risks As much as colo providers need to ensure attackers don’t use compromised company IT or building systems to attack customers, at the same time they have to be vigilant that their customers aren’t posing a potential risk to the company or its other customers.

“Are our customers a risk? Absolutely,” acknowledges Montoya, “We have to understand our customer base to understand what risks they may bring us.”

Montoya notes that there are many threat actors that are interested in using colo companies to perform what he calls upstreaming - also known as island hopping or supply chain attacks - where a provider is compromised in order to disrupt or pivot to customers.

“They're not necessarily interested in us, but maybe they're interested in just disrupting our customers,” he says. “Our concern is how they would use our facilities or our services to try to disrupt the services of our customers.

“You think about some of the big system integrator and telecom breaches that have happened over recent years; it was less about going after those companies and more about going after their customers.”

As a result, Montoya says Equinix does a lot of analysis on who would be interested in its customers, whether that’s through disrupting a facility or compromising a network and attempting to pivot into a customer’s environment (which he unsurprisingly says would be ‘incredibly hard to do’).

He does note, however, that companies hosting problematic content on Equinix infrastructure are higher on that risk register than an actor hopping from their cage into its interconnection environment.

“Our concerns with customers are less around can they pivot to one of our physical services and probably more who are our customers and are they bringing other concerns to us, such as the events of January in the US.”

Bass agrees that customers can attract added interest from unwanted eyes, saying Flexential has a number of clients it doesn’t talk about to avoid becoming more of a potential target for sophisticated actors.

For now, however, the industry standard of colos protecting the building and leaving customer hardware well alone remains in place, leaving operators forced to remain vigilant but hands-off.

“We do see, in some cases, activities that come to us because customers may have poor hygiene in their environment,” says Montoya.

“We will alert them and help them understand the potential risk in their environment. But we don't have control over how our customers perform their own hygiene.”

Flexential offers incident response services, but can only help if requested by the customer. Like other colo firms, it needs to make sure customer incidents aren’t in danger of bleeding out while remaining hands largely hands-off. He notes it is often the smaller ‘mom and pop’ businesses that end up having security challenges.

“We want to make sure that we understand exactly what's happening on the edges of all customer environments so that we can see if they're having some sort of security incident or issue. We want to ensure that that is not it's not getting out and going to anyone else.”

“But on the flip side, it is their environment. We're not going to go in and make changes to it without them and working with them on those issues.”

Supply chain security gets a new focus A number of companies DCD spoke to noted that the recent SolarWinds breach – where attackers compromised the company’s Orion IT monitoring and management software to gain highly privileged access to its customer’s networks – has driven growing interest and focus on supply chain security.

Enterprise customers are now wanting to make sure the supply chains of their own supply chain are secure. Audits from those customers about controls, compliance, security are growing in number and detail, leading to their suppliers asking the same from colo providers.

“[Our customers] are making sure that they are secure so that they can prove to their customers that they are secure; that customer data is secure in their environment, which could also be part of our environment,” says Bass.

As a result, merely being compliant with any given compliance requirement or standard – whether NIST, ISO, Cyber Essentials, SOC, HIPAA, PCI, or any number of others – is no longer good enough.

Montoya notes that not only are the number of audit requests increasing significantly, but the intensity of those audits have also increased. Where in previous years customers would be happy with a copy of the desired compliance certificate, they are becoming more knowledgeable and creating their own audits with customized controls.

“A lot of customers now like to create their own control view and bring increased inspection on controls,” he says. “Where they might have previously had 20 additional controls, suddenly we see in some of these audits they’re doing 100 additional customized controls for review.”

At the same time, colo providers must take closer looks at their own supply chains. Every vendor employed – whether to help the company operate its own business or provide a service to customers – creates a potential risk for both the colo and its customers.

Target’s 2014 data breach via a compromised HVAC provider remains one of the most notorious examples of supply chain breach, and one that’s very relevant for a data center industry reliant on air conditioning. But risks can come from almost any supplier.

“You really have to understand the supply chain that you're relying on to deliver your services, whether those services are to protect your core data, protect your core business or products, or protect your customers,” explains Montoya.

“We've implemented a third-party audit process as well as what we call continuous assurance which helps us take our key suppliers and evaluate them for their cyber risk in a much more real-time basis.”

The future of colo The reality is the colo landscape is both changing rapidly while still remaining the same; some companies will always want some standard hosting services, and providers will still need to protect their core IT and their buildings, just with the added complexity of multi-cloud.

“I think traditional colo is always going to be around,” says Bass. “Even if it's not the small company coming by and buying colo from you, data has to live somewhere, all these SaaS applications have to live somewhere.

“The customer profile might change and I certainly think we're going to see a more mixed hybrid type of approach coming,” concludes Bass.

Some customers will always only ever want you to be a landlord to host their cages, while others will want much more. It’s up to colo providers to be ready to offer what customers need, but do it securely.