The current proliferation of virtual machines (VMs) in data centers has raised a series of concerns, demanding procedural changes and role redefinition in all areas of IT.

Virtualisation “is very disruptive”, says Brendon Gillespie, virtualisation and storage architect at consulting firm Systems Implementers (SI). “The one thing it does is change how you think about security.”

Gillespie is part of the SI team hired to consolidate the data centers at the US military’s Hill Air Force Base in north Utah.

Currently, the data center has about 120 ESX hosts and runs about 600 VMs. Gillespie anticipates this will increase to about 1,000 once the project is complete.

SI tailored the security industry’s generally recognised components of effective incident handling for virtual environments, says Gillespie.

The basic components of incident handling are: recognising the incident is happening; containing the effects; eradicating the risk; recovering from the damage; and prevention.

“You’re actually better off in a virtualised environment,” Gillespie says. The advantage is in the ability to freeze or suspend a VM, while the first step in handling an incident with a physical machine is powering it off. The latter results in loss of everything that was loaded into the machine’s RAM, including what may be crucial forensic material.

SI identified four areas where virtualisation merits adjustments to security procedures: tracking; containment; seizure; and sanitisation.

TRACKING
“There is a lag time between when an incident is discovered and when it has actually started,” says Gillespie. That lag time can be days or even weeks and it is important to have as much information as possible in log files to track events during that time.

Having strong logging procedures is more difficult, but also more important, in a cluster or a cloud environment, where VM instances can “float” among several pieces of hardware. “People unfamiliar with virtualisation… get really concerned about that. You want to tell them you have a solid grasp of log files. If you can’t tell them that you’re confident you have all the data logged, they will get really unhappy with you and start shutting down a lot more than what you have available.”

Log files are widely spread throughout a virtualised network and it is important to try to centralise as much of that data as possible. Most of it can be found in the hyperviser syslog, VM guest logs, VCentre event and task and automation logs.

A guest log is created on a VM’s host disk every time it runs. It contains VM configuration and snapshot data, for example. According to Gillespie, the best way to centralise VM guest logs is to periodically copy them through the operating system’s console.

VCentre has log files in two locations: application logs are stored on the VCentre server in the local disk; and event and task logs are stored inside the database.

Automation log location depends on the thirdparty automation products the system deploys. “As you install a third-party product, you must remember where it stores its log file data and how you can centralise that.”

CONTAINMENT
The goal is to contain or suppress a VM that has been compromised without shutting it down. “If you want to watch what the hacker is doing let them play… without letting them know.”

The conventional way of setting up such a “honeypot” in a physical environment is bridging the compromised system through a physical appliance that records all IP traffic.

A VM can be contained by null routing its MAC address. A custom virtual bridge can be created in advance and deployed after an incident is detected. Virtual switches and pfSense (free download) or vShield can be used to create such a bridge. Gillespie expects these solutions to come ready-made from third-party vendors in the future.

SEIZURE
Compromise usually happens when a user breaks the law, using a VM within a cluster. If “the law enforcement finds out about it, they will give you a seizure notice saying, ‘we want this guy’s machine’. And you’re going to say, ‘Well, it’s virtual’.”

If the officers are shown the stack of servers hosting the offending VM, they will take the entire stack. That can lead to a catastrophic amount of downtime which can be avoided by having a plan for VM seizure, approved and signed by upper management.

Gillespie recommends four steps for VM seizure. The first is to suspend the VM and wait for it to write its memory out to disk.

The second step is to maintain the chain of custody by creating SHA-1 signatures, recognised by Federal Information Processing Standards Publications (FIPS) as signatures that cannot be repudiated. “You don’t want to have… an attorney saying you didn’t properly seize the machines.” The hyperviser console allows the user to run an SHA-1 command on every file in the VM’s directory.

The third step is to copy the VM to removable storage and hand it to the law enforcement officers with a signed affidavit.

The final step is to keep a copy of the VM on the network for as long as upper management or the law enforcement allows.

SANITISATION
Sanitisation is the process of clearing sensitive data before equipment is retired. In the case of a security incident, “you need to scrub the swap file, the disk file and all the back-ups”. Back-up files are most difficult to scrub because of their tendency to proliferate.