A few years ago, Dark Reading’s Executive Editor Kelly Jackson Higgins stated what most security pundits knew, but did not want to admit, “There’s a growing sense of fatalism: It’s no longer if or when you get hacked, but the assumption is you have already been hacked, with a focus on minimizing the damage.”

This is not something data center operators want to hear.

Don’t give up

The apparent ability of technically advanced adversaries to defeat current best-practice security systems at will may be fueling this feeling of futility. However, Cisco, to the company’s credit, isn’t ready to give up. Managed Threat Defense (released in 2014) is a Cisco service designed to protect clients against known intrusions, zero-day attacks, and advanced persistent threats.

To make Managed Threat Defense a viable deterrent, Omar Santos, a principal engineer at CiscoProduct Security Incident Response Team (PSIRT) says Cisco developed the following (blog post):

• The ability to capture full packet-level data and extract protocol metadata to create a unique profile of each customer’s network
• A global network of Security Operation Centers to obtain alerts regarding nefarious activities
• The technology to meld alert data, the company’s profile, and Cisco’s threat intelligence together, creating an action plan

Almost immediately, Cisco analysts noticed that the amount of alert data needing processing was far more than they had envisioned. The deluge of data from the service’s clients, which included several large enterprise organizations, was so overwhelming, analysts could not separate what was important from simple log entries.

Existing methods are too slow

Due to the amount of captured data, adversaries were able to capitalize on the length of time required to analyze the intel and formulate a response. “During a breach, sensitive customer information and intellectual property are compromised, putting the company’s reputation, resources, and intellectual property at risk,” explains Pablo Salazar, former manager for Cisco Security Solutions in this Cisco blog post. “Quickly identifying and resolving the issue is critical, but traditional approaches to security incident investigation can be time-consuming.”

According to Salazar traditional methods entail researching:

• Reports from a security incident and event manager (and running batch queries on other telemetry sources for additional context)
• External threat intelligence sources to uncover proactive warnings to potential attacks
• Network forensics tools with full packet capture and historical records in order to determine context

To respond to the data overload and better serve Managed Threat Defense clients, developers from Cisco and Hortonworks came up with OpenSOC, a security analytics framework.

Analytics platform

OpenSOC provides a platform for the application of anomaly detection and incident forensics by employing big data and machine analysis. “By integrating numerous elements of the Hadoop ecosystem such as Storm, Kafka, and Elasticsearch; OpenSOC provides a scalable platform incorporating capabilities such as full-packet capture indexing, storage, data enrichment, stream processing, batch processing, real-time search, and telemetry aggregation,” explains Salazar. “It also provides a centralized platform to enable security analysts to detect and respond rapidly to advanced security threats.”

OpenSOC key elements

To whittle down the raw data into actionable information as soon as possible, Salazar mentions the OpenSOC development team focused on three key elements:

• Context:
  The first order of business is to manage the mountains of captured data. “OpenSOC ingests data and pushes it to various processing units for advanced computation and analytics, providing the necessary context for security protection and the ability for efficient information storage,” writes Salazar. “It provides visibility and the information required for successful investigation, remediation, and forensic work.”
• Real-time:
  Crunching the data in as near real time as possible is as important as figuring out what data is actionable. That means applying threat intelligence, geolocation, and DNS information to the collected data at scale. If it works out, analysts can make their decisions based on accurate and timely information.
• Centralized Perspective:
  Having the correct information available quickly still falls short if it’s not in an understandable format. “The interface presents alert summaries with threat intelligence and enrichment data unique to an alert on a single page,” adds Salazar. “The advanced search capabilities and full packet-extraction tools are available for investigation without the need to pivot between multiple tools.”

Put simply, what Salazar points out is that by using OpenSOC, security analysts can — via a single tool — navigate through data important to them, and avoid wading through immense amounts of unstructured data. “It can be tailored to ingest and view any telemetry, whether it is for specialized medical equipment or custom-built point of sale devices,” suggests Salazar. “By leveraging Hadoop, OpenSOC also has the building blocks to scale the amount of data it collects, stores, and analyzes based on the needs of the network.”

An open source project

In 2014, Cisco and Hortonworks released OpenSOC to open source. Soon after that the OpenSOC project started. “The OpenSOC project is a collaborative open source development project dedicated to providing an extensible and scalable advanced security analytics tool,” says the OpenSOC project website. “It has strong foundations in the Apache Hadoop Framework and values collaboration for high-quality community-based open source development.”

The current OpenSOC framework provides the following capabilities:

• Extensible spouts and parsers for attaching OpenSOC to monitor any telemetry source
• Extensible enrichment framework for any telemetry stream
• Anomaly detection and real-time rules-based alerts for any telemetry stream
• Hadoop-backed storage for telemetry stream with a customizable retention time
• Automated real-time indexing for telemetry streams backed by Elastic Search
• Telemetry correlation and SQL query capability for data stored in Hadoop backed by Hive
• ODBC/JDBC compatibility and integration with existing analytics tools
• Designed to scale up to handle millions of messages per second

It is the hope of those involved in the OpenSOC Project that the framework will continually evolve, improving organizations’ ability to handle security incident response. To that end, the project has the following goals in mind:

• To provide a collaborative open source community for development of an advanced security analytics tool
• To encourage open communication for additional features and identification of deficiencies
• To identify feature enhancements to to improve OpenSOC

Open SOC is unique in being an open source project that incorporates security tools, big data capture, and machine learning into one platform. However, in order to keep up with the bad guys, the project is a work in progress, with the OpenSOC project members welcoming help with development, building enhancements, and implementation support.