IBM announced a dual approach to big data management with BLU Accelerator and IBM PureData System for Hadoop which combines IBM InfoSphere BigInsights and IBM System X hardware.

The PureData System for Hadoop H1001 includes developer tools, analytic functions, and administration and management capabilities, as well as the latest versions of Hadoop and associated projects.

BLU Acceleration

“BLU Acceleration” which is the work of hundreds of IBM developers and researchers in labs around the world, is an industry-first innovation which combines techniques to improve analytical performance and simplify administration.

BLU Acceleration software extends the capabilities of traditional in-memory systems – which allows data to be loaded into Random Access Memory instead of hard disks for faster performance – by providing in-memory performance even when data sets exceed the size of the memory. During testing, some queries in a typical analytics workload were more than 1000 times faster when using the combined innovations of BLU Acceleration.

Innovations in BLU Acceleration include “data skipping,” which allows the ability to skip over data that doesn’t need to be analyzed, such as duplicate information; the ability to analyze data in parallel across different processors; and greater ability to analyze data transparently to the application, without the need to develop a separate layer of data modeling. Another industry-first advance in BLU Acceleration is called “actionable compression,” where data no longer has to be decompressed to be analyzed.

IBM PureData System for Hadoop features

Exploratory environment for data analysts to help understand new and emerging data sources; a landing area for data from other sources, enabling aggregation, initial integration, visualization, and exploration before data moves to other parts of the analytic ecosystem; an integrated management console for the entire system; built-in analytics and enterprise functionality, on top of Hadoop technology, to help meet big data enterprise requirements; integrated data archiving and transfer capabilities with connectivity to enterprise data warehouse systems

Integrates with IBM DB2, IBM Netezza , IBM PureData System for Analytics, and IBM InfoSphere Guardium.

Big data customers

BNSF Railway Company, one of the largest freight rail transportation networks in North America, is using IBM BLU Acceleration to more quickly understand the vast amounts of data from the organization’s 1700 servers that tracks maintenance, weather, scheduling, inventory, safety and deliveries.

BNSF operates more than 1,400 trains a day on 32,500 route miles of track in 28 states and two Canadian provinces.

Kent Collins, Database Solutions Architect, BNSF Railway Company, said: “Working with IBM we’re performing tasks more quickly than ever before, for example, one of the queries improved over 100-fold, and our storage consumption went down by about 10 times. One of the things that impressed us the most about BLU Acceleration is its simplicity. We just load the data and run queries.”

The new system can reduce from weeks to minutes the ramp-up time organizations need to adopt enterprise-class Hadoop technology with powerful, easy-to-use analytic tools and visualization for both business analysts and data scientists. In addition, it provides enhanced Big Data tools for monitoring, development and integration with many more enterprise systems.

“Big data is about using all data in context at the point of impact,” said Bob Picciano, general manager, IBM Information Management. “With the innovations we are delivering, now every organization can realize value quickly by leveraging existing skills as well as adopt new capabilities for speed and exploration to improve business outcomes.”

Kelley Blue Book, www.kbb.com, a US provider of new and used car information, will be evaluating the new PureData System for Hadoop to analyze click stream data created by users on its website. The company will be able to analyze this information, including social media data, to see what topics visitors care most about, such as used and new-vehicle prices, accident reports, safety recall and warranty data, and car shopper reviews.

“Kelley Blue Book collects all kinds of data from various sources, so managing the efficiency of data is critical to grow our business,” said Steve Chow, vice president of technology and data intelligence for Kelley Blue Book. “We see many opportunities to leverage the PureData System for Hadoop as a strategic platform to expand on our analytic ecosystem and tap the value of social media, text and machine data to get a better view of our consumers and customers to improve their overall experience on KBB.com.”

Hadoop is open-source software used to organize and analyze vast amounts of structured and unstructured data, such as posts to social media sites, digital pictures and videos, online transaction records, and cell phone location data.