Pivotal has launched the latest release of its Hadoop distribution Pivotal HD and put into general availability Gemfire XD, an in-memory database system that integrates with the Hadoop distribution and with HAWQ, the company's technology that marries Hadoop with the common database programming language SQL.

Combination of the three – Pivotal HD 2.0, HAWQ and GemFire XD – is a foundation for Pivotal's “business data lake” architecture. The idea is to abstract access to the wide variety of data from all the different sources available to an enterprise so that it looks like a single “data lake,” which can then be used by developers to build applications that take advantage of big data and analytics.

In its new Hadoop distro, Pivotal has added integration with GraphLab, an open source framework for graph analytics used by data scientists and analysts, but the most important additions happened in HAWQ.

The new MADlib machine learning library adds more than 50 in-database analytics algorithms. Also an open source project, MADlib was started by what was then Greenplum and researchers at the University of California, Berkeley.

Another big addition to HAWQ is language translation, which enables developers to use popular languages R, Python and Java to run SQL queries. Michael Cucchi, who runs product marketing for Pivotal's data solutions unit, said this feature would widen the pool of developers capable of using analytics.

“There is a deficit of people that understand and are technically adapt to working with Hadoop and leveraging advanced data analytics,” he said.

Finally, the latest release adds beta support for Parquet files, a popular storage format for Hadoop, which opens a way for HAWQ's SQL query engine to use this file type. HAWQ inherited a series of features and capabilities from Greenplum, one of which is a proprietary file type, Cucchi explained.

Parquet is open and popular and has “inherent performance characteristics inside the actual file format,” he said.

The company announced that it would integrate Pivotal HD with GemFire XD in October 2013. Today, it announced general availability of the in-memory database, integrated with the Hadoop distro.

“GemFire XD is migration of and porting of SQLFire [Pivotal's in-memory SQL database] intelligence into a technology that can access HDFS and HAWQ,” Cucchi said. HDFS, or Hadoop Distributed File System, is the way Hadoop stores data.

Pivotal was spun out of EMC and VMware – with GE taking a 10% stake – in 2013. The company combines a number of acquisitions both parent firms have made over the past several years (including database companies Greenplum and GemStone and agile software development shop Pivotal Labs) and has former VMware CEO Paul Maritz at the helm.

The company wants to help traditional enterprises use agile development to write modern software applications that take advantage of big data and analytics tools in the same way the internet giants, such as Google, Yahoo and Facebook, have been writing theirs.