The Defense Advances Research Projects Agency (DARPA) is promoting the creating of open-source software tools that can help with the processing and analysis of big data, providing US$3m in funding to software company Continuum Analytics.

The company is using an open source programming language called Python, which is currently used by the oil and gas, physics and signal processing industries.

The funding, according to Continuum Analytics President Peter Wang, will take Python’s abilities one step further, helping to develop the NumPy extension, which adds support for large, multi-dimensional arrays and matrices, and SciPy, the open-source library of algorithms, to create visualization techniques for the interactive exploration of large multi-dimensional data sets.

DARPA’s XDATA program is focused heavily on the warfighting environment, where a “virtual” net of sensors and communications systems offer levels of battlefield awareness.

It said in times of battle, data usage can see extreme spikes, and current Department of Defence systems cannot handle or analyse the information that results from this.

“To maximize mission effectiveness and enhance national security, the Department of Defense (DoD) is now challenged to more efficiently fuse, analyze and disseminate the massive volumes of data this network produces,” DARPA said on its website.

“The volume and characteristics of the data, and the range of applications for data analysis, require a fundamentally new approach to data science, analysis and incorporation into mission planning on timelines consistent with operational tempo.”

DARPA said projects born out of its XDATA funding, however, will also be able to be used by other US government agencies.

“XDATA is a cornerstone of the government’s big data initiative, and we are excited to be a part of bringing open-source tools to the defense industry,” Wang said.

“The tools we develop through this project will give DARPA a solid foundation for continued innovation in the rapid analysis and visual exploration of massive, interconnected data from heterogeneous sources.”

Continuum Analytics, which is led by Wang and Dr Travis Oliphant (CEO), both known for their open-source expertise through Python, will develop new tools – Blaze, Numba and Bokeh - as part of the new XDATA project that will make NumPy and SciPy useful in the big data domain.

“As part of DARPA’s XDATA project, the team will work on a new dynamic visualization system for interactive visual exploration of large, complex data sets,” Continuum Analytics said.

“Using a novel Python-based approach, Continuum Analytics will address scalability, interactivity and extensibility at its core, while maintaining a simple conceptual model for non-programmer end users.”   

DARPA program manager Chris White said XDATA plans on releasing open-source software toolkits that can help with bring the applied mathematics, data visualization and computer science communities together to find solutions to the challenges being raised by big data.

“It's a great time to leverage recent commercial and academic advances in processing large amounts of data for analysis,” White said. 
“We are calling on all technical communities with expertise in this area to help us ensure our men and women in uniform have the benefit of the best information we can provide.”