The US National Academy of Sciences has commissioned a study on long-term cold storage solutions for the Office of the Director of National Intelligence.

The 'rapid expert consultation,' available in full here, looks at different archival data storage technologies for the intelligence community (IC) as the data storage requirements soar.

"In the near future, IC member agencies expect to maintain amounts of data at a scale comparable to that of a large corporation like Meta or Amazon," the report states, suggesting surveillance data could rival that of an entire hyperscale businesses.

"The IC has the potential to be one of the largest customers for cold data storage because of its wide-ranging need for information.

It's important to note, however, that some of the authors of the report declared competing interests; in particular, one is employed by Microsoft, is developing the Project Silica cold storage solution.

Another works for a company that contracts with Ceramic Data Solutions, which is tied to the Cerabyte cold storage product. Both are detailed in the report as potential future solutions.

What we have now

Under current technologies, the report looks at hard disk drives (HDDs), noting that heat-assisted magnetic recording (HAMR) advancements could "produce hundreds of terabytes (TB) of HDD capacity per unit."

The report continues: "The current HDD price on average is expected to be around $0.015/GB. With technological advancements like HAMR, cost decline is expected to continue. The expectation for HAMR is between 15–25 percent compound annual growth rate. That said, the super-paramagnetic effect, a physical limit on the amount of data that can be stored on a hard disk, poses long-term challenges to the future scaling rates of HDDs."

HDDs only last around 5-10 years, but the researchers believe that the IC could request that HDD manufacturers produce specially designed HDDs with extended lifespans.

The next technology analyzed is magnetic tape, which primarily follows the magnetic linear tape open (LTO) open-format standard. LTO releases have historically occurred on a roughly 3–4-year cadence, each time roughly doubling storage capacities.

That improvement has slowed as of late, but LTO members believe that they have a decade at least before they hit the same super-paramagnetic effect limits.

Tapes can last 30 years if stored properly, and are easy to dispose of. However, they are 'write once, read many' technologies.

Next, the report looks at SSDs, which are the "workhorse for primary storage today." It adds: "It is to be expected that NAND flash will scale to thousands of layers and through packaging innovations as well as multi-wafer-to-wafer bond structures. For example, Samsung has discussed the possibility of the capacity of a single SSD increasing to 1PB, with pricing to advance to below $20/TB by the end of this decade."

Within the decade, the researchers predict rack capacities of more than 100PB. However, SSDs are not suitable for long-term storage.

Similarly, magnetoresistive random access memory (MRAM) is not optimized or suitable for long-term storage. "However, MRAM may have a role to play as part of a larger storage system that requires extremely quick data recall," the report says.

More long-term storage focused are optical storage systems like Blu-ray, which Meta has used in the past for its cold storage. "This effort seems to have been quietly abandoned," the researchers say, noting limitations in data storage density.

The future

The report profiles a handful of upcoming technologies that could handle the coming wave of data.

"There are only two candidate technology domains that hold promise to meet these scaling goals: organic storage, with the lead technology being DNA data storage, and inorganic storage, with the lead technology being optical or particle data storage, such as Microsoft’s Project Silica and Cerabyte’s Ceramic Nano Memory."

It adds that the US government should fund disruptive new innovative data storage technologies, pointing to the existing Molecular Information Storage Technologies (MIST) program out of the Intelligence Advanced Research Projects Activity.

MIST aims to explore the use of scalable sequenced-controlled polymers for long-term data storage.

DNA storage could offer "tremendous density" and last for thousands of years, but the technology is still too expensive and slow. The MIST program aims to make DNA data storage at 1TB/system at $1/GB for enterprise archival use with end-to-end workflows on a tabletop by 2025.

Microsoft is a little further along with its Project Silica, which uses a femtosecond laser to encode data in fused silica glass by creating layers of three-dimensional nanoscale gratings and deformations at various depths and angles.

In 2023, the company said that it was now capable of storing 7TB on one glass sheet. Once the data is encoded on the glass, it requires no further energy to maintain its state for the next 10,000 years.

The product is in the prototype phase, but the company plans to roll it out to Azure data centers in the future.

Rival Cerabyte uses a glass substrate with ceramic nano-coating as a data storage medium. "Using femtosecond laser technology, initial densities of 125GB per data medium with 100 nm bit size, data center rack densities up to 100PB are achievable," the report says.

"This can be scaled to 1TB+ per data medium with 30 nm bit size, translating to data center rack densities of up to 1EB using particle beam technology, which has been demonstrated in proof-of-concept studies."

Should helium ion beam technology scale bit size down to 3 nm, this could "enable up to 100EB data center rack storage capacities by the middle of this century."

Cerabyte plans to work with research laboratories this year, which should provide more insight into its product roadmap.

The need

Even outside the IC's desire to store more than ever, there are many industry observers that are concerned that the desires for data storage are set to dramatically outstrip what current technologies can store.

"The fear of vertical market failure is driven by the slowing rate of improvements to HDD and tape storage and the need for storage providers to continue to compete on storage pricing," the report says. "This leads to smaller profit margins on storage, which disincentives manufacturers from assuming innovation risks when investing in new technology."

At the same time, "a small number of companies have come to dominate the zettabyte-scale archival storage market. The supply and demand side consolidation increases the risk of a single point of failure that could disrupt the entire archival data storage media marketplace."

The authors again call for more government support to help fund the future of storage.

Back in 2022, we profiled Silica, Cerabyte, DNA storage, and other long-term storage solutions in an award-winning profile of the last data centers. Give it a read today.