National Library of New Zealand Preserve Digital Heritage with Hitachi Solutions

Overview

Challenge

• Collect, share and preserve New Zealand’s digital heritage cost-effectively, while meeting international digital preservation standards.

Solution

• Collect, share and preserve New Zealand’s digital heritage cost-effectively, while meeting international digital preservation standards.

Outcome

• Robust data management with rich, custom metadata capabilities and intelligent archiving
• Provable chain of custody and guaranteed integrity.
• Data loss prevention, disaster recovery and geographic redundancy.
• Innovation and technology enables National Library of New Zealand to preserve and manage digital heritage collections, ensuring data integrity and authenticity.

Challenge

The National Library of New Zealand builds collections of social, cultural, creative, economic, scientific and other outputs and outcomes of New Zealanders. Its job is to preserve the country’s history and cultural identity, and make these collections available to researchers, historians, teachers, students and others, now and in the future.

The National Library of New Zealand’s data center was aging, and the city of Wellington had been hit by a serious earthquake. In addition to drivers such as geographic redundancy and disaster recovery, the library undertook a project to optimize digital storage. This project was set to address increasing storage needs, reduce the cost of storage and find a future-proofed solution for its significant data management requirements.

“As a collecting institution, our storage requirements will only ever grow. We need to make the process more affordable so we can collect even more digital treasures to share with New Zealanders in the future,” said Bill Macnaught, National Librarian.

The National Digital Heritage Archive (NDHA) houses the National Library’s “born digital” and digitized collections. The NDHA is an ecosystem, about a petabyte in size, comprising end-to-end digital preservation workflows for over 200 discrete formats.

“We take physical objects, digitize them and preserve those digital versions. And we must also collect and preserve what is called ‘born digital’ material, material that has only ever been in a digital form,” said Program Director, Preservation Research, Steve Knight.

He and his team are responsible for the long-term safekeeping of digital heritage content that has been identified and collected as being published in New Zealand in digital form.

“We also collect the .nz web domain and have over five million pages of historical newspapers. The content we need to collect is only going to grow both in terms of volume and complexity,” said Knight.

When it comes to preserving the authenticity of a digital collection, every change to, or impact on, an object must become part of the bundle of information that is stored with that object so its provenance can be verified and trusted.

“Our job is to make sure that in 100 years’ time the person using our digital archive has the confidence and trust in the National Library of New Zealand to accept that each object is what it is supposed to be. And, if work has been done to ensure ongoing accessibility — say moving from WordStar (a word processing application commonly used in the 1980s) to PDF — they can see the record of every change stored with that digital object,” said Knight.

Solution

In 2011, the New Zealand Government mandated that its departments use one of three sovereign data centers, which includes Revera. In 2015, the library completed transfer of all its permanent digital collections to the Revera data center under an infrastructure-as-a-service model. Looking to reduce its costs for digital storage and gain control of its unstructured data, the library decided to undertake a greenfield project with Hitachi Vantara and Hitachi Vantara Partner Revera.

Knight and his team believed that object storage would be a scalable and cost-effective solution. They also needed a robust data management layer so the library could ensure that the integrity of its digital objects is maintained over time. While they could see the value of the cloud, they wanted a private cloud option that would service their particular needs as a government department.

The solution is end-to-end, 900TB, integrated object storage underpinned by Hitachi Content Platform, which has been white-labeled as Cascade and is delivered through Revera as a service.

There are two fully virtual instances of Hitachi Content Platform, running in two Revera data centers. The dual-site design provides file object protection and, for the first time, the National Library of New Zealand has a fully redundant data recovery site.

Outcome

“This collection belongs to all New Zealanders and technology makes it possible, at last, to make it very easy for any New Zealander to get access to the wealth of knowledge that we have collected on their behalf,” said Macnaught.

When the building next door had to be demolished following the earthquake in 2016, the need for geographic redundancy became very real for the library. But the benefits go well beyond fulfilling disaster recovery requirements according to Knight, who had been tasked with finding cost savings for the organization, and New Zealand taxpayers. “It is certainly not an easy or common thing to have a greenfield technology project that is actually going to save you money,” said Knight. “We have a few months to go, but the expectation is that the material cost savings we envisioned at the beginning of the project will be realized.”

The solution offers the library the ability to store, archive and access unstructured data. It can optimize storage with automated compression, rebuild large data sets quickly when necessary, and reduce costs overall.

Pairing intelligent archiving capabilities with advanced metadata features and tools that allow for tagging digital objects with custom metadata means that the library can fulfill its responsibility to preserve the digital provenance of its collections. The chain of custody for each digital object can be proven, and the write once, read many (WORM) storage guarantees authenticity, thanks to file object protection and versioning.

“National Library of New Zealand, working with Hitachi Vantara and Revera, has co-created a solution at the cutting edge of digital storage, to support the long-term safekeeping of digital cultural heritage. This will help ensure that it is available, accessible, meaningful and authentic for generations to come,” said Knight.