More than 2,300 records managers and archivists from around the world gathered in Washington, DC, last week to talk about public records and managing the massive amount of new information being created by technology. Discussions at the conference made it clear that the open data community can benefit from connecting with and learning from people in the records management and archival communities. We share many of the same challenges and goals with determining how governments can best share information and preserve access to it.

Last week’s conference, Archives*Records 2014: Ensuring Access, was a joint meeting of the Council of State Archivists, Society of American Archivists, and National Association of Government Archives & Records Administrators. Topics discussed ranged from copyright law to appraising records for determining what to keep permanently, but some of the conversations most relevant to those interested in open data centered around electronic records and metadata.

How electronic records management can inform open data

It’s been clear for some time that those dealing with all stages of the life cycle of records are crucial stakeholders in advancing open data. Understanding the creation, context and value of records is central to the roles of records managers and archivists, which makes them ready to answer many of the key questions along the path to open data.

They have already thought about how to assess and address potential liabilities with the release of certain kinds of information, for instance. Records managers and archivists often handle sensitive or personally identifiable information ranging from employment records to crime statistics, and they have to decide, for each type of record, how long to keep information and what should be accessible to anyone. Creating records retention schedules and designating records series helps make this process manageable. Open data advocates and those involved with sharing information proactively online could benefit from talking to records managers and archivists about these kinds of processes in place to appropriately handle sensitive information while still sharing as much information as possible with the public.

Records managers and archivists have already thought about the costs associated with storing and sharing records, too. At last week’s conference, there was discussion of a balancing test for preserving information that could help open data advocates make assessments: does the value of keeping a record outweigh the cost of keeping it over time? That’s not always an easy question to answer, but it can provide a starting point for the conversation about what information should be permanently accessible. Deciding how to handle all of the emails from a government official’s administration, for instance, can help illustrate steps in the process of choosing what to preserve. Emails about coffee cake in the breakroom might not be of permanent value, but those directly related to decision-making clearly warrant preserving as context over time, no matter the cost.

Records managers and archivists also know, perhaps more clearly than anyone else involved in the life cycle of records, how to assess the value of information and weigh the challenges of preserving that information in a time when technology is changing rapidly, posing the threat of leaving many records in obsolete formats without the proper attention. Records managers, archivists and open data advocates should be having a dialogue around open formats and how such formats could help with access to and preservation of records throughout their life cycle.

Records managers and archivists are even looking at emerging questions such as whether, and how, to archive social media as records and how to deal with archiving massive amounts of email. Open data advocates are figuring out how to handle these records, too, and there is an opportunity for creating a comprehensive view of how to manage these new sources of information. Archiving social media and emails is something we’ve seen done successfully in a few places already and being discussed in many other situations that emphasize the importance of determining how to handle such data as records crucial to understanding the government decision-making process.

The importance of metadata

Many of the discussions at the conference about managing electronic records also emphasized the importance of metadata, which is data about data. Metadata is there to help anyone understand details about the creation and uses of the primary data. We all have a clear stake in making sure metadata is as complete, credible and useful as possible.

Some records managers and archivists talked about considering metadata as something that would be primarily for their use since they are often expected to do the research for users accessing records. Open data advocates would argue that is an outdated approach as more information is proactively released online for anyone to access and use. Records managers and archivists will need to reassess the mindset that metadata is only important to them, and open data advocates have a key role to play in helping with that shift by emphasizing the importance of usable metadata for everyone.

Education and continuing the conversation

Everyone invested in the future of access to records could benefit from shared guidance in these areas and more, and the good news is that the records management and archival communities are working on building out an online hub for exactly that purpose. A new online resource center developed by the Council of State Archivists aims to help records managers and archivists from anywhere connect and share experiences, guidance and ideas about managing electronic records. The resource center will be a living resource and grow to include helpful references such as lists of inventories that have been done cataloging state and local government records.

Open data advocates should use this resource center to strengthen the understanding of how records management, archives and open data are all part of the new life cycle of records. The State Electronic Records Program Framework that’s located in the resource center, for example, is closely aligned with the goals of open data. It looks at how to preserve digital records, ensure records are trustworthy and usable, and avoid having records saved in formats that will become technically obsolete. Knowing about, sharing, contributing to, and building on resources like this will be crucial to showing how all of our work intersects.

There is much to be learned from starting a dialogue between the open data, records management and archival communities. Both open data and the records management communities face similar challenges. We are increasingly sharing our knowledge and resources online, and now it’s time to align ourselves as groups with key roles to play in the goal of ensuring access to and preservation of records. All of our work will be stronger for it.