Over time, local government open data programs have evolved from publishing large quantities of open datasets, toward improving usability and impact of open data as a tool for residents to advocate for their communities. When publishing data is the priority, metrics for success (like number of datasets published) are clear and easy to track. But as cities try to enact more effective and impactful community engagement around open data, quantifiable metrics have become much more difficult to define. As a city’s open data programs evolve, the approach to evaluating the outcomes of the program changes as well.

Many cities are now developing open data programs that encourage communities to use data to problem solve, advocate, or hold local government accountable. In an effort to help cities track and understand the potential outcomes of taking a community-centered approach to open data, we have developed a community engagement impact framework that outlines the potential impacts associated with various engagement strategies around open data.

The framework presents a logic model with a sequence of related stages that connect inputs and activities to associated outputs, outcomes and impacts. Each stage of the model has corresponding metrics that cities can track and measure over time to monitor and ultimately evaluate their efforts.

The problem

Conducting community engagement around open data requires cities to expand their work far beyond the internal maintenance of a data portal. What outcomes can cities expect if they commit additional resources, such as staff and funding, to the ongoing work around interacting and communicating with partners and community members?

Many cities struggle to track their engagement efforts or evaluate how community use of open data can improve relationships between residents and government. Quantifying and measuring impact is a difficult endeavor for any social program, but it is particularly challenging when the desired outcomes are as abstract as increasing civic engagement among constituents.

The solution

Cities should proactively conduct and track community engagement efforts that lead to quantifiable outputs and ongoing relational engagement between the city and its residents. The framework lists various inputs and activities necessary for implementing this solution. Outputs are the most immediate results of the intervention.

Examples of inputs include the target number of annual community meetings around open data or the ability to showcase successful open data user stories on the data portal webpage. Activities include launching and fostering partnerships with businesses or nonprofits that could use open data in their work. A corresponding output would be the number organizations citing open data as a source in their resources. These introduce metrics that are trackable, though some require more effort to monitor than others.

Residents gather for a community meeting. Source: Chris Martin

Expected impact

The solution will lead communities to becoming more civically engaged, using open data as a means to understand local issues, advocate, and hold local governments accountable. The framework describes outcomes and impacts that can be assessed after implementing the solution.

Outcomes and impacts, two terms that often seem synonymous, are differentiated here by their scope. Outcomes look more at the short to medium-term effects of the intervention, often associated with indicators that relay a quantifiable change in outputs (an example being the trends in dataset downloads over time). Impacts take a broader look at changes in resident behavior that go beyond uses of open data and detail the potential for enhanced civic engagement and a growing interest in data-driven governance.

Using this framework

As outcomes and impacts become more abstract, methods of assessing them require greater effort, and for many broader impacts there is no simple quantitative metric to point to the program’s success. But qualitative assessments are still worth making, and intermediary metrics can still help local governments set goals and move their engagement efforts forward. 

The degree to which local governments can apply this framework will vary depending on capacity and scope of existing community engagement efforts, but we hope that it can provide some guidance and inspiration for cities at different stages of their open data engagement efforts.

Conducting community engagement allows local governments to build relationships and gain qualitative insight on open data’s impact in the lives of residents and partner organizations. If cities do not take the additional step to invest in these efforts, they will remain in the dark as to how their commitments to transparency and open governance are successfully empowering the communities they intend to serve.