On Monday, September 23rd, New York City released a plan to, as Government Technology put, open “all” its data. Pursuant to section two of Local Law 11 of 2012 (§23-506 of the Administrative Code of the City of New York) — also known as NYC’s Open Data Policy — the long awaited agency compliance plan does complete the monolithic task of  listing all NYC agency public datasets, with scheduled release times of no later than the end of 2018, but there are ways it could have been more inclusive and comprehensive.

Here’s the plan:

(The dataset list is also available in CSV, JSON, PDF, RDF, RSS, XLS, XLSX, XML formats on NYC’s newly revamped open data portal).

Why the Plan Is Great

Creating a public inventory of government owned information is fundamental in data management, empowers public and media oversight, and assists in thoughtful, organized, prioritization schedules — and it’s something we haven’t seen done before at the local U.S. level. The Sunlight Foundation has consistently called for open data policy law and implementation to require a complete inventory of information holdings, and even with the passing the of U.S. federal Open Data executive order on May 9th, and its accompanying guidance which included a provision that requires U.S. agencies and interagency groups create and maintain an enterprise data inventory, these calls have thus far remained unanswered. NYC’s release of their Open Data Plan and agency public data holdings is to date, the most comprehensive list of government held information compelled by an open data policy. Moreover, New York City is the first U.S. government that has completed a comprehensive dataset inventory, beating counties, states, and the federal government to this open data milestone. To get a feel for the landscape of public information audits and indexes, check out our round-up of international and federal agency examples (and plans for other local inventories) here. NYC’s Open Data Plan, which identifies and schedules for release over 400 datasets across 44 agencies, also includes new datasets that bolster government accountability and transparency, such as Department of Corrections and Probation data, ACRIS data, Conflict of Interest data, OMB datasets, and daily procurement bids and award data. This level of data release from a local government (in 2014!) is groundbreaking and should be celebrated.

Why This Plan Could Be Better

While NYC’s open data plan is an important milestone, it bears noting that while the plan is massive (434 datasets to be exact) and includes much of NYC’s data, it does not all of the data relevant to NYC.  The data inventory covers structured data controlled by New York City agencies, but a lot of information relevant to municipal processes fall outside these bounds. Ninety-three of the 434 datasets listed are scheduled for release currently by 12/31/2018, the last day mandated by Local Law 11. This makes us wonder: Is this date a failure of ambition or one made in vain? Without transparency into how the scheduling criteria was assessed it is impossible for citizens, journalists, and other third party watchdogs to assess how ambitious the scheduled dates for release are or are not.

A closer look at  the open data plan, read in conjunction with Local Law 11 of 2012, shows that the plan only applies to certain kinds of data, doesn’t identify what data exists that is not fit for release, misses the opportunity for involving public input in the prioritization process, and obscures just how lengthy of a process NYC dataset release will be.

Let’s break these down these concepts further and explore why the tailoring of NYC’s current open data plan could be more comprehensive and is, in its current form, a far cry from “all of NYC’s data.”

The Plan Is Only as Good as the Law Behind It

NYC’s open data policy strictly and carefully defines data to what might be interpreted to a layman as numbers in columns §23-501 b. More specifically, the legal definition of NYC’s data excludes narrative forms and image files (such as designs, drawings, maps, photos, or scanned copies of original documents from its purview). This narrow definition of data provides NYC built-in protection against certain copyright issues and discourages the use of narrative records saved in .PDF files (often the enemy of open data advocates), but it also severely limits what government information can be released to the public. This definition explains why agency reports and narrative procedures, such as crime reports, the zoning process, city council meeting minutes, etc., are not included in this dataset release schedule and are not, as of this writing, hosted on the NYC open data portal. Sunlight advocates for complete datasets, with context, to provide the most legible government and provide for maximum accountability, and a lot of the time, this information is housed by the government in forms not considered for release, or inventory, under NYC’s current open data policy.

Further, NYC’s open data policy only applies to data controlled by city agencies, where “agency” (§23-501 a.) is defined as, “an office, administration, department, division, bureau, board, commission, advisory committee or other governmental entity performing a governmental function of the city of New York.” In practice, this means that NYC Board of Election data, New York Housing Authority data, Metropolitan Transportation Agency data, just to name a few datasets, are not considered under the policy or included in the dataset release schedule. While limiting the scope of open data policies to city controlled agencies is legally and practically easier to orchestrate, it also limits what city-relevant information is getting released to the public, leaving out information key to decision-making legibility, creating blind spots in accountability, and (for those not familiar with the intricacies of their municipal administration) advancing misleading expectations about what releasing “all” the data means.