As New York Mayor Bill de Blasio’s first term draws to a close, new laws passed in New York City in 2017 have made the metropolis an international trailblazer in open government data and algorithmic transparency.

One bill mandates more transparency for how New York uses algorithms in decision-making, creating a task force to examine the issue. The other bill gives the public more ability to hold the city accountable for the implementation of its landmark open data legislation.

While the algorithmic transparency bill that passed has flaws, it sets a new bar for open government in the 21st century. The New York City Liberties Union praised the passage of the final legislation as the “first in the nation” to recognize that algorithmic bias “must be subject to public scrutiny and a mechanism to remedy flaws and biases.”

Councilman James Vacca, chair of the Technology Committee, introduced the legislation, pointing to the high school selection process, New York Police Department resource allocations, fire protection initiatives and benefits eligibility determinations as potential examples.

A report authored by the Technology Committee staff in support of the legislation specifically cited ProPublica’s “machine bias” investigation, which demonstrated that racially biased algorithms  were being used within the criminal justice system.

Several legal aid groups, BetaNYC the New York Civil Liberties Union, the Brennan Center for Justice and Cornell Tech faculty members expressed support for the law at a contentious, packed hearing this fall, which revealed a rift within NYC technology community.

Technologists who work with open source technology, however, argued at the hearing that transparency could help address flaws in products and in fact help improve security.

The bill they favored, however, didn’t make it into law, with opposition from both New York City itself and industry trade groups.

The city testified at the hearing that while it appreciated the bill’s transparency goals, the bill’s initial language would raise serious security concerns, had too broad a scope applying to the inner workings of all city software, and would be too challenging to implement with unintended consequences.

“Computers do not unilaterally made decisions,” the city also stressed. “Even if it were possible to make this information available, the code is such a small part of decision-making. …On the whole algorithms supplement rather than replace the decision-making process made by city agencies.”

Tech:NYC, a policy group that represents companies in the larger New York City technology industry, criticized the bill in its original form as “unworkable,” because it could require publication of confidential and proprietary information with negative security or competitive impacts.

The algorithmic transparency legislation headed for de Blasio’s signature requires the mayor to create a task force to provide recommendations on how to share information on agencies’ automated decision-making with the public, and how to address instances where automated decision-making results in harm for people.

To date, New York City has made a project library of code from the Mayor’s Office of Data Analytics available.

Opening New York data to New Yorkers

The open data legislation that passed this fall is intended to make the city’s 2012 open data open data law sustainable beyond its original 2018 deadline, and to make compliance easier to track.

“Open data is a tremendously important tool for increasing transparency in government, and for empowering New Yorkers with the information they need to solve problems and make real change in their communities,” said de Blasio, in a statement regarding the bill.

“I want to thank the City Council Tech Committee for their ongoing partnership with the City, working with DoITT and MODA to put the greatest amount—and the best quality—open data at New Yorkers’ fingertips. I applaud the leadership taken by Chair Vacca and the rest of Council to ensure open data remains a priority for all city agencies.”

The legislation follows another recently enacted law that codifies the practice of the city publishing its budget in a machine-readable format and comes at the same time that the New York City Council announced that it is making its legislative data more accessible to third-parties with the launch of an application programming interface, a long-held priority of open data advocates.

The key part of the new open data legislation comes in the form of a requirement for a single overview of all public datasets in relation to the open data law. The law, spearheaded by then-Council Member Gale Brewer and signed by Mayor Michael Bloomberg, required that all public data sets be made available in machine-readable format on the open data portal by December 31, 2018, and the publication of annual compliance plans beginning in 2014.

At a Council hearing in September, both the administration and advocacy groups such as Reinvent Albany and BetaNYC expressed concern that an originally envisioned extension of the deadline until 2021 could be counterproductive, with Reinvent Albany noting that many agencies already appear to be “procrastinating” by scheduling data releases close to December 2018.

The final version of the legislation keeps that deadline, but requires the city to publish beginning in September 2018 a real-time overview of the status of all datasets classified as public on or after March 7, 2012 and  their compliance with specific open data publication mandates.

Until now, the lack of a single overview has resulted in “lack of transparency, reduces accountability, and reduces pressure on agencies to comply with the open data law,” Reinvent Albany testified.

Reinvent Albany and other groups have separately criticized the city for not releasing a legally mandated database of lobbying information.

Another challenge was evident at the final Technology Committee hearing that examined the latest annual agency compliance review on December 6. The hearing was chaired by Councilman Vacca, whose tenure ends in December due to term limits.

In their testimony, city officials acknowledged – and advocacy groups present stressed – that limited resources were a challenge for agencies to comply and fully realize the potential of the city’s open data law.

Out of three agencies reviewed this year, the groups singled out the New York Fire Department as making the worst impression in this year’s review. NYFD supplied incomplete information to demonstrate its compliance, as compared to the engagement demonstrated by the Department of Buildings.

In her September testimony, Brewer – now Manhattan Borough President – cited the removal of public data from federal websites almost immediately after President Trump’s inauguration as a negative precedent.

“There’s no federal law to protect public information from an administration that does not believe in facts or transparency,” she asserted. “By contrast, we are assembled here today to discuss making our transparency laws stronger and more effective.”

The new legislation does not, however, include a private right of action, which would provide grounds for legal action to compel open data publication.

This is a priority Brewer, Reinvent Albany and BetaNYC all reiterated in their September testimony.

“Were New Yorkers one day to elect a mayor like the current leader of our federal executive, open data would be endangered,” Brewer said.

“Private right of action on open data probably isn’t coming anytime soon,” John Kaehny, executive director of Reinvent Albany wrote in an email. “The City Law Department doesn’t want to volunteer to be sued. We haven’t decided whether to pursue it with the new City Council next session.”