APIs and mashups are the bread and butter of the Sunlight Labs. In fact, the official name of the the Labs is actually The Sunlight Mashup Labs, so it is perhaps not surprising that we headed off to the MashupCamp 3 at MIT, Cambridge a couple of weeks ago to geek out and to see the latest happenings, trends and players in the world of mashups and technology.

There was a lot of cool stuff demoed, debated and hacked at the camp but one theme that really stood out for me was tools that made scraping easier and more effective than ever before. In short, tools to create instant APIs, without programming — yes, really.

To be more concrete: suppose there is a website that you visit every day and you read the headlines. However, you prefer to read these stories in your favorite RSS reader but this website does not provide an RSS feed. You can use these tools to create such a feed yourself.

Or, suppose you want to buy a second-hand iBook from Craigslist and are not in a rush; you would rather wait for the right deal. Well, you could manually check the listings everyday but you would rather be alerted if an iBook comes up for sale within a certain distance from home and under a certain price. To do this programmatically, you need to grab the listings for computer ads and then you can parse them to see in any match your criteria. You can use these tools to create an API for Craigslist computer ads, under a search query of “iBook”, in a matter of minutes.

Anything that is presented on a webpage, certainly a static webpage, is something that could be scraped. The problem is that scraping is not easy. Quality of HTML structure varies considerably. While some sites always use valid W3C XHTML, use CSS with lots of descriptive class names and sensible divs that divide different sections of the page, many do not. Do a view source of a random myspace page and you will see what I mean: it is a huge mess.

So, to scrape a page when it does not have a clear or consistent structure, or to scrape a page that requires a login (say, your AOL buddy list) is not at all easy. It can be done but it takes time and, moreover, because one is scraping who knows how long that scraper will last. The content provider makes a change in page structure and your scraper has to be reworked. Finally, one has to be a coder: you need to use tools such as curl or php to get the page contents, you may need tools such as html parsers to navigate the content, and you may need competency in regex to pull out precisely what you need from the page. Finally, you may actually need a server to create a page to display the final, desired content; and not everyone has such resources. (We will see below how one can dispense with this latter requirement.) Any tools that provide instant APIs from arbitrary webpages, especially for non-progammers, has to be good news.

Two organizations in this space who demoed at Mashup Camp are Dapper and OpenKapow. Now that I’ve had a chance to play around with these technologies, I want to take this opportunity here to review them, giving an unbiased critique.

Dapper

Dapper is a US startup based in Israel founded by Eran Shir and Jon Aizen and is currently in open beta. While all services are free at the moment, large projects or commercial uses of Dapper may be charged in the future. However, if Dapper works as well as it is claimed then I can see many organizations making good use of Dapper and saving in development costs even in a fee-based structure. Why do I say this? What precisely is Dapper?

Dapper is a web-based tool and service for scraping, creating APIs from potentially any webpage and then generating output of that API in a variety of formats including HTML, XML, RSS and even Google Maps.

One starts by entering a URL and the webpage is displayed within Dapper. One can click through different pages and add each page to a “basket”. Thus, one could add say pages 1, 2, 3, 4 of a blog. Dapper then analyzes these pages to work out the structure, say what is a static header and footer and what is dynamic content. One then gets to “play”. That is, clicking on a story title (should) highlights all other story titles in the page. As such, one is confident that Dapper will grab the correct content from the page when you specify precisely what you want your API, or “Dapp”, should do.

I have to say that I had mixed success with this. While Dapper correctly identified the story titles for techcrunch.com and http://www.followthemoney.org/Newsroom/index.phtml, it did not do so on sunlightfoundation.com and it could not work out my intention of grabbing rows or columns on http://opensecrets.org/orgs/list.asp?order=A (but it could get the org titles). However, this is still a beta and Dapper certainly does work: there are hundreds of user-created Dapps that one can browse and use.

After playing, one can then progress to creating the API proper: one clicks on a desired element, gives it a name and one can then define a group, e.g. specify that this story title, this author name and this number of diggs are all related as one unit. After that, one can preview the API, i.e. check what is pulled out by the API. If all is well, one is done. Then the real fun begins…

Dapper provides the API in a impressive variety of formats: XML, HTML, RSS, Alerts, iCalendar (transforms the output of a Dapp into an iCalendar which can be used in Google Calendar, Sunbird, iCal, and other programs), Google maps (places locations directly onto a map), Google gadgets, Netvibes, image loop, email, link to another Dapp, CSV, JSON, YAML, XSL, and fork it as another Dapp. This APIs are published and hosted at Dapper (hence one doesn’t need their own server to provide an API). Dapper also allows one to define parameters for the API such as a {query} or {page}.

Some of the resultant URLs are mess though. So, Dapp allows one to define a service that provides a nice clean URL. That is, instead of say http://www.dappit.com/RunDapp?dappName=sunlightlabs&v=1&thisparam=y&thatparam=z… one can instead provide users with say http://www.dappit.com/services/sunlightlabs.>