[okfn-help] Request for scraping and keeping in closer contact

Mon May 10 15:05:32 BST 2010

Dear Aidan and Julian,

This is following up a conversation I've had with Aidan just now in Newcastle :)

1. Can you guys do some scraping of datasets out of your 4IP allocated work:

Top 3 from here would be very useful:

<http://www.ckan.net/package/hmrc-national-statistics-income-tax-and-personal-incomes>

2. Closer integration of CKAN and scraperwiki:

  * Would like to have scraperwiki datasets automatically registered
into CKAN. Use of API in either direction.
  * Have a "scrape this dataset using scraperwiki" option on ckan.net
packages. Suggest this could be a plugin to CKAN.

3. Licensing of datasets on scraperwiki (different from the code!).
Please can you default license on scraperwiki datasets to e.g. the
Open Database License <http://www.opendatacommons.org/licenses/odbl/>
(with appropriate disclaimer saying: "This license only applies to the
rights the scraper of the dataset has in this dataset. If there are
other rights (e.g. of original creator of the data) these are
obviously not covered by this license -- you can only license what you
own!").

4. Talking again about sharing/reuse of tools e.g. in relation to
"Artefact" store (plain old file storage + a bit of metadata) that
Julian and I talked about first last Summer! Also worth talking more
generally about overlap in tools/problems. E.g. We have done a lot of
work over last 6 months at our end trying out key/value stores and
other nosql/graph solutions. See
<http://wiki.okfn.org/SoftwareTools/Databases> and
<http://knowledgeforge.net/okfn/tasks/wiki/ServiceArchitecture>

5. Further to last point: we should find a place to share information
more about what we are up to. Is there a shared mailing list we could
use, or at least a way of having montly catchup chats? (The OKF have a
regular irc meeting on Tuesdays from 6pm on #okfn on irc.oftc.net)

Rufus