[okfn-help] [get.theinfo] datahub 0.8 is available

Thu Dec 3 14:27:34 GMT 2009

2009/12/2 Lukasz Szybalski <szybalski at gmail.com>:
> On Wed, Dec 2, 2009 at 2:24 PM, Jonathan Gray <jonathan.gray at okfn.org> wrote:
[...]
> Datahub is a tool that will create a new python package with some
> sample files in it to help you crawl,parse,load your data.
>
> so you start by (http://pypi.python.org/pypi/datahub/0.8.90dev)
>
> 1. paster create -t datahub
>
> this will create a skeleton of a python project that has 3 main sufolders.
>
> myapp/
> myapp/crawl
> myapp/parse
> myapp/load

I like the layout that you have developed here.

This is somewhat similar to what datapkg create <...> will do. However
we just create a very basic layout (like a simple python package) at
the moment.  An example of current datapkg package along these lines:

<http://knowledgeforge.net/econ/hg/file/tip/econdata/uk_house_prices>

One with a non-pythonic layout is:

<http://knowledgeforge.net/econ/hg/file/tip/econdata/browser_stats>

We have recently been talking quite a bit about what structure we
should use: be it none (leave it to users) or something like R or
Debian or ... We have also been asking whether we need to support
multiple structures of "just one" (tm). I'd be interested in your
thoughts here.

[...]

> So that is the basics of the datahub. At this point there is no way to
> list other datahub packages, there is no way to query for some
> keywords, there is no set hosting you need to use.
>
> datapkg on the other hand seem to do the later....query, search and
> upload/load packages?????
>
> Let me know what exactly datapkg does at this point?

Yes datapkg allows you to register packages on ckan.net, query
existing packages on ckan. We're just in the process of reworking the
install support -- this is rather more complex than in the code case
because of a need to support package payloads which are e.g. apis
rather than actual chunks of data.

Current datapkg documentation (for trunk):

<http://knowledgeforge.net/ckan/doc/datapkg/>

Instructions for installation are here if you want to give it a whirl:

<http://knowledgeforge.net/ckan/doc/datapkg/install.html>

You'll need to install from the mercurial repository to get up to date code ...

Rufus