[ckan-dev] problem with package importer

[Rufus Pollock]

Hi Thomas,

I'm cc'ing the ckan-dev list as that's where we handle 'dev' questions.

On 9 December 2010 08:52, Scheel, Thomas
<thomas.scheel at fokus.fraunhofer.de> wrote:
> Hello ckan-Team,
>
> I’m Thomas from Fraunhofer Institute in Germany. For evaluation test I
> installed a local copy of ckan and every think works fine. For testing, I
> would like to import several datasets into the database and I think the
> package importer would be the best for my purposes. Despite searching your
> documentation I don’t find any information, where to find and how to use the
> package importer. In the ckan/lib folder I only find an importer.py and an
> spreadsheet_importer.py script, but I’m not sure how to use them.
>
> It would be great, if you could help me with this problem. The create
> package function of the web frontend or the api works fine, but is kind of
> cumbersome for a bulk of packages.

Absolutely, your best approach is almost certainly to use the API and
the ckanclient library. We have several examples of import scripts
using ckanclient here:

<http://knowledgeforge.net/ckan/ckanext/file/4a3c4d033db7/bin/>

For example here's a loader that loaded data from a google docs
spreadsheet into a CKAN instance:

<http://knowledgeforge.net/ckan/ckanext/file/4a3c4d033db7/bin/ckanload-italy-nexa>

> Another thing I noticed, if I’m downloading a data package via datapkg
> download ckan://iso-3166-2-data . like the example in the manual, it is
> working fine. But when I try to download other packages, especially in the
> german ckan version (de.ckan.org) I always get exceptions while reading
> information (datapkg info) or downloading the package. Is the german version
> incompatible to datapkg or strikes the download while reading faulty or
> incomplete records?

How are you trying to search offenedaten.de. At the moment (we are
changing this as more CKAN instances appear) you will have to set
ckan.url in your [index:ckan] section to:

ckan.url = http://offenedaten.de

I was then able to successfully query e.g.:

$ datapkg search ckan:// statistik
$ datapkg info ckan://destatis-statistik-21411
$ datapkg download ckan://destatis-statistik-21411 .
# there will now be a /tmp/destatis-statistik-21411 directory

I note many offenedaten.de packages have no download resources in
which case datapkg download will exit silently and do nothing. This
should probably be made more apparent (many offenedaten packages have
no download resources at the moment!).

One way to see this is to run in verbose mode:

datapkg download --verbose ... ...

Rufus