[ckan-dev] problem with package importer

Thu Dec 9 10:54:35 UTC 2010

Thomas,

Just to add to Rufus' comments, if you're not using Google docs, then
take a look at the infrastructure for doing data imports in ckanext,
which handles more complicated reloading/merging updated data, reads
CSV/XLS etc. It's been used for a raft of data importers, providing
multiple examples and there are tests for it all too. Take a look at
the documentation here:
http://knowledgeforge.net/ckan/doc/ckan/loader_scripts.html

David

On 9 December 2010 09:35, Rufus Pollock <rufus.pollock at okfn.org> wrote:
> Hi Thomas,
>
> I'm cc'ing the ckan-dev list as that's where we handle 'dev' questions.
>
> On 9 December 2010 08:52, Scheel, Thomas
> <thomas.scheel at fokus.fraunhofer.de> wrote:
>> Hello ckan-Team,
>>
>> I’m Thomas from Fraunhofer Institute in Germany. For evaluation test I
>> installed a local copy of ckan and every think works fine. For testing, I
>> would like to import several datasets into the database and I think the
>> package importer would be the best for my purposes. Despite searching your
>> documentation I don’t find any information, where to find and how to use the
>> package importer. In the ckan/lib folder I only find an importer.py and an
>> spreadsheet_importer.py script, but I’m not sure how to use them.
>>
>> It would be great, if you could help me with this problem. The create
>> package function of the web frontend or the api works fine, but is kind of
>> cumbersome for a bulk of packages.
>
> Absolutely, your best approach is almost certainly to use the API and
> the ckanclient library. We have several examples of import scripts
> using ckanclient here:
>
> <http://knowledgeforge.net/ckan/ckanext/file/4a3c4d033db7/bin/>
>
> For example here's a loader that loaded data from a google docs
> spreadsheet into a CKAN instance:
>
> <http://knowledgeforge.net/ckan/ckanext/file/4a3c4d033db7/bin/ckanload-italy-nexa>
>
>> Another thing I noticed, if I’m downloading a data package via datapkg
>> download ckan://iso-3166-2-data . like the example in the manual, it is
>> working fine. But when I try to download other packages, especially in the
>> german ckan version (de.ckan.org) I always get exceptions while reading
>> information (datapkg info) or downloading the package. Is the german version
>> incompatible to datapkg or strikes the download while reading faulty or
>> incomplete records?
>
> How are you trying to search offenedaten.de. At the moment (we are
> changing this as more CKAN instances appear) you will have to set
> ckan.url in your [index:ckan] section to:
>
> ckan.url = http://offenedaten.de
>
> I was then able to successfully query e.g.:
>
> $ datapkg search ckan:// statistik
> $ datapkg info ckan://destatis-statistik-21411
> $ datapkg download ckan://destatis-statistik-21411 .
> # there will now be a /tmp/destatis-statistik-21411 directory
>
> I note many offenedaten.de packages have no download resources in
> which case datapkg download will exit silently and do nothing. This
> should probably be made more apparent (many offenedaten packages have
> no download resources at the moment!).
>
> One way to see this is to run in verbose mode:
>
> datapkg download --verbose ... ...
>
> Rufus
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
>