[ckan-dev] problem with package importer

Thu Dec 16 15:43:30 UTC 2010

Hello *,

first thanks for the help with the data import from spreadsheets.
I made a customized import script which inherits from SimpleGoogleSpreadsheetLoader. 
When I run the script (with --ckan-api-location=http://127.0.0.1:5000/api --ckan-api-key=xxx --google-spreadsheet-key=xxx --google-email=xxx --google-password=xxx)
It reads the fields of the spreadsheet an exits with:

Reading Google spreadsheet. Please wait...
Working area of spreadsheet: top-left (1, 1); bottom-right (6, 10).
There are 10 headings: unique_id, title, tags, notes, url, resources-0-url, resource-0-format, author, maintainer, license
There are 3 entities: amt far statistik berlin-brandenburg, bundesgesetztracker, auslanderstatistik
There are 3 metadata packages with titles extracted from the spreadsheet.
Putting 3 packages on CKAN running at http://127.0.0.1:5000/api

Traceback (most recent call last):
  File "/home/ckanuser/ckanext/bin/ckanload-aidprojdata", line 29, in <module>
    AidProjectsLoader().run()
  File "/home/ckan-user/pyenv/src/ckanclient/ckanclient/loaders/base.py", line 105, in run
    self.put_packages_on_ckan()
  File "/home/ckanuser/pyenv/src/ckanclient/ckanclient/loaders/base.py", line 121, in put_packages_on_ckan
    registered_package = self.ckanclient.package_entity_get(package['name'])
  File "/home/ckanuser/pyenv/src/ckanclient/ckanclient/__init__.py", line 322, in package_entity_get
    self.open_url(url)
  File "/home/ckanuser/pyenv/src/ckanclient/ckanclient/__init__.py", line 288, in open_url
    result = super(CkanClient, self).open_url(url, *args, **kwargs)
  File "/home/ckanuser/pyenv/src/ckanclient/ckanclient/__init__.py", line 189, in open_url
    content_type = self.last_headers['Content-Type']
  File "/usr/lib/python2.6/rfc822.py", line 388, in __getitem__
    return self.dict[name.lower()]
KeyError: 'content-type'

Does anybody know what the problem is? For me, it is very strange, bacause the script works fine for several imports and I already load some data in the database. Now it seems that the request doesn't arrive at the server side.

As python rookie I would be very thankful for any useful hint. 

Many thanks in advance

Thomas Scheel

-----Ursprüngliche Nachricht-----
Von: friedrich.lindenberg at gmail.com [mailto:friedrich.lindenberg at gmail.com] Im Auftrag von Friedrich Lindenberg
Gesendet: Donnerstag, 9. Dezember 2010 12:07
An: rufus.pollock at okfn.org; CKAN Development Discussions
Cc: Scheel, Thomas
Betreff: Re: [ckan-dev] problem with package importer

Hi Thomas,

On Thu, Dec 9, 2010 at 10:35 AM, Rufus Pollock <rufus.pollock at okfn.org> wrote:
>> Another thing I noticed, if I’m downloading a data package via datapkg
>> download ckan://iso-3166-2-data . like the example in the manual, it is
>> working fine. But when I try to download other packages, especially in the
>> german ckan version (de.ckan.org) I always get exceptions while reading
>> information (datapkg info) or downloading the package. Is the german version
>> incompatible to datapkg or strikes the download while reading faulty or
>> incomplete records?
>
> How are you trying to search offenedaten.de. At the moment (we are
> changing this as more CKAN instances appear) you will have to set
> ckan.url in your [index:ckan] section to:
>
> ckan.url = http://offenedaten.de
>
> I was then able to successfully query e.g.:
>
> $ datapkg search ckan:// statistik
> $ datapkg info ckan://destatis-statistik-21411
> $ datapkg download ckan://destatis-statistik-21411 .
> # there will now be a /tmp/destatis-statistik-21411 directory
>
> I note many offenedaten.de packages have no download resources in
> which case datapkg download will exit silently and do nothing. This
> should probably be made more apparent (many offenedaten packages have
> no download resources at the moment!).

It is also worth noting that offenedaten/de.ckan is based on an
outdated and somewhat unstable version CKAN. Daniel and I should
invest some time into upgrading this while keeping plugins and theming
intact.

Also: why is it that datapkg will only download the first listed
resource, is there anything speaking against an "--all" switch?

- Friedrich

>
> One way to see this is to run in verbose mode:
>
> datapkg download --verbose ... ...
>
> Rufus
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
>