[ckan-dev] problem with import of latin character

Mon May 30 14:42:16 UTC 2011

On 30 May 2011 14:49, <p.romain at cg33.fr> wrote:

>
> Hi,
>
> Here is the output from python import_data_file.py test_importer_full.xls
>
> 2011-05-30 15:10:20,880 DEBUG [ckan.lib.base] Access OK.
> 2011-05-30 15:10:20,916 DEBUG [ckan.lib.base] Retrieving request params:
> UnicodeMultiDict([('{"name": "annakarenina", "title": "A Novel By Tolstoy",
> "version": "0.7a", "url": "http://www.annakarenina.com", "n
> otes": "Some test notes\\n\\n### A 3rd level heading\\n\\n**Some bolded
> text.**\\n\\n*Some italicized text.*\\n\\nForeign characters:\\nu with
> umlaut \\u00fc\\n66-style quote \\u201c\\nforeign word: th\\u00fc
> mb\\n \\nNeeds escaping:\\nleft arrow <\\n\\n<http://ckan.net/>\\n\\n",
> "state": "active", "license_id": "other-open", "tags": "russian tolstoy",
> "groups": "david roger", "extras": {"genre": "romantic novel",
>  "original media": "book"}, "resources": [{"url": "
> http://www.annakarenina.com/download/x=1&y=2", "format": "plain text",
> "description": "Full text. Needs escaping: \\" Umlaut: \\u00fc", "hash":
> "abc123", "al
> t_url": "alt123", "size": "123"}, {"url": "
> http://www.annakarenina.com/index.json", "format": "json", "description":
> "Index of the novel", "hash": "def456", "alt_url": "alt345", "size":
> "345"}]}', u'1')])
> 2011-05-30 15:10:20,916 DEBUG [ckan.lib.base] Retrieving request POST:
> UnicodeMultiDict([('{"name": "annakarenina", "title": "A Novel By Tolstoy",
> "version": "0.7a", "url": "http://www.annakarenina.com", "not
> es": "Some test notes\\n\\n### A 3rd level heading\\n\\n**Some bolded
> text.**\\n\\n*Some italicized text.*\\n\\nForeign characters:\\nu with
> umlaut \\u00fc\\n66-style quote \\u201c\\nforeign word: th\\u00fcmb
> \\n \\nNeeds escaping:\\nleft arrow <\\n\\n<http://ckan.net/>\\n\\n",
> "state": "active", "license_id": "other-open", "tags": "russian tolstoy",
> "groups": "david roger", "extras": {"genre": "romantic novel", "
> original media": "book"}, "resources": [{"url": "
> http://www.annakarenina.com/download/x=1&y=2", "format": "plain text",
> "description": "Full text. Needs escaping: \\" Umlaut: \\u00fc", "hash":
> "abc123", "alt_
> url": "alt123", "size": "123"}, {"url": "
> http://www.annakarenina.com/index.json", "format": "json", "description":
> "Index of the novel", "hash": "def456", "alt_url": "alt345", "size":
> "345"}]}', u'1')])
> 2011-05-30 15:10:20,917 DEBUG [ckan.lib.base] Request data extracted:
> {u'name': u'annakarenina', u'title': u'A Novel By Tolstoy', u'url': u'
> http://www.annakarenina.com', u'notes': u'Some test notes\n\n### A 3
> rd level heading\n\n**Some bolded text.**\n\n*Some italicized
> text.*\n\nForeign characters:\nu with umlaut \xfc\n66-style quote
> \u201c\nforeign word: th\xfcmb\n \nNeeds escaping:\nleft arrow <\n\n<
> http://ckan
> .net/>\n\n', u'tags': u'russian tolstoy', u'state': u'active', u'version':
> u'0.7a', u'groups': u'david roger', u'license_id': u'other-open',
> u'resources': [{u'hash': u'abc123', u'description': u'Full text. Ne
> eds escaping: " Umlaut: \xfc', u'format': u'plain text', u'url': u'
> http://www.annakarenina.com/download/x=1&y=2', u'alt_url': u'alt123',
> u'size': u'123'}, {u'hash': u'def456', u'description': u'Index of the n
> ovel', u'format': u'json', u'url': u'
> http://www.annakarenina.com/index.json', u'alt_url': u'alt345', u'size':
> u'345'}], u'extras': {u'genre': u'romantic novel', u'original media':
> u'book'}}
> 2011-05-30 15:10:20,925 DEBUG [ckan.controllers.apiv1.package] Created
> object annakarenina
> 2011-05-30 15:10:20,940 ERROR [ckan.controllers.apiv1.package] unexpected
> keywords {'alt_url': u'alt123', 'size': u'123'}

These columns the spreadsheet are 'resource extras', which were introduced
in CKAN release 1.3.2. Which version of CKAN are you loading into? The debug
looks suspiciously like an old release. With the latest code I get this and
no error:

2011-05-30 15:37:42,528 DEBUG [ckan.logic.action.create] Created object
annakarenina

> We noticed that the ckanclient is calling the apiv1 : is this correct ?

API version is specified through the base location in your script. e.g. you
append '/2' to your base_location (e.g. 'http://localhost:5000/api/2') and
use API version 2. However, v1 should do the same as v2 for loading packages
- so this is not a concern here.

David

>
>
> Cordialement,*
> Pascal Romain**
> responsable domaine applicatif gestion documentaire*
> SPEC*
> Direction des Systèmes d'Information*
> 05 56 99 33 33 poste 6643*
> * <http://www.gironde.fr/>
>
>
> *Rufus Pollock <rufus.pollock at okfn.org>*
> Envoyé par : ckan-dev-bounces at lists.okfn.org
>
> 26/05/2011 14:07
> Veuillez répondre à
> rufus.pollock at okfn.org; Veuillez répondre à
> CKAN Development Discussions <ckan-dev at lists.okfn.org>
>
> A
> CKAN Development Discussions <ckan-dev at lists.okfn.org>
> cc
> y.ngarboui at cg33.fr
> Objet
> Re: [ckan-dev] problem with import of latin character
>
>
>
>
> 2011/5/26  <p.romain at cg33.fr>:
> >
> > Hi,
> >
> > We wrote a script for the bulk import of our catalogues of data packages
> but
> > we can't import data with latin characters. Is there something to set in
> the
> > python script that we are missing (encoding or something..)
>
> To add to David's comment:
>
> It would be useful to see the output from script and also log output
> from the server -- if you could run it with paster serve {config-file}
> you'll see out put from server in terminal.
>
> Rufus
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
>
>
> __________________________________________________________________
>
> Ce message et toutes les pièces jointes sont confidentiels et établis à
> l'intention exclusive de ses destinataires. Ce message ne constitue pas un
> document officiel. Seuls les documents revêtus de la signature du Président
> du Conseil Général ou d'un de ses délégataires sont de nature à engager le
> Département.
> Toute utilisation ou diffusion non autorisée est interdite. Tout message
> électronique est susceptible d'altération et le Département de la Gironde
> décline toute responsabilité au titre de ce message s'il a été altéré,
> déformé, falsifié.
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20110530/5c395803/attachment-0001.html>