[ckan-dev] problem with import of latin character

Tue May 31 12:14:32 UTC 2011

Hi David,

Thank you very much for taking some time for looking into our problems.
If I believe the CHANKGELOg.txt file we use now the v1.3.2 version. 
I am a bit confused about the version because the web page displays a 
v1.4.a mention
Maybe I messed up the ckan distribution but I am not sure if I =should do 
an upgrade or not.
Actually our problem of latin character disapeared although we did not 
change anything to the code.

As you will see in a different post regarding production deployment, I am 
a bit lost in the process of setting up a production environnement with 
CKAN...

Cordialement,
Pascal Romain
responsable domaine applicatif gestion documentaire
SPEC
Direction des Systèmes d'Information
05 56 99 33 33 poste 6643

David Read <david.read at okfn.org> 
Envoyé par : ckan-dev-bounces at lists.okfn.org
30/05/2011 16:42
Veuillez répondre à
CKAN Development Discussions <ckan-dev at lists.okfn.org>

A
CKAN Development Discussions <ckan-dev at lists.okfn.org>
cc
ckan-dev-bounces at lists.okfn.org, y.ngarboui at cg33.fr
Objet
Re: [ckan-dev] problem with import of latin character

On 30 May 2011 14:49, <p.romain at cg33.fr> wrote:

Hi,

Here is the output from python import_data_file.py test_importer_full.xls

2011-05-30 15:10:20,880 DEBUG [ckan.lib.base] Access OK.
2011-05-30 15:10:20,916 DEBUG [ckan.lib.base] Retrieving request params: 
UnicodeMultiDict([('{"name": "annakarenina", "title": "A Novel By 
Tolstoy", "version": "0.7a", "url": "http://www.annakarenina.com", "n
otes": "Some test notes\\n\\n### A 3rd level heading\\n\\n**Some bolded 
text.**\\n\\n*Some italicized text.*\\n\\nForeign characters:\\nu with 
umlaut \\u00fc\\n66-style quote \\u201c\\nforeign word: th\\u00fc
mb\\n \\nNeeds escaping:\\nleft arrow <\\n\\n<http://ckan.net/>\\n\\n", 
"state": "active", "license_id": "other-open", "tags": "russian tolstoy", 
"groups": "david roger", "extras": {"genre": "romantic novel",
 "original media": "book"}, "resources": [{"url": "
http://www.annakarenina.com/download/x=1&y=2", "format": "plain text", 
"description": "Full text. Needs escaping: \\" Umlaut: \\u00fc", "hash": 
"abc123", "al
t_url": "alt123", "size": "123"}, {"url": "
http://www.annakarenina.com/index.json", "format": "json", "description": 
"Index of the novel", "hash": "def456", "alt_url": "alt345", "size": 
"345"}]}', u'1')])
2011-05-30 15:10:20,916 DEBUG [ckan.lib.base] Retrieving request POST: 
UnicodeMultiDict([('{"name": "annakarenina", "title": "A Novel By 
Tolstoy", "version": "0.7a", "url": "http://www.annakarenina.com", "not
es": "Some test notes\\n\\n### A 3rd level heading\\n\\n**Some bolded 
text.**\\n\\n*Some italicized text.*\\n\\nForeign characters:\\nu with 
umlaut \\u00fc\\n66-style quote \\u201c\\nforeign word: th\\u00fcmb
\\n \\nNeeds escaping:\\nleft arrow <\\n\\n<http://ckan.net/>\\n\\n", 
"state": "active", "license_id": "other-open", "tags": "russian tolstoy", 
"groups": "david roger", "extras": {"genre": "romantic novel", "
original media": "book"}, "resources": [{"url": "
http://www.annakarenina.com/download/x=1&y=2", "format": "plain text", 
"description": "Full text. Needs escaping: \\" Umlaut: \\u00fc", "hash": 
"abc123", "alt_
url": "alt123", "size": "123"}, {"url": "
http://www.annakarenina.com/index.json", "format": "json", "description": 
"Index of the novel", "hash": "def456", "alt_url": "alt345", "size": 
"345"}]}', u'1')])
2011-05-30 15:10:20,917 DEBUG [ckan.lib.base] Request data extracted: 
{u'name': u'annakarenina', u'title': u'A Novel By Tolstoy', u'url': u'
http://www.annakarenina.com', u'notes': u'Some test notes\n\n### A 3
rd level heading\n\n**Some bolded text.**\n\n*Some italicized 
text.*\n\nForeign characters:\nu with umlaut \xfc\n66-style quote 
\u201c\nforeign word: th\xfcmb\n \nNeeds escaping:\nleft arrow <\n\n<
http://ckan
.net/>\n\n', u'tags': u'russian tolstoy', u'state': u'active', u'version': 
u'0.7a', u'groups': u'david roger', u'license_id': u'other-open', 
u'resources': [{u'hash': u'abc123', u'description': u'Full text. Ne
eds escaping: " Umlaut: \xfc', u'format': u'plain text', u'url': u'
http://www.annakarenina.com/download/x=1&y=2', u'alt_url': u'alt123', 
u'size': u'123'}, {u'hash': u'def456', u'description': u'Index of the n
ovel', u'format': u'json', u'url': u'
http://www.annakarenina.com/index.json', u'alt_url': u'alt345', u'size': 
u'345'}], u'extras': {u'genre': u'romantic novel', u'original media': 
u'book'}}
2011-05-30 15:10:20,925 DEBUG [ckan.controllers.apiv1.package] Created 
object annakarenina
2011-05-30 15:10:20,940 ERROR [ckan.controllers.apiv1.package] unexpected 
keywords {'alt_url': u'alt123', 'size': u'123'}

These columns the spreadsheet are 'resource extras', which were introduced 
in CKAN release 1.3.2. Which version of CKAN are you loading into? The 
debug looks suspiciously like an old release. With the latest code I get 
this and no error:

2011-05-30 15:37:42,528 DEBUG [ckan.logic.action.create] Created object 
annakarenina

We noticed that the ckanclient is calling the apiv1 : is this correct ?

API version is specified through the base location in your script. e.g. 
you append '/2' to your base_location (e.g. 'http://localhost:5000/api/2') 
and use API version 2. However, v1 should do the same as v2 for loading 
packages - so this is not a concern here.

David

Cordialement,
Pascal Romain
responsable domaine applicatif gestion documentaire
SPEC
Direction des Systèmes d'Information
05 56 99 33 33 poste 6643

Rufus Pollock <rufus.pollock at okfn.org>
Envoyé par : ckan-dev-bounces at lists.okfn.org
26/05/2011 14:07

Veuillez répondre à
rufus.pollock at okfn.org; Veuillez répondre à
CKAN Development Discussions <ckan-dev at lists.okfn.org>

A
CKAN Development Discussions <ckan-dev at lists.okfn.org>
cc
y.ngarboui at cg33.fr
Objet
Re: [ckan-dev] problem with import of latin character

2011/5/26  <p.romain at cg33.fr>:
>
> Hi,
>
> We wrote a script for the bulk import of our catalogues of data packages 
but
> we can't import data with latin characters. Is there something to set in 
the
> python script that we are missing (encoding or something..)

To add to David's comment:

It would be useful to see the output from script and also log output
from the server -- if you could run it with paster serve {config-file}
you'll see out put from server in terminal.

Rufus

_______________________________________________
ckan-dev mailing list
ckan-dev at lists.okfn.org
http://lists.okfn.org/mailman/listinfo/ckan-dev

__________________________________________________________________

Ce message et toutes les pièces jointes sont confidentiels et établis à 
l'intention exclusive de ses destinataires. Ce message ne constitue pas un 
document officiel. Seuls les documents revêtus de la signature du 
Président du Conseil Général ou d'un de ses délégataires sont de nature à 
engager le Département. 
Toute utilisation ou diffusion non autorisée est interdite. Tout message 
électronique est susceptible d'altération et le Département de la Gironde 
décline toute responsabilité au titre de ce message s'il a été altéré, 
déformé, falsifié.

_______________________________________________
ckan-dev mailing list
ckan-dev at lists.okfn.org
http://lists.okfn.org/mailman/listinfo/ckan-dev

_______________________________________________
ckan-dev mailing list
ckan-dev at lists.okfn.org
http://lists.okfn.org/mailman/listinfo/ckan-dev

__________________________________________________________________

Ce message et toutes les pièces jointes sont confidentiels et établis à l'intention exclusive de ses destinataires. Ce message ne constitue pas un document officiel. Seuls les documents revêtus de la signature du Président du Conseil Général ou d'un de ses délégataires sont de nature à engager le Département. 
Toute utilisation ou diffusion non autorisée est interdite. Tout message électronique est susceptible d'altération et le Département de la Gironde décline toute responsabilité au titre de ce message s'il a été altéré, déformé, falsifié.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20110531/28754ba8/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 7254 bytes
Desc: not available
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20110531/28754ba8/attachment-0001.jpe>