[ckan-dev] corupted JSON-string with API search

[Scheel, Thomas]

Hello devs,

 

after testing the new CKAN 1.7.1 we found some strange handling of the values in some extra fields. We store some JSON arrays or objects in extra fields like this:

"extras": {

                "comments":  [

                               {

                                               "date": "Tue Jan 31 12:01:53 GMT 2012",

                                               "comment": "comment 123",

                                               "userId": "10875"

                               },

                               {

                                               "comment": "lorem ipsum ertzui",

                                               "date": "Wed Aug 15 12:26:12 CEST 2012",

                                               "userId": "niko3"

                               }

                ], 

                ...

}

which is no problem so far.

When we call the dataset via api/rest/dataset/... CKAN returns the correct result, but when we call the dataset with the search API ith all_fields=1 parameter CKAN returns this corrupted string: 

 

"comments": "{u'comment': u'comment 123', u'date': u'Tue Jan 31 12:01:53 GMT 2012', u'userId': u'10875'} {u'date': u'Wed Aug 15 12:26:12 CEST 2012', u'comment': u'lorem ipsum ertzui', u'userId': u'niko3'}",

 

As you can see the array is removed with all comma separators between the array fields. Also all keys and values were suffixed with a u so it is actually no valid JSON anymore.

 

I'm not sure what causes this behavior but it is very unsatisfying because it destroys the functionality to add your own data to extra fields like it was in older releases. 
Is there any possibility that CKAN returns the correct result in a valid JSON syntax like it does in get dataset (api/rest/dataset/...) and why CKAN manipulates the value of the data field?
 
When I searched the documentation what may cause this behavior I found this sample result in http://docs.ckan.org/en/latest/apiv3.html 
 
 {'count': 2,

'results': [ { <snip> }, { <snip> }],

'search_facets': {u'tags': {'items': [{'count': 1,

                                        'display_name': u'tolstoy',

                                        'name': u'tolstoy'},

                                       {'count': 2,

                                        'display_name': u'russian',

                                        'name': u'russian'}

                                      ]

                            }

                  }

}

 

Which is actually no valid JSON too and may cause parsing errors of JSON frameworks or parsers.

 

Thanks in advance.

 

Best / Viele Grüße,

 

Thomas Scheel

---

Fraunhofer-Institut für Offene 

Kommunikationssysteme FOKUS 

Kompetenz-Zentrum eGovernment

 

Kaiserin-Augusta-Allee 31

10589 Berlin, Deutschland

Tel.  + 49 30 3463 9225

E-Mail: Thomas.Scheel at fokus.fraunhofer.de <mailto:Thomas.Scheel at fokus.fraunhofer.de> 

Internet: www.fokus.fraunhofer.de <http://www.fokus.fraunhofer.de> 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20120815/9d752f09/attachment.html>