[ckan-discuss] CKAN CSV alert

[David Read]

Christos,

Very interesting to see your linked data. Looks great! Will Waites has
been doing something similar and it will be interesting to see the
fruits of these.

David

On 8 June 2010 00:54, koumenides c.l. (clk1v07) <clk1v07 at ecs.soton.ac.uk> wrote:
> Hi David,
>
> Thanks for the update. I think it would also be helpful if you could keep the order of the columns consistent across different releases. But I think this is something that you have already suggested.
>
> Everything else is perfectly fine on our end. The CSV files parse without problems. We have a linked data version @ http://bagatelles.ecs.soton.ac.uk/psi/federator/data.gov.uk (you can use the URIs from data.gov.uk to access the records) which we are working on to integrate with other catalogues.
>
> Thanks again for the update.
>
> Regards,
> Christos
> ________________________________________
> From: d.t.read at gmail.com [d.t.read at gmail.com] On Behalf Of David Read [david.read at okfn.org]
> Sent: 07 June 2010 16:28
> To: koumenides c.l. (clk1v07)
> Cc: rufus.pollock at okfn.org; Jonathan Gray; CKAN discuss
> Subject: Re: [ckan-discuss] CKAN CSV alert
>
> Christos,
>
> Related to your enquiry, I've done some work on the CSV representation
> of the data.gov.uk data, as found at: http://ckan.net/dump/
>
> The escaping of quote marks is now improved. The CSV imported fine
> into OpenOffice, but I understand that the mix of escaping and
> quotation characters in the specific example you gave may well fox all
> but the most sophisticated of parsers. I've improved the data itself,
> avoiding extraneous slashes and quotes (which were down to mis-reading
> the escape characters during import from another database), and the
> record of interest should now parse fine, now looking like this: The
> """"lower quartile"""" property
>
> There are columns for up to 128 resources per package, which is a lot,
> but this magnitude is needed for about a dozen packages, which have
> figures released weekly, going back a few years. I agree this bulks
> out the CSV file, since most records don't use these columns, but such
> is the CSV format. If we just used a single column with JSON-formatted
> resources (your suggestion), then you might as well use the fully JSON
> formatted file that we supply.
>
> To make it more readable to the human-eye, I've sorted the ordering of
> the columns into a sensible order (rather allowing the random one) so
> that you get: "id","name","title","version","url","author","author_email"
> etc. This is also changed for the JSON file.
>
> One last thing: I've changed the file compression from gzip to zip, so
> that Windows machines can open these files more easily.
>
> Many thanks for the feedback - I hope these changes cover your wishes.
> Any more comments are most welcome.
>
> David
>
> On 12 May 2010 14:38, koumenides c.l. (clk1v07) <clk1v07 at ecs.soton.ac.uk> wrote:
>> Hi,
>>
>> Thanks for your response. I will keep you posted of any other issues that might arise :)
>>
>> Best,
>> Christos
>> ________________________________________
>> From: okfn.rufus.pollock at googlemail.com [okfn.rufus.pollock at googlemail.com] On Behalf Of Rufus Pollock [rufus.pollock at okfn.org]
>> Sent: 12 May 2010 13:43
>> To: Jonathan Gray
>> Cc: koumenides c.l. (clk1v07); CKAN discuss
>> Subject: Re: CKAN CSV alert
>>
>> Dear Christos,
>>
>> Thanks for reporting this. We'll look into this and try to get this
>> fixed asap. In the mean time you could try using the JSON dump which
>> is also available.
>>
>> Rufus
>>
>> On 12 May 2010 13:21, Jonathan Gray <jonathan.gray at okfn.org> wrote:
>>> Thanks for flagging this up, Christos! I'm copying this to our
>>> ckan-discuss list.
>>>
>>> Rufus, David, John: any ideas what might be up?
>>>
>>> All the best,
>>>
>>> Jonathan
>>>
>>> On Wed, May 12, 2010 at 1:58 PM, koumenides c.l. (clk1v07)
>>> <clk1v07 at ecs.soton.ac.uk> wrote:
>>>> Hi,
>>>>
>>>> I believe there is an error in all the CSV files of the CKAN dump - after - the January release.
>>>>
>>>> The error I find while parsing the files is this thing ->  The \\"lower quartile\"\"
>>>> appearing inside an element, which breaks everything after.
>>>>
>>>> The same record has a name "ratio_of_lower_quartile_workplace_earnings_to_lower_quartile_house_prices".
>>>>
>>>> Regards,
>>>>
>>>> Christos Koumenides
>>>>
>>>> School of Electronics and Computer Science
>>>> University of Southampton
>>>> Southampton, SO17 1BJ
>>>> United Kingdom
>>>> ______________________
>>>> clk1v07 at ecs.soton.ac.uk
>>>> clk08r at ecs.soton.ac.uk
>>>> (+44) 7975795758
>>>
>>>
>>>
>>> --
>>> Jonathan Gray
>>>
>>> Community Coordinator
>>> The Open Knowledge Foundation
>>> http://blog.okfn.org
>>>
>>> http://twitter.com/jwyg
>>> http://identi.ca/jwyg
>>>
>>
>>
>>
>> --
>> Open Knowledge Foundation
>> Promoting Open Knowledge in a Digital Age
>> http://www.okfn.org/ - http://blog.okfn.org/
>>
>> _______________________________________________
>> ckan-discuss mailing list
>> ckan-discuss at lists.okfn.org
>> http://lists.okfn.org/mailman/listinfo/ckan-discuss
>>
>