[ckan-discuss] Trouble installing CKAN

Sun Aug 19 07:28:42 BST 2012

I recently uploaded CSV files greater than 50MB in size to datahub.io (links below), and the Data Preview gives the unhelpful error: "Could not load preview: DataProxy returned an error (Data transformation failed. Reason: )" The Data API also does not work.

http://datahub.io/en/dataset/bafc5264-c2b0-44d5-a76d-2215b0e1c9da/resource/415d32d9-aa4f-491c-8556-6e895e7eef01
http://datahub.io/en/dataset/96888a16-4be5-4bf1-9dc6-793f6541e94d/resource/15b717a5-4088-4443-8b70-81555dde237c
http://thedatahub.org/dataset/registre-qc/resource/9afe589f-2cc6-4d0c-bfa3-8c79c889a8f8
http://datahub.io/en/dataset/registre-ca/resource/473efd3b-0d3d-41d2-b136-9d3249220449

So, in order to discover (and maybe fix) the error, I installed the latest CKAN locally. I also sent an email about this error via the Datahub contact form. I'm now having trouble setting up the DataStore. The documentation is thin: http://docs.ckan.org/en/latest/datastore.html

When I enabled the DataStore without setting up Nginx or adding the Datastorer plugin, I got a JavaScript error: "Uncaught TypeError: Object.keys called on non-object recline.js:3180". I already had ElasticSearch running.

I then installed the Datastorer plugin, started the celery daemon and restarted the CKAN server. I added a new resource, and now I get this error message in the preview: "Could not load preview: DataProxy returned an error (Request Error: Backend did not respond after 5 seconds)" and this JavaScript error: "Failed to load resource: the server responded with a status of 500 (Internal Server Error) http://jsonpdataproxy.appspot.com/?callback=...&url=http:////storage/f/2012-08-19T050704/ca_corps_scraper.csv&..."

I'm using local storage, which given the above error seems to not work with the Recline Data Explorer. I've submitted a pull request to document this. https://github.com/okfn/ckan/pull/103

I installed boto and configured CKAN for S3. After restarting CKAN and celeryd and starting a new resource upload of a 50MB CSV, I see "POST http://my-bucket-name.s3.amazonaws.com/ 400 (Bad Request)" in the Chrome console. I try a smaller CSV (4MB) and it works. I go halfway with a 25MB CSV and it also works. (????) Also, these CSVs are now considered to be binary/octet-stream. Why? When using local file storage and on datahub.io, they were considered text/csv.

With S3 hitting a dead end, I then setup Google Storage, where I needed to know that I had to click the "Interoperable Access" button to get to the legacy system of access keys and secrets. Google seems to prefer the use of OAuth 2.0. With Google Storage however, uploading any size CSV causes this error to appear in the console: "Failed to load resource: the server responded with a status of 400 (Bad Request) http://jpmckinney-ckan.commondatastorage.googleapis.com/".

Anyway, the main issue is that the links at the start of this post have errors in the Data Preview section. It'd be great to know how to correct the S3 and Google issues, though.

James