[ckan-discuss] Trouble installing CKAN

Sun Aug 19 18:19:57 BST 2012

I'm going through more documentation today. "ckan.storage.directory" (mentioned in the docs) doesn't seem to be used in the code. Isn't the correct name for this setting "ckan.storage_dir"? Seen here: http://docs.ckan.org/en/ckan-1.7.1/configuration.html Also, yesterday, I had also forgotten to set site_url. This pull request adds a note about setting site_url in appropriate parts of the documentation: https://github.com/okfn/ckan/pull/102

It seems that local file storage should work with Data Preview, so I'm now looking for other sources for the error. In the celeryd log, I'm getting URL unobtainable. When I make the error message more helpful (see pull request: https://github.com/okfn/ckanext-archiver/pull/2) I get "URL unobtainable: HTTP 404 on 'http://4m58.localtunnel.com//en/storage/f/2012-08-19T154553/dons_aux_partis_politiques_du_quebec.csv'"

This is due to the double slash. Maybe if I were running under Apache or other, the double slashes would be normalized to a single slash and the URL would resolve. Whatever the case, the double slash issue should be resolved. It turns out on Aug 1, this regression was introduced by https://github.com/okfn/ckan/commit/e2073a37ac16acb1233a19f42d6473e0a2065b75 Fixed in pull request: https://github.com/okfn/ckan/pull/104

With that fixed, I now get this error in Data Preview: "Could not load preview: DataStore returned an error (Elastic Search did not return a mapping)". I assume this is because I haven't set up Nginx to proxy to ElasticSearch. Will try that now.

Please merge my pull requests!

James

On 2012-08-19, at 2:28 AM, James McKinney wrote:

> I recently uploaded CSV files greater than 50MB in size to datahub.io (links below), and the Data Preview gives the unhelpful error: "Could not load preview: DataProxy returned an error (Data transformation failed. Reason: )" The Data API also does not work.
> 
> http://datahub.io/en/dataset/bafc5264-c2b0-44d5-a76d-2215b0e1c9da/resource/415d32d9-aa4f-491c-8556-6e895e7eef01
> http://datahub.io/en/dataset/96888a16-4be5-4bf1-9dc6-793f6541e94d/resource/15b717a5-4088-4443-8b70-81555dde237c
> http://thedatahub.org/dataset/registre-qc/resource/9afe589f-2cc6-4d0c-bfa3-8c79c889a8f8
> http://datahub.io/en/dataset/registre-ca/resource/473efd3b-0d3d-41d2-b136-9d3249220449
> 
> So, in order to discover (and maybe fix) the error, I installed the latest CKAN locally. I also sent an email about this error via the Datahub contact form. I'm now having trouble setting up the DataStore. The documentation is thin: http://docs.ckan.org/en/latest/datastore.html
> 
> When I enabled the DataStore without setting up Nginx or adding the Datastorer plugin, I got a JavaScript error: "Uncaught TypeError: Object.keys called on non-object recline.js:3180". I already had ElasticSearch running.
> 
> I then installed the Datastorer plugin, started the celery daemon and restarted the CKAN server. I added a new resource, and now I get this error message in the preview: "Could not load preview: DataProxy returned an error (Request Error: Backend did not respond after 5 seconds)" and this JavaScript error: "Failed to load resource: the server responded with a status of 500 (Internal Server Error) http://jsonpdataproxy.appspot.com/?callback=...&url=http:////storage/f/2012-08-19T050704/ca_corps_scraper.csv&..."
> 
> I'm using local storage, which given the above error seems to not work with the Recline Data Explorer. I've submitted a pull request to document this. https://github.com/okfn/ckan/pull/103
> 
> I installed boto and configured CKAN for S3. After restarting CKAN and celeryd and starting a new resource upload of a 50MB CSV, I see "POST http://my-bucket-name.s3.amazonaws.com/ 400 (Bad Request)" in the Chrome console. I try a smaller CSV (4MB) and it works. I go halfway with a 25MB CSV and it also works. (????) Also, these CSVs are now considered to be binary/octet-stream. Why? When using local file storage and on datahub.io, they were considered text/csv.
> 
> With S3 hitting a dead end, I then setup Google Storage, where I needed to know that I had to click the "Interoperable Access" button to get to the legacy system of access keys and secrets. Google seems to prefer the use of OAuth 2.0. With Google Storage however, uploading any size CSV causes this error to appear in the console: "Failed to load resource: the server responded with a status of 400 (Bad Request) http://jpmckinney-ckan.commondatastorage.googleapis.com/".
> 
> Anyway, the main issue is that the links at the start of this post have errors in the Data Preview section. It'd be great to know how to correct the S3 and Google issues, though.
> 
> James
> 
> 
> 
> 
> 