[ckan-dev] Uploading 500MB - 1GB files via API fails.

Tue Jan 7 08:31:44 UTC 2014

CKAN by itself did not set a limit for file uploads until 2.2 (which is not
yet released anyway). So this error is coming from Apache, Nginx, or WSGI.
You'll have to confirm which layer it's coming from.

Nigel Babu

Developer  |  @nigelbabu <https://twitter.com/nigelbabu>

The Open Knowledge Foundation <http://okfn.org/>

Empowering through Open Knowledge

http://okfn.org/  |  @okfn <http://twitter.com/OKFN>  |  OKF on
Facebook<https://www.facebook.com/OKFNetwork> |
Blog <http://blog.okfn.org/>  |  Newsletter<http://okfn.org/about/newsletter>

 CKAN | http://ckan.org/ | @CKANproject
<http://twitter.com/CKANproject> |the world’s leading open-source data
portal platform

On 6 January 2014 20:01, Henrik Aagaard Sørensen <BU1G at tmf.kk.dk> wrote:

> I’m trying to upload files to CKAN, from one server which is on the same
> network as CKAN, which is from 500MB to 1GB large.
>
>
>
> It fails with the error:
>
>   File "test.py", line 26, in <module>
>
>     ckan.add_package_resource(config.get('Main', 'ckan_dataset'),
> filename, name=config.get('Main', 'filename'), resource_type='data',
> format='csv')
>
>   File "/usr/local/src/ckanclient/ckanclient/__init__.py", line 631, in
> add_package_resource
>
>     url, msg = self.upload_file(file_path)
>
>   File "/usr/local/src/ckanclient/ckanclient/__init__.py", line 585, in
> upload_file
>
>     errcode, body =
> self._post_multipart(auth_dict['action'].encode('ascii'), fields, files)
>
>   File "/usr/local/src/ckanclient/ckanclient/__init__.py", line 504, in
> _post_multipart
>
>     response = urlopen(request)
>
>   File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
>
>     return _opener.open(url, data, timeout)
>
>   File "/usr/lib/python2.7/urllib2.py", line 406, in open
>
>     response = meth(req, response)
>
>   File "/usr/lib/python2.7/urllib2.py", line 519, in http_response
>
>     'http', request, response, code, msg, hdrs)
>
>   File "/usr/lib/python2.7/urllib2.py", line 444, in error
>
>     return self._call_chain(*args)
>
>   File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
>
>     result = func(*args)
>
>   File "/usr/lib/python2.7/urllib2.py", line 527, in http_error_default
>
>     raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
>
> urllib2.HTTPError: HTTP Error 500: Internal Server Error
>
>
>
> Small files (100MB) do work perfectly.
>
> I’ve changed the nginx.conf-files to accept 1000M in client_max_body_size.
> What else should I do?
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20140107/6e6b2c04/attachment-0003.html>