[ckan-dev] Uploading 500MB - 1GB files via API fails.

[Henrik Aagaard Sørensen]

The error log comes from nginx. Apache does not report any error.

Fra: ckan-dev [mailto:ckan-dev-bounces at lists.okfn.org] På vegne af Nigel Babu
Sendt: 7. januar 2014 09:32
Til: CKAN Development Discussions
Emne: Re: [ckan-dev] Uploading 500MB - 1GB files via API fails.

CKAN by itself did not set a limit for file uploads until 2.2 (which is not yet released anyway). So this error is coming from Apache, Nginx, or WSGI. You'll have to confirm which layer it's coming from.


Nigel Babu

Developer  |  @nigelbabu<https://twitter.com/nigelbabu>

The Open Knowledge Foundation<http://okfn.org/>

Empowering through Open Knowledge

http://okfn.org/  |  @okfn<http://twitter.com/OKFN>  |  OKF on Facebook<https://www.facebook.com/OKFNetwork>  |  Blog<http://blog.okfn.org/>  |  Newsletter<http://okfn.org/about/newsletter>

CKAN | http://ckan.org/ | @CKANproject<http://twitter.com/CKANproject> | the world’s leading open-source data portal platform

On 6 January 2014 20:01, Henrik Aagaard Sørensen <BU1G at tmf.kk.dk<mailto:BU1G at tmf.kk.dk>> wrote:
I’m trying to upload files to CKAN, from one server which is on the same network as CKAN, which is from 500MB to 1GB large.

It fails with the error:
  File "test.py", line 26, in <module>
    ckan.add_package_resource(config.get('Main', 'ckan_dataset'), filename, name=config.get('Main', 'filename'), resource_type='data', format='csv')
  File "/usr/local/src/ckanclient/ckanclient/__init__.py", line 631, in add_package_resource
    url, msg = self.upload_file(file_path)
  File "/usr/local/src/ckanclient/ckanclient/__init__.py", line 585, in upload_file
    errcode, body = self._post_multipart(auth_dict['action'].encode('ascii'), fields, files)
  File "/usr/local/src/ckanclient/ckanclient/__init__.py", line 504, in _post_multipart
    response = urlopen(request)
  File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 406, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 519, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 444, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 527, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 500: Internal Server Error

Small files (100MB) do work perfectly.
I’ve changed the nginx.conf-files to accept 1000M in client_max_body_size. What else should I do?

_______________________________________________
ckan-dev mailing list
ckan-dev at lists.okfn.org<mailto:ckan-dev at lists.okfn.org>
https://lists.okfn.org/mailman/listinfo/ckan-dev
Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20140107/4ac19b9b/attachment-0003.html>