[ckan-dev] Uploading 500MB - 1GB files via API fails.

Henrik Aagaard Sørensen BU1G at tmf.kk.dk
Mon Jan 6 14:31:13 UTC 2014


I'm trying to upload files to CKAN, from one server which is on the same network as CKAN, which is from 500MB to 1GB large.

It fails with the error:
  File "test.py", line 26, in <module>
    ckan.add_package_resource(config.get('Main', 'ckan_dataset'), filename, name=config.get('Main', 'filename'), resource_type='data', format='csv')
  File "/usr/local/src/ckanclient/ckanclient/__init__.py", line 631, in add_package_resource
    url, msg = self.upload_file(file_path)
  File "/usr/local/src/ckanclient/ckanclient/__init__.py", line 585, in upload_file
    errcode, body = self._post_multipart(auth_dict['action'].encode('ascii'), fields, files)
  File "/usr/local/src/ckanclient/ckanclient/__init__.py", line 504, in _post_multipart
    response = urlopen(request)
  File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 406, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 519, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 444, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 527, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 500: Internal Server Error

Small files (100MB) do work perfectly.
I've changed the nginx.conf-files to accept 1000M in client_max_body_size. What else should I do?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20140106/5f45c861/attachment-0002.html>


More information about the ckan-dev mailing list