[ckan-dev] Uploading 500MB - 1GB files via API fails.

[Henrik Aagaard Sørensen]

I'm trying to upload files to CKAN, from one server which is on the same network as CKAN, which is from 500MB to 1GB large.

It fails with the error:
  File "test.py", line 26, in <module>
    ckan.add_package_resource(config.get('Main', 'ckan_dataset'), filename, name=config.get('Main', 'filename'), resource_type='data', format='csv')
  File "/usr/local/src/ckanclient/ckanclient/__init__.py", line 631, in add_package_resource
    url, msg = self.upload_file(file_path)
  File "/usr/local/src/ckanclient/ckanclient/__init__.py", line 585, in upload_file
    errcode, body = self._post_multipart(auth_dict['action'].encode('ascii'), fields, files)
  File "/usr/local/src/ckanclient/ckanclient/__init__.py", line 504, in _post_multipart
    response = urlopen(request)
  File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 406, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 519, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 444, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 527, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 500: Internal Server Error

Small files (100MB) do work perfectly.
I've changed the nginx.conf-files to accept 1000M in client_max_body_size. What else should I do?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20140106/5f45c861/attachment-0002.html>