[ckan-discuss] datastore size limit?

Fri Mar 28 11:38:34 UTC 2014

Great you got this resolved and definitely something to document. Would you
be up for making an edit to the relevant documentation, which I think is
here:

https://github.com/ckan/ckan/blob/master/doc/maintaining/datastore.rst

Rufus

On 27 March 2014 10:08, Toby Ealden <toby.ealden at gmail.com> wrote:

> Changing MAX_CONTENT_LENGTH in
> /usr/lib/ckan/datapusher/src/datapusher/datapusher/jobs.py solves the
> problem.
>
> Is this not configurable from a settings file somewhere? The default value
> is 10MB which seems a bit low.
>
> Also, this doesn't appear to be documented anywhere....
>
>
> On 27 March 2014 09:09, Toby Ealden <toby.ealden at gmail.com> wrote:
>
>> Looking at the /tmp/ckan_service.log I see:
>>
>> Job "push_to_datastore (trigger: RunTriggerNow, run = True, next run at:
>> None)" raised an exception
>> Traceback (most recent call last):
>>   File
>> "/usr/lib/ckan/datapusher/lib/python2.7/site-packages/apscheduler/scheduler.py",
>> line 512, in _run_job
>>     retval = job.func(*job.args, **job.kwargs)
>>   File "/usr/lib/ckan/datapusher/src/datapusher/datapusher/jobs.py", line
>> 257, in push_to_datastore
>>     cl=cl, max_cl=MAX_CONTENT_LENGTH))
>> JobError: Resource too large to download: 21209799 > max (10485760).
>>
>> Anyone know where this limit is configured?
>>
>>
>> On 27 March 2014 08:32, Toby Ealden <toby.ealden at gmail.com> wrote:
>>
>>> Thanks for the response.
>>>
>>> To be clear, I am uploading the files through the web interface and the
>>> upload appears to complete with no problems. In fact I can then go on to
>>> preview the data through the web interface.
>>>
>>> The problem is that the datastore API does not recognise the resource
>>> id. So for some reason the upload hasn't been processed by the datastore.
>>> Another symptom is the fact that the "Data API" button is missing from the
>>> resource page of the 25MB CSV (see attachments).
>>>
>>> [image: Inline images 2]
>>>
>>> [image: Inline images 1]
>>>
>>>
>>>
>>>
>>> On 26 March 2014 19:24, Rufus Pollock <rufus.pollock at okfn.org> wrote:
>>>
>>>> On Wednesday, 26 March 2014, Toby Ealden <toby.ealden at gmail.com> wrote:
>>>>
>>>>> Can nobody help with this? Am I missing something obvious?
>>>>>
>>>>
>>>> Sorry for delay here - one of the things we're trying to improve is the
>>>> helpdesk route for people like yourself :-)
>>>>
>>>>
>>>>> On 25 March 2014 17:48, Toby Ealden <toby.ealden at gmail.com> wrote:
>>>>>
>>>>>> Hello
>>>>>>
>>>>>> Is there a size limit restriction on resources added to the datastore.
>>>>>>
>>>>>> I have successfully set up a CKAN instance with the datastore enabled.
>>>>>>
>>>>>> I have the following scenario:
>>>>>>
>>>>>> - I add a 275KB CSV dataset to my CKAN instance through the web
>>>>>> interface.
>>>>>> - I can successfully perform a datastore_search on this dataset
>>>>>> - I add a 25MB CSV dataset
>>>>>> - When I try and access the datastore using the resource ID I get
>>>>>> 'resource not found'
>>>>>>
>>>>>
>>>> I don't believe there is any datastore size limit. I'd really need a
>>>> bit more debug info. An immediate suggestion is chunk the upload of your 25
>>>> MB CSV to avoid any risk that you're just encountering a webserver timeout
>>>> on the request.
>>>>
>>>> Rufus
>>>>
>>>>
>>>>>
>>>>>> All insight gratefully received.
>>>>>>
>>>>>> Toby Ealden
>>>>>>
>>>>>>
>>>>>
>>>>
>>>> --
>>>>
>>>>
>>>> * Rufus Pollock Founder and CEO | skype: rufuspollock | @rufuspollock
>>>> <https://twitter.com/rufuspollock> The Open Knowledge Foundation
>>>> <http://okfn.org/> Empowering through Open Knowledge http://okfn.org/
>>>> <http://okfn.org/> | @okfn <http://twitter.com/OKFN> | OKF on Facebook
>>>> <https://www.facebook.com/OKFNetwork> |  Blog <http://blog.okfn.org/>  |
>>>>  Newsletter <http://okfn.org/about/newsletter> *
>>>>
>>>>
>>>
>>
>

-- 

*Rufus PollockFounder and CEO | skype: rufuspollock | @rufuspollock
<https://twitter.com/rufuspollock>The Open Knowledge Foundation
<http://okfn.org/>Empowering through Open Knowledgehttp://okfn.org/
<http://okfn.org/> | @okfn <http://twitter.com/OKFN> | OKF on Facebook
<https://www.facebook.com/OKFNetwork> |  Blog <http://blog.okfn.org/>  |
 Newsletter <http://okfn.org/about/newsletter>*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.okfn.org/pipermail/ckan-discuss/attachments/20140328/3c1e8dde/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ckan-data-api.png
Type: image/png
Size: 12460 bytes
Desc: not available
URL: <https://lists.okfn.org/pipermail/ckan-discuss/attachments/20140328/3c1e8dde/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ckan-no-data-api.png
Type: image/png
Size: 8690 bytes
Desc: not available
URL: <https://lists.okfn.org/pipermail/ckan-discuss/attachments/20140328/3c1e8dde/attachment-0003.png>