[ckan-dev] Slow upload / 2

Thu Jan 17 16:35:33 UTC 2019

What version of CKAN are you using?

If you watch your memory usage, is it spiking to roughly the size of the
file you're uploading when CKAN is at 100% CPU? Some versions had an issue
with uploads where FileStorage would end up loading the entire file into
memory before dumping to disk.

You also typically do not want nginx to buffer uploads unless you're using
the nginx upload module to move the file directly into its final
destination and a CKAN extension to support this.

On Thu, Jan 17, 2019 at 11:12 AM Harald von Waldow <
harald.vonwaldow at eawag.ch> wrote:

> Hi everybody
>
> I narrowed that down a bit:
>
> Local dev-server: network, Apache, NGINX are not involved.
>
> 1. Upload (the file is completely transmitted to local dev-server): 30 s
>
> 2. CKAN uses CPU a lot, no significant disk activity,
>    no debug-log messages: 30 s
>
> 3. CKAN enters  ResourceUpload.upload (lib/uploader.py)
>    and writes file to final destination: 8 s
>
> What happens in 2. ?
>
> Grateful for any ideas.
>
> Thanks,
> Harald
>
> On Wed, 16 Jan 2019 19:07:51 +0100
> Harald von Waldow <harald.vonwaldow at eawag.ch> wrote:
>
> > Hi everybody!
> >
> > I am suffering from very slow upload speeds. I have a default setup,
> > Apache proxied by NGINX. NGINX buffers, so I can separate +/- the time
> > the data is on the wire from the time it takes it to travel from NGINX
> > to disk.
> >
> > Network speed is as expected, but the second part takes very long.
> >
> > E.g. a 6.2 GiB file uploads to NGINX with 116 MiB/s but dripples from
> > there to the disk at  24 MiB/s. During that period, Apache (ckan)
> > almost maxes out the CPU.
> >
> > I can exclude that the disk is the problem.
> >
> > Any ideas?
> >
> > Thanks,
> > Harald
> >
>
>
>
> --
> Harald von Waldow
> Eawag
> ICT Services
> Research Data Management
> Ueberlandstrasse 133
> 8600 Duebendorf
> http://www.eawag.ch
> https://opendata.eawag.ch
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20190117/3ee5f240/attachment-0002.html>