[ckan-dev] Slow upload / 2

Thu Jan 17 16:22:37 UTC 2019

I've had success with the cProfile module and runsnakerun to pin down
this sort of thing. There's a profile command in the cli but it looks
like it only does GET requests. Should still be useful as an example.

On Thu, Jan 17, 2019 at 11:12 AM Harald von Waldow
<harald.vonwaldow at eawag.ch> wrote:
>
> Hi everybody
>
> I narrowed that down a bit:
>
> Local dev-server: network, Apache, NGINX are not involved.
>
> 1. Upload (the file is completely transmitted to local dev-server): 30 s
>
> 2. CKAN uses CPU a lot, no significant disk activity,
>    no debug-log messages: 30 s
>
> 3. CKAN enters  ResourceUpload.upload (lib/uploader.py)
>    and writes file to final destination: 8 s
>
> What happens in 2. ?
>
> Grateful for any ideas.
>
> Thanks,
> Harald
>
> On Wed, 16 Jan 2019 19:07:51 +0100
> Harald von Waldow <harald.vonwaldow at eawag.ch> wrote:
>
> > Hi everybody!
> >
> > I am suffering from very slow upload speeds. I have a default setup,
> > Apache proxied by NGINX. NGINX buffers, so I can separate +/- the time
> > the data is on the wire from the time it takes it to travel from NGINX
> > to disk.
> >
> > Network speed is as expected, but the second part takes very long.
> >
> > E.g. a 6.2 GiB file uploads to NGINX with 116 MiB/s but dripples from
> > there to the disk at  24 MiB/s. During that period, Apache (ckan)
> > almost maxes out the CPU.
> >
> > I can exclude that the disk is the problem.
> >
> > Any ideas?
> >
> > Thanks,
> > Harald
> >
>
>
>
> --
> Harald von Waldow
> Eawag
> ICT Services
> Research Data Management
> Ueberlandstrasse 133
> 8600 Duebendorf
> http://www.eawag.ch
> https://opendata.eawag.ch
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev