[ckan-dev] Slow upload / 2
ian at excess.org
Thu Jan 17 16:22:37 UTC 2019
I've had success with the cProfile module and runsnakerun to pin down
this sort of thing. There's a profile command in the cli but it looks
like it only does GET requests. Should still be useful as an example.
On Thu, Jan 17, 2019 at 11:12 AM Harald von Waldow
<harald.vonwaldow at eawag.ch> wrote:
> Hi everybody
> I narrowed that down a bit:
> Local dev-server: network, Apache, NGINX are not involved.
> 1. Upload (the file is completely transmitted to local dev-server): 30 s
> 2. CKAN uses CPU a lot, no significant disk activity,
> no debug-log messages: 30 s
> 3. CKAN enters ResourceUpload.upload (lib/uploader.py)
> and writes file to final destination: 8 s
> What happens in 2. ?
> Grateful for any ideas.
> On Wed, 16 Jan 2019 19:07:51 +0100
> Harald von Waldow <harald.vonwaldow at eawag.ch> wrote:
> > Hi everybody!
> > I am suffering from very slow upload speeds. I have a default setup,
> > Apache proxied by NGINX. NGINX buffers, so I can separate +/- the time
> > the data is on the wire from the time it takes it to travel from NGINX
> > to disk.
> > Network speed is as expected, but the second part takes very long.
> > E.g. a 6.2 GiB file uploads to NGINX with 116 MiB/s but dripples from
> > there to the disk at 24 MiB/s. During that period, Apache (ckan)
> > almost maxes out the CPU.
> > I can exclude that the disk is the problem.
> > Any ideas?
> > Thanks,
> > Harald
> Harald von Waldow
> ICT Services
> Research Data Management
> Ueberlandstrasse 133
> 8600 Duebendorf
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
More information about the ckan-dev