[ckan-dev] Slow upload / 2

Thu Jan 17 16:12:11 UTC 2019

Hi everybody

I narrowed that down a bit:

Local dev-server: network, Apache, NGINX are not involved.

1. Upload (the file is completely transmitted to local dev-server): 30 s

2. CKAN uses CPU a lot, no significant disk activity,
   no debug-log messages: 30 s

3. CKAN enters  ResourceUpload.upload (lib/uploader.py)
   and writes file to final destination: 8 s

What happens in 2. ?

Grateful for any ideas.

Thanks,
Harald

On Wed, 16 Jan 2019 19:07:51 +0100
Harald von Waldow <harald.vonwaldow at eawag.ch> wrote:

> Hi everybody!
> 
> I am suffering from very slow upload speeds. I have a default setup,
> Apache proxied by NGINX. NGINX buffers, so I can separate +/- the time
> the data is on the wire from the time it takes it to travel from NGINX
> to disk.
> 
> Network speed is as expected, but the second part takes very long.
> 
> E.g. a 6.2 GiB file uploads to NGINX with 116 MiB/s but dripples from
> there to the disk at  24 MiB/s. During that period, Apache (ckan)
> almost maxes out the CPU.
> 
> I can exclude that the disk is the problem.
> 
> Any ideas?
> 
> Thanks,
> Harald
> 

-- 
Harald von Waldow
Eawag
ICT Services
Research Data Management
Ueberlandstrasse 133
8600 Duebendorf
http://www.eawag.ch
https://opendata.eawag.ch
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20190117/2e6fb777/attachment-0002.sig>