[ckan-dev] Problems uploading files in CKAN

[Adil Hasan]

Hello,

We are deploying CKAN in container in Kubernetes and we have mounted our
storage in the container. We have /var/lib/ckan mounted. We also pointed
the cache_dir to the mounted storage.

We are able to upload small files, but when we try to upload large files
(>500MB) through the web interface we notice that the upload just hangs
after some period of time.

Looking in the developer tools (in Firefox) it seems that the network
monitoring suggests that XMB has been transferred and the transfer is
complete. But, X is less than the size of the file.

When I look at the ckan web page the page looks as if the transfer is
still progressing (in fact it's stuck). I cannot click the 'Add' button.

We have increased the timeout for the apache web server:

RequestReadTimeout header=90,MinRate=500
RequestReadTimeout body=90,MinRate=500

and we also increased the nginx (our proxy server):

nginx.ingress.kubernetes.io/proxy-connect-timeout: "600"
nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
nginx.ingress.kubernetes.io/proxy-read-timeout: "600"

We tried to see where ckan uploads the file to whilst the transfer is
progressing, but we cannot find where it goes. Does anyone know?

Has anyone experienced this before? Does anyone have a suggestion as to
how to get around this issue?

Many thanks,
adil