[ckan-dev] Problems uploading files in CKAN

Wed Dec 4 11:30:36 UTC 2019

Hello Andrew,
thanks for the quick response. We have the max_resourse_size set to 10GB
and we have the client_max_body_size in nginx set high as well. What we
see is that sometimes ~500MB is transferred, sometimes, 3GB. All
different amounts of data are transferred before the web interface
claims that the transfer has completed when it hasn't and the web
interface page appears to be frozen (at least the Add button is inactive
which suggests that some part of CKAN assumes the upload is still
ongoing).
many thanks,
adil

On Wed, Dec 04, 2019 at 10:09:58AM +0000, Andrew Brooks wrote:
> There is a setting in the .ini file: ckan.max_resource_size
> and nginx has a limit which I think is client_max_body_size
> datapusher has a limit in datapusher_settings.py which is MAX_CONTENT_LENGTH
> 
> On Wed, 4 Dec 2019 at 09:46, Adil Hasan <paradox2005 at gmail.com> wrote:
> 
> > Hello,
> >
> > We are deploying CKAN in container in Kubernetes and we have mounted our
> > storage in the container. We have /var/lib/ckan mounted. We also pointed
> > the cache_dir to the mounted storage.
> >
> > We are able to upload small files, but when we try to upload large files
> > (>500MB) through the web interface we notice that the upload just hangs
> > after some period of time.
> >
> > Looking in the developer tools (in Firefox) it seems that the network
> > monitoring suggests that XMB has been transferred and the transfer is
> > complete. But, X is less than the size of the file.
> >
> > When I look at the ckan web page the page looks as if the transfer is
> > still progressing (in fact it's stuck). I cannot click the 'Add' button.
> >
> > We have increased the timeout for the apache web server:
> >
> > RequestReadTimeout header=90,MinRate=500
> > RequestReadTimeout body=90,MinRate=500
> >
> > and we also increased the nginx (our proxy server):
> >
> > nginx.ingress.kubernetes.io/proxy-connect-timeout: "600"
> > nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
> > nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
> >
> > We tried to see where ckan uploads the file to whilst the transfer is
> > progressing, but we cannot find where it goes. Does anyone know?
> >
> > Has anyone experienced this before? Does anyone have a suggestion as to
> > how to get around this issue?
> >
> > Many thanks,
> > adil
> > _______________________________________________
> > ckan-dev mailing list
> > ckan-dev at lists.okfn.org
> > https://lists.okfn.org/mailman/listinfo/ckan-dev
> > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
> >

> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev