[ckan-dev] Problems uploading files in CKAN

[Adil Hasan]

Hello Andrea,

thank you very much for the info! I'll check what our nginx settings are
compared with your and see if it makes a difference. I think the
proxy_buffering could possibly help. I will doble check and report back.

thanks,
adil

On Thu, Dec 05, 2019 at 04:28:58PM +0100, Andrea Stella wrote:
> Hi all, 
> to upload large files (>4GB) through the api we have configured in our nginx 
> 
> client_max_body_size 15G; 
> proxy_buffering off; 
> proxy_connect_timeout 1200; 
> proxy_send_timeout 1200; 
> proxy_read_timeout 1200; 
> send_timeout 1200; 
> keepalive_timeout 60; 
> 
> let me know if this solves the problem. 
> a. 
> 
> 
> Da: "Adil Hasan" <paradox2005 at gmail.com> 
> A: "ckan dev" <ckan-dev at lists.okfn.org> 
> Inviato: Mercoledì, 4 dicembre 2019 12:30:36 
> Oggetto: Re: [ckan-dev] Problems uploading files in CKAN 
> 
> Hello Andrew, 
> thanks for the quick response. We have the max_resourse_size set to 10GB 
> and we have the client_max_body_size in nginx set high as well. What we 
> see is that sometimes ~500MB is transferred, sometimes, 3GB. All 
> different amounts of data are transferred before the web interface 
> claims that the transfer has completed when it hasn't and the web 
> interface page appears to be frozen (at least the Add button is inactive 
> which suggests that some part of CKAN assumes the upload is still 
> ongoing). 
> many thanks, 
> adil 
> 
> On Wed, Dec 04, 2019 at 10:09:58AM +0000, Andrew Brooks wrote: 
> > There is a setting in the .ini file: ckan.max_resource_size 
> > and nginx has a limit which I think is client_max_body_size 
> > datapusher has a limit in datapusher_settings.py which is MAX_CONTENT_LENGTH 
> > 
> > On Wed, 4 Dec 2019 at 09:46, Adil Hasan <paradox2005 at gmail.com> wrote: 
> > 
> > > Hello, 
> > > 
> > > We are deploying CKAN in container in Kubernetes and we have mounted our 
> > > storage in the container. We have /var/lib/ckan mounted. We also pointed 
> > > the cache_dir to the mounted storage. 
> > > 
> > > We are able to upload small files, but when we try to upload large files 
> > > (>500MB) through the web interface we notice that the upload just hangs 
> > > after some period of time. 
> > > 
> > > Looking in the developer tools (in Firefox) it seems that the network 
> > > monitoring suggests that XMB has been transferred and the transfer is 
> > > complete. But, X is less than the size of the file. 
> > > 
> > > When I look at the ckan web page the page looks as if the transfer is 
> > > still progressing (in fact it's stuck). I cannot click the 'Add' button. 
> > > 
> > > We have increased the timeout for the apache web server: 
> > > 
> > > RequestReadTimeout header=90,MinRate=500 
> > > RequestReadTimeout body=90,MinRate=500 
> > > 
> > > and we also increased the nginx (our proxy server): 
> > > 
> > > nginx.ingress.kubernetes.io/proxy-connect-timeout: "600" 
> > > nginx.ingress.kubernetes.io/proxy-send-timeout: "600" 
> > > nginx.ingress.kubernetes.io/proxy-read-timeout: "600" 
> > > 
> > > We tried to see where ckan uploads the file to whilst the transfer is 
> > > progressing, but we cannot find where it goes. Does anyone know? 
> > > 
> > > Has anyone experienced this before? Does anyone have a suggestion as to 
> > > how to get around this issue? 
> > > 
> > > Many thanks, 
> > > adil 
> > > _______________________________________________ 
> > > ckan-dev mailing list 
> > > ckan-dev at lists.okfn.org 
> > > https://lists.okfn.org/mailman/listinfo/ckan-dev 
> > > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev 
> > > 
> 
> > _______________________________________________ 
> > ckan-dev mailing list 
> > ckan-dev at lists.okfn.org 
> > https://lists.okfn.org/mailman/listinfo/ckan-dev 
> > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev 
> 
> _______________________________________________ 
> ckan-dev mailing list 
> ckan-dev at lists.okfn.org 
> https://lists.okfn.org/mailman/listinfo/ckan-dev 
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev 
> 
> -- 
> Andrea Stella 
> __________________________ 
> YACME S.r.l. - Gruppo MakeIT 
> Via Majani, 2 
> 40122 Bologna 
> 
> Informativa sul Trattamento dei Dati Personali 
> [ http://www.make-it.it/privacy-policy | Privacy Policy ] 

> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev