[ckan-dev] Problems uploading files in CKAN

[Andrea Stella]

Hi all, 
to upload large files (>4GB) through the api we have configured in our nginx 

client_max_body_size 15G; 
proxy_buffering off; 
proxy_connect_timeout 1200; 
proxy_send_timeout 1200; 
proxy_read_timeout 1200; 
send_timeout 1200; 
keepalive_timeout 60; 

let me know if this solves the problem. 
a. 


Da: "Adil Hasan" <paradox2005 at gmail.com> 
A: "ckan dev" <ckan-dev at lists.okfn.org> 
Inviato: Mercoledì, 4 dicembre 2019 12:30:36 
Oggetto: Re: [ckan-dev] Problems uploading files in CKAN 

Hello Andrew, 
thanks for the quick response. We have the max_resourse_size set to 10GB 
and we have the client_max_body_size in nginx set high as well. What we 
see is that sometimes ~500MB is transferred, sometimes, 3GB. All 
different amounts of data are transferred before the web interface 
claims that the transfer has completed when it hasn't and the web 
interface page appears to be frozen (at least the Add button is inactive 
which suggests that some part of CKAN assumes the upload is still 
ongoing). 
many thanks, 
adil 

On Wed, Dec 04, 2019 at 10:09:58AM +0000, Andrew Brooks wrote: 
> There is a setting in the .ini file: ckan.max_resource_size 
> and nginx has a limit which I think is client_max_body_size 
> datapusher has a limit in datapusher_settings.py which is MAX_CONTENT_LENGTH 
> 
> On Wed, 4 Dec 2019 at 09:46, Adil Hasan <paradox2005 at gmail.com> wrote: 
> 
> > Hello, 
> > 
> > We are deploying CKAN in container in Kubernetes and we have mounted our 
> > storage in the container. We have /var/lib/ckan mounted. We also pointed 
> > the cache_dir to the mounted storage. 
> > 
> > We are able to upload small files, but when we try to upload large files 
> > (>500MB) through the web interface we notice that the upload just hangs 
> > after some period of time. 
> > 
> > Looking in the developer tools (in Firefox) it seems that the network 
> > monitoring suggests that XMB has been transferred and the transfer is 
> > complete. But, X is less than the size of the file. 
> > 
> > When I look at the ckan web page the page looks as if the transfer is 
> > still progressing (in fact it's stuck). I cannot click the 'Add' button. 
> > 
> > We have increased the timeout for the apache web server: 
> > 
> > RequestReadTimeout header=90,MinRate=500 
> > RequestReadTimeout body=90,MinRate=500 
> > 
> > and we also increased the nginx (our proxy server): 
> > 
> > nginx.ingress.kubernetes.io/proxy-connect-timeout: "600" 
> > nginx.ingress.kubernetes.io/proxy-send-timeout: "600" 
> > nginx.ingress.kubernetes.io/proxy-read-timeout: "600" 
> > 
> > We tried to see where ckan uploads the file to whilst the transfer is 
> > progressing, but we cannot find where it goes. Does anyone know? 
> > 
> > Has anyone experienced this before? Does anyone have a suggestion as to 
> > how to get around this issue? 
> > 
> > Many thanks, 
> > adil 
> > _______________________________________________ 
> > ckan-dev mailing list 
> > ckan-dev at lists.okfn.org 
> > https://lists.okfn.org/mailman/listinfo/ckan-dev 
> > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev 
> > 

> _______________________________________________ 
> ckan-dev mailing list 
> ckan-dev at lists.okfn.org 
> https://lists.okfn.org/mailman/listinfo/ckan-dev 
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev 

_______________________________________________ 
ckan-dev mailing list 
ckan-dev at lists.okfn.org 
https://lists.okfn.org/mailman/listinfo/ckan-dev 
Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev 

-- 
Andrea Stella 
__________________________ 
YACME S.r.l. - Gruppo MakeIT 
Via Majani, 2 
40122 Bologna 

Informativa sul Trattamento dei Dati Personali 
[ http://www.make-it.it/privacy-policy | Privacy Policy ] 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20191205/7da5a566/attachment-0002.html>