[ckan-dev] Problems uploading files in CKAN

Adrià Mercader adria.mercader at okfn.org
Fri Dec 13 10:32:18 UTC 2019


Hi all,

This list will shut down next week so I encourage you to follow the
discussion in the new mailing list. Check here for details on how to
migrate: https://ckan.org/2019/11/22/migrating-our-mailing-lists/

Best,

Adrià

On Thu, 5 Dec 2019 at 18:41, Adil Hasan <adilhasan2 at gmail.com> wrote:
>
> Hello Andrea,
>
> thank you very much for the info! I'll check what our nginx settings are
> compared with your and see if it makes a difference. I think the
> proxy_buffering could possibly help. I will doble check and report back.
>
> thanks,
> adil
>
> On Thu, Dec 05, 2019 at 04:28:58PM +0100, Andrea Stella wrote:
> > Hi all,
> > to upload large files (>4GB) through the api we have configured in our nginx
> >
> > client_max_body_size 15G;
> > proxy_buffering off;
> > proxy_connect_timeout 1200;
> > proxy_send_timeout 1200;
> > proxy_read_timeout 1200;
> > send_timeout 1200;
> > keepalive_timeout 60;
> >
> > let me know if this solves the problem.
> > a.
> >
> >
> > Da: "Adil Hasan" <paradox2005 at gmail.com>
> > A: "ckan dev" <ckan-dev at lists.okfn.org>
> > Inviato: Mercoledì, 4 dicembre 2019 12:30:36
> > Oggetto: Re: [ckan-dev] Problems uploading files in CKAN
> >
> > Hello Andrew,
> > thanks for the quick response. We have the max_resourse_size set to 10GB
> > and we have the client_max_body_size in nginx set high as well. What we
> > see is that sometimes ~500MB is transferred, sometimes, 3GB. All
> > different amounts of data are transferred before the web interface
> > claims that the transfer has completed when it hasn't and the web
> > interface page appears to be frozen (at least the Add button is inactive
> > which suggests that some part of CKAN assumes the upload is still
> > ongoing).
> > many thanks,
> > adil
> >
> > On Wed, Dec 04, 2019 at 10:09:58AM +0000, Andrew Brooks wrote:
> > > There is a setting in the .ini file: ckan.max_resource_size
> > > and nginx has a limit which I think is client_max_body_size
> > > datapusher has a limit in datapusher_settings.py which is MAX_CONTENT_LENGTH
> > >
> > > On Wed, 4 Dec 2019 at 09:46, Adil Hasan <paradox2005 at gmail.com> wrote:
> > >
> > > > Hello,
> > > >
> > > > We are deploying CKAN in container in Kubernetes and we have mounted our
> > > > storage in the container. We have /var/lib/ckan mounted. We also pointed
> > > > the cache_dir to the mounted storage.
> > > >
> > > > We are able to upload small files, but when we try to upload large files
> > > > (>500MB) through the web interface we notice that the upload just hangs
> > > > after some period of time.
> > > >
> > > > Looking in the developer tools (in Firefox) it seems that the network
> > > > monitoring suggests that XMB has been transferred and the transfer is
> > > > complete. But, X is less than the size of the file.
> > > >
> > > > When I look at the ckan web page the page looks as if the transfer is
> > > > still progressing (in fact it's stuck). I cannot click the 'Add' button.
> > > >
> > > > We have increased the timeout for the apache web server:
> > > >
> > > > RequestReadTimeout header=90,MinRate=500
> > > > RequestReadTimeout body=90,MinRate=500
> > > >
> > > > and we also increased the nginx (our proxy server):
> > > >
> > > > nginx.ingress.kubernetes.io/proxy-connect-timeout: "600"
> > > > nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
> > > > nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
> > > >
> > > > We tried to see where ckan uploads the file to whilst the transfer is
> > > > progressing, but we cannot find where it goes. Does anyone know?
> > > >
> > > > Has anyone experienced this before? Does anyone have a suggestion as to
> > > > how to get around this issue?
> > > >
> > > > Many thanks,
> > > > adil
> > > > _______________________________________________
> > > > ckan-dev mailing list
> > > > ckan-dev at lists.okfn.org
> > > > https://lists.okfn.org/mailman/listinfo/ckan-dev
> > > > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
> > > >
> >
> > > _______________________________________________
> > > ckan-dev mailing list
> > > ckan-dev at lists.okfn.org
> > > https://lists.okfn.org/mailman/listinfo/ckan-dev
> > > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
> >
> > _______________________________________________
> > ckan-dev mailing list
> > ckan-dev at lists.okfn.org
> > https://lists.okfn.org/mailman/listinfo/ckan-dev
> > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
> >
> > --
> > Andrea Stella
> > __________________________
> > YACME S.r.l. - Gruppo MakeIT
> > Via Majani, 2
> > 40122 Bologna
> >
> > Informativa sul Trattamento dei Dati Personali
> > [ http://www.make-it.it/privacy-policy | Privacy Policy ]
>
> > _______________________________________________
> > ckan-dev mailing list
> > ckan-dev at lists.okfn.org
> > https://lists.okfn.org/mailman/listinfo/ckan-dev
> > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev



More information about the ckan-dev mailing list