[ECODP-dev] ODP Performance of http://open-data.europa.eu/open-data/data

Mon Mar 4 17:40:20 UTC 2013

Hi Olivier,

Thanks for your email. I'm sorry about the issues which you're currently
experiencing. However I would disagree with your comment that CKAN isn't
ready for production or for hosting large amounts of data - CKAN is the
leading system for publishing open data and has been used in production by
major sites like Data.gov.uk (~8000 datasets) and many others around the
world without performance or scaling issues. We are also currently working
with the US Government's to migrate portions of their Data.gov site to
CKAN, a choice they wouldn't have made if they weren't confident in CKAN's
ability to scale.

Obviously in this case we are working at arms length without access to the
deployment or test environment which adds a layer of complexity but we are
obviously very keen to help you resolve the issue that you are
experiencing. With this in mind our lead developer and I are available to
work with you to resolve this as a priority. Perhaps we could have a call
tomorrow morning to establish the current status and next steps? It would
also be very helpful to see the logs from the live server.

Kind regards,

Darwin

Darwin Peltan
Project Manager

The Open Knowledge Foundation
http://www.okfn.org

Skype: darwinp
Twitter: @darwin

On 4 March 2013 07:51, ISOARD Olivier (OP) <
Olivier.ISOARD at publications.europa.eu> wrote:

> Bastiaan,****
>
> ** **
>
> The symptom you described is actual present from the very beginning of the
> production mid 12/2012. Nothing new there. ****
>
> ** **
>
> For info, after complementary testing we did, our conclusion is that the
> response time for a particular dataset directly depends on publisher is
> belongs to. More the publisher has dataset published on ODP, more the
> response time is low. If you do the test yourself and test (1) with PUBL
> and (2) with ESTAT, we will see that the response time is much faster for
> PUBL than for ESTAT, because on 1 hand there are only 12 DS published and
> on the other hand, more than 5000 DS.****
>
> ** **
>
> So, it’s normal you cannot get the same figures from your testing env, as
> it only contains 50 dataset maximum. Load it with 6000 DS and you will see.
> ****
>
> ** **
>
> PS: We have more and more the feeling that CKAN is not that well design to
> host such amount of data. It’s probably OK for a PoC or a prototype,
> probably not for a production system…****
>
> ** **
>
> regards****
>
> ** **
>
> *Olivier ISOARD **
> **Projets Inf**ormatiques*****
>
> *_**__**_**__**_**__**__**_**__**_**__***
>
> * *
>
> *Office des publications de l'Union européenne *
> Direction A – Core Business Services
> Unité A3 - Projets Informatiques
>
> 2985 Luxembourg • LUXEMBOURG
> Tel: +352 292942360 • Fax: +352 292944604
> olivier.isoard at publications.europa.eu*
> **http://publications.europa.eu*****
>
> ** **
>
> *From:* Bastiaan Deblieck [mailto:bastiaan.deblieck at tenforce.com]
> *Sent:* Friday, March 01, 2013 10:35 AM
> *To:* ZAJAC Agnieszka (OP); ISOARD Olivier (OP); HOHN Norbert (OP)
> *Cc:* Project list for EC ODP CKAN project
> *Subject:* ODP Performance of http://open-data.europa.eu/open-data/data***
> *
>
> ** **
>
> Hello,****
>
> ** **
>
> Most probably you are aware that the response times of the
> http://open-data.europa.eu/open-data/data are very bad. ****
>
> - If you click a dataset reference a 1st time than it can take more than
> 10 seconds for it to appear. Once it is cached, so upon a 2nd click, it
> appears instantaneously. ****
>
> ** **
>
> As part of your slow upload Bert has been doing some investigations too
> and made some comparisons between our test environment and
> http://open-data.europa.eu/open-data/data****
>
> - He can see the problem on http://open-data.europa.eu/open-data/data but
> there is no problem at all in the TenForce test environment.****
>
> ** **
>
> Please let us know if you wish to set up a call to discuss this topic. We
> have limited availability this afternoon. We are unavailable on Monday, and
> are again available on Tuesday.****
>
> ** **
>
> Best,
> Bastiaan
> ****
>
> ** **
>
> --
> Bastiaan Deblieck
> Semantic Technology Business Unit Manager****
>
> http://www.tenforce.com/
> T: +32 16 31 48 60
> M:+32 475 95 49 32****
>
> _______________________________________________
> Ecodp-dev mailing list
> Ecodp-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ecodp-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.okfn.org/mailman/private/ecodp-dev/attachments/20130304/14ddd189/attachment.html>