[ckan-dev] Cannot download files from File Storage even if the files are there

Wed Feb 5 11:14:49 UTC 2014

Hi Marco,

This has been happening enough that I spent some time writing a script to
generate that file. Can you test with https://github.com/ckan/ofs-hero and
let me know if that fixes your problems? Ideally, you'd want to backup your
existing persisted_state.json file before you replace it with the new one.

Please file bugs against the repo if you run into any issues with the
script.

Nigel Babu

Developer  |  @nigelbabu <https://twitter.com/nigelbabu>

The Open Knowledge Foundation <http://okfn.org/>

Empowering through Open Knowledge

http://okfn.org/  |  @okfn <http://twitter.com/OKFN>  |  OKF on
Facebook<https://www.facebook.com/OKFNetwork> |
Blog <http://blog.okfn.org/>  |  Newsletter<http://okfn.org/about/newsletter>

 CKAN | http://ckan.org/ | @CKANproject
<http://twitter.com/CKANproject> |the world’s leading open-source data
portal platform

On 4 February 2014 18:37, Nigel Babu <nigel.babu at okfn.org> wrote:

> Hey Marco,
>
> We've seen this happen before and I'm looking into it to see what can be
> done. I'll update the list when I have a solution. Future versions of CKAN
> use a new filestore should not run into this problem.
>
> Nigel Babu
>
> Developer  |  @nigelbabu <https://twitter.com/nigelbabu>
>
> The Open Knowledge Foundation <http://okfn.org/>
>
> Empowering through Open Knowledge
>
> http://okfn.org/  |  @okfn <http://twitter.com/OKFN>  |  OKF on Facebook<https://www.facebook.com/OKFNetwork> |
> Blog <http://blog.okfn.org/>  |  Newsletter<http://okfn.org/about/newsletter>
>
>  CKAN | http://ckan.org/ | @CKANproject <http://twitter.com/CKANproject> |the world’s leading open-source data portal platform
>
>
> On 4 February 2014 14:09, Marco Combetto <marcomb at hotmail.com> wrote:
>
>> Hi
>>
>> Yes. We have this file and it seems not containing all filenames.
>> I have the feeling that at some point it was corrupted.
>>
>> Is there a way to rebuild it?
>>
>>
>> Marco Combetto
>> marcomb at hotmail.com
>> Mobile: +39 3316329971
>> www.*linkedin*.com/in/*marco*mb
>>
>>
>>
>>
>> ------------------------------
>> From: nigel.babu at okfn.org
>> Date: Tue, 4 Feb 2014 09:42:08 +0530
>> To: ckan-dev at lists.okfn.org
>> Subject: Re: [ckan-dev] Cannot download files from File Storage even if
>> the files are there
>>
>>
>> Hi Marco,
>>
>> Can you look at your storage folder? There should be a JSON file called
>> "persisted_state.json". Check if the files you're looking for are in there.
>>
>> Nigel Babu
>>
>> Developer  |  @nigelbabu <https://twitter.com/nigelbabu>
>>
>> The Open Knowledge Foundation <http://okfn.org/>
>>
>> Empowering through Open Knowledge
>>
>> http://okfn.org/  |  @okfn <http://twitter.com/OKFN>  |  OKF on Facebook<https://www.facebook.com/OKFNetwork> |
>> Blog <http://blog.okfn.org/>  |  Newsletter<http://okfn.org/about/newsletter>
>>
>>  CKAN | http://ckan.org/ | @CKANproject <http://twitter.com/CKANproject>|the world’s leading open-source data portal platform
>>
>>
>> On 3 February 2014 21:45, Marco Combetto <marcomb at hotmail.com> wrote:
>>
>> Hallo
>>
>> I am facing a critical problem on our CKAN production site (we are
>> running a 2.2a version, release early December)
>>
>> http://dati.trentino.it
>>
>> From last Friday (30 Jan 2014)  our users are unable to download any
>>  file stored on our the CKAN storage
>> except the one uploaded after 30 January
>>
>> I absure you, we did't any deploy or any update on CKAN
>> code/configuration from December 2013
>> Of course, we have users (Group Admin, not SYS ADM) working on (using
>> the Web Interface, they were updating some metadata)
>>
>>  The tipical error we see in the log is:
>>
>> [Mon Feb 03 12:02:08 2014] [error] [client 172.31.50.18] Error - <class
>> 'pairtree.storage_exceptions.FileNotFoundException'>:
>> [Mon Feb 03 12:02:12 2014] [error] 2014-02-03 12:02:12,510 INFO
>>  [ckan.lib.base]  /dataset render time 0.149 seconds
>> [Mon Feb 03 12:02:14 2014] [error] 2014-02-03 12:02:14,046 INFO
>>  [ckan.lib.base]  /dataset render time 0.092 seconds
>> [Mon Feb 03 12:02:14 2014] [error] 2014-02-03 12:02:14,115 INFO
>>  [ckan.lib.base]  /storage/f/2013-06-20T093441/MITT---OpenData---v2.pdf
>> render time 5.292 seconds
>> [Mon Feb 03 12:02:14 2014] [error] 2014-02-03 12:02:14,278 INFO
>>  [ckan.lib.base]  /api/i18n/it render time 0.002 seconds
>> [Mon Feb 03 12:02:16 2014] [error] 2014-02-03 12:02:16,614 WARNI
>> [ckan.lib.maintain] Use `c.search_facets` instead.
>> [Mon Feb 03 12:02:16 2014] [error] 2014-02-03 12:02:16,741 INFO
>>  [ckan.lib.base]  /group/e08070e1-f829-43cf-af0a-83ee6f1e4c3f render time
>> 0.246 seconds
>>
>> Please note that "MITT---OpenData---v2.pdf" is the file that I am trying
>> to download (example)
>>
>> We are sure that pairtree storage is configured well (we did the
>> configuration 1 month ago and everything was secured)
>>
>> We rebuild SOLR indexes, we tried to step back with a POSTGRES backup
>> dated 29 January, we delete the cache.. we triple checked the
>> configuration... but no way to sort out the problem described before
>>
>> We have a file storage of 19GB of files.. (more than 1 year activity,
>> moving from 1.8 to 2.2)
>> The total number of files is 13159 files in 9055
>>
>> Any body faced similar problem?
>> Any suggestion of what could be happened?
>> Any suggestion to help us on troubleshooting this file storage management
>>
>>
>>
>> Marco Combetto
>> marcomb at hotmail.com
>> Mobile: +39 3316329971
>> www.*linkedin*.com/in/*marco*mb
>>
>>
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>
>>
>>
>> _______________________________________________ ckan-dev mailing list
>> ckan-dev at lists.okfn.org https://lists.okfn.org/mailman/listinfo/ckan-devUnsubscribe:
>> https://lists.okfn.org/mailman/options/ckan-dev
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20140205/2b8d17ea/attachment-0003.html>