[ckan-dev] Cannot download files from File Storage even if the files are there

Wed Feb 5 14:02:27 UTC 2014

Great! We are testing it. Of course we will report you asap
Thanks!

Marco Combetto
marcomb at hotmail.com
Mobile: +39 3316329971
www.linkedin.com/in/marcomb 

From: nigel.babu at okfn.org
Date: Wed, 5 Feb 2014 16:44:49 +0530
To: ckan-dev at lists.okfn.org
Subject: Re: [ckan-dev] Cannot download files from File Storage even if the files are there

Hi Marco,

This has been happening enough that I spent some time writing a script to generate that file. Can you test with https://github.com/ckan/ofs-hero and let me know if that fixes your problems? Ideally, you'd want to backup your existing persisted_state.json file before you replace it with the new one.

Please file bugs against the repo if you run into any issues with the script.

Nigel Babu

Developer  |  @nigelbabu

The Open Knowledge Foundation

Empowering through Open Knowledge

http://okfn.org/  |  @okfn  |  OKF on Facebook  |  Blog  |  Newsletter

CKAN | http://ckan.org/ | @CKANproject | the world’s leading open-source data portal platform

On 4 February 2014 18:37, Nigel Babu <nigel.babu at okfn.org> wrote:

Hey Marco,

We've seen this happen before and I'm looking into it to see what can be done. I'll update the list when I have a solution. Future versions of CKAN use a new filestore should not run into this problem.

Nigel Babu

Developer  |  @nigelbabu

The Open Knowledge Foundation

Empowering through Open Knowledge

http://okfn.org/  |  @okfn  |  OKF on Facebook  |  Blog  |  Newsletter

CKAN | http://ckan.org/ | @CKANproject | the world’s leading open-source data portal platform

On 4 February 2014 14:09, Marco Combetto <marcomb at hotmail.com> wrote:

Hi
Yes. We have this file and it seems not containing all filenames. I have the feeling that at some point it was corrupted.
Is there a way to rebuild it?

Marco Combetto
marcomb at hotmail.com
Mobile: +39 3316329971
www.linkedin.com/in/marcomb 

From: nigel.babu at okfn.org
Date: Tue, 4 Feb 2014 09:42:08 +0530
To: ckan-dev at lists.okfn.org

Subject: Re: [ckan-dev] Cannot download files from File Storage even if the files are there

Hi Marco,

Can you look at your storage folder? There should be a JSON file called "persisted_state.json". Check if the files you're looking for are in there.

Nigel Babu

Developer  |  @nigelbabu

The Open Knowledge Foundation

Empowering through Open Knowledge

http://okfn.org/  |  @okfn  |  OKF on Facebook  |  Blog  |  Newsletter

CKAN | http://ckan.org/ | @CKANproject | the world’s leading open-source data portal platform

On 3 February 2014 21:45, Marco Combetto <marcomb at hotmail.com> wrote:

Hallo
I am facing a critical problem on our CKAN production site (we are running a 2.2a version, release early December)
http://dati.trentino.it

>From last Friday (30 Jan 2014)  our users are unable to download any  file stored on our the CKAN storageexcept the one uploaded after 30 January
I absure you, we did't any deploy or any update on CKAN code/configuration from December 2013

Of course, we have users (Group Admin, not SYS ADM) working on (using the Web Interface, they were updating some metadata)

The tipical error we see in the log is:
[Mon Feb 03 12:02:08 2014] [error] [client 172.31.50.18] Error - <class 'pairtree.storage_exceptions.FileNotFoundException'>: [Mon Feb 03 12:02:12 2014] [error] 2014-02-03 12:02:12,510 INFO  [ckan.lib.base]  /dataset render time 0.149 seconds

[Mon Feb 03 12:02:14 2014] [error] 2014-02-03 12:02:14,046 INFO  [ckan.lib.base]  /dataset render time 0.092 seconds[Mon Feb 03 12:02:14 2014] [error] 2014-02-03 12:02:14,115 INFO  [ckan.lib.base]  /storage/f/2013-06-20T093441/MITT---OpenData---v2.pdf render time 5.292 seconds

[Mon Feb 03 12:02:14 2014] [error] 2014-02-03 12:02:14,278 INFO  [ckan.lib.base]  /api/i18n/it render time 0.002 seconds[Mon Feb 03 12:02:16 2014] [error] 2014-02-03 12:02:16,614 WARNI [ckan.lib.maintain] Use `c.search_facets` instead.

[Mon Feb 03 12:02:16 2014] [error] 2014-02-03 12:02:16,741 INFO  [ckan.lib.base]  /group/e08070e1-f829-43cf-af0a-83ee6f1e4c3f render time 0.246 seconds
Please note that "MITT---OpenData---v2.pdf" is the file that I am trying to download (example)

We are sure that pairtree storage is configured well (we did the configuration 1 month ago and everything was secured)
We rebuild SOLR indexes, we tried to step back with a POSTGRES backup dated 29 January, we delete the cache.. we triple checked the configuration... but no way to sort out the problem described before

We have a file storage of 19GB of files.. (more than 1 year activity, moving from 1.8 to 2.2) The total number of files is 13159 files in 9055
Any body faced similar problem?  

Any suggestion of what could be happened?Any suggestion to help us on troubleshooting this file storage management

Marco Combetto

marcomb at hotmail.com
Mobile: +39 3316329971
www.linkedin.com/in/marcomb 

_______________________________________________

ckan-dev mailing list

ckan-dev at lists.okfn.org

https://lists.okfn.org/mailman/listinfo/ckan-dev

Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev

_______________________________________________
ckan-dev mailing list
ckan-dev at lists.okfn.org
https://lists.okfn.org/mailman/listinfo/ckan-dev
Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev 		 	   		  

_______________________________________________

ckan-dev mailing list

ckan-dev at lists.okfn.org

https://lists.okfn.org/mailman/listinfo/ckan-dev

Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev

_______________________________________________
ckan-dev mailing list
ckan-dev at lists.okfn.org
https://lists.okfn.org/mailman/listinfo/ckan-dev
Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20140205/5e92c7c2/attachment-0003.html>