[ckan-dev] Help! - data resources gone missing (Aaron McGlinchy)

Florian May florian.wendelin.mayer at gmail.com
Fri Oct 24 07:01:55 UTC 2014 

Commented on https://github.com/ckan/ckan/issues/1779

On Fri, Oct 24, 2014 at 2:52 PM, Florian May <
florian.wendelin.mayer at gmail.com> wrote:

> Hi all,
>
> I'm also running against this bug in CKAN 2.2b, which is very
> disconcerting for my end users as they lose attached files without a chance
> of getting them back.
> Has a bug already been reported?
>
> I managed to work around the bug in a very specific case - I
> programmatically upload working versions of annual reports to our CKAN, and
> as the upload iterates over the 16 attached resources, some of them
> sometimes get marked as "deleted". As I have configured the upload, I've
> got the resource_id - my end users don't have this luxury! My workaround is
> to retrieve all resource dicts (gettable api, resource_show), set their
> status to "active", retrieve the parent dataset's dict (package_show), then
> update the parent's dictionary's "resources" key with the list of resource
> dictionaries, and finally update the package using this modified package
> dictionary. A short write-up is here:
> http://data-demo.dpaw.wa.gov.au/dataset/report-automation/resource/ad551de0-5ef2-49e9-8b33-62de737a5aca
>
> Cheers,
> Florian
>
> Dept Parks & Wildlife, Western Australia
>
>
> On Tue, Jun 10, 2014 at 1:16 PM, Nigel Babu <nigel.babu at okfn.org> wrote:
>
>> Hi Aaron,
>>
>> Yes, it does sound like it's a bug. Can you file one against
>> github.com/ckan/ckan? I'll try to reproduce the issue locally meanwhile
>> and
>> confirm what's going on.
>>
>> On Mon, Jun 09, 2014 at 03:39:21AM +0000, Aaron McGlinchy wrote:
>> > Bump on my query as to whether this is a further bug (after we applied
>> the fix for the first issue it seemed to be resolved, but similar issue
>> arose when deleting resources).
>> >
>> > ------------------------------
>> >
>> > Message: 3
>> > Date: Wed, 28 May 2014 22:42:36 +0000
>> > From: Aaron McGlinchy <McGlinchyA at landcareresearch.co.nz>
>> > To: "ckan-dev at lists.okfn.org" <ckan-dev at lists.okfn.org>
>> > Subject: Re: [ckan-dev] Help! - data resources gone missing
>> > Message-ID:
>> >         <
>> A924326C14714F409FB3703B815FF9FD1225F672 at HERMES.landcare.ad.landcareresearch.co.nz
>> >
>> >
>> > Content-Type: text/plain; charset="us-ascii"
>> >
>> > We updated the datapusher to this commit
>> d433bd8a20ffd82ea684dc0c2a63bfc11dda5ada.
>> >
>> > This seemed to fix the problem of resources disappearing during
>> uploading of other resources.  However today I just found that when I went
>> into a dataset that had multiple (6) resources, and decided to delete one
>> of them (the 3rd in the list in this case), on completion of the delete
>> resources 4,5 and 6 had disappeared as well.
>> >
>> > When I deleted the resource, I went into Manage | Edit resource,
>> clicked on the Remove button, and then clicked the Delete button.  Possibly
>> removing the file/link first then deleting has triggered an additional
>> error.
>> >
>> > Does this sound like a further bug?
>> >
>> > Cheers
>> > Aaron
>> > ------------------------------
>> >
>> > Message: 2
>> > Date: Tue, 20 May 2014 23:39:58 +0000
>> > From: Aaron McGlinchy <McGlinchyA at landcareresearch.co.nz>
>> > To: "ckan-dev at lists.okfn.org" <ckan-dev at lists.okfn.org>
>> > Subject: Re: [ckan-dev] Help! - data resources gone missing
>> > Message-ID:
>> >         <
>> A924326C14714F409FB3703B815FF9FD1225C186 at HERMES.landcare.ad.landcareresearch.co.nz
>> >
>> >
>> > Content-Type: text/plain; charset="us-ascii"
>> >
>> > Aaargh!  It has happened again, this time I did not attempt to reorder
>> resources (I did edit some resource descriptions, and spontaneousy all but
>> one of the approx 5 resources I had uploaded disappeared).
>> >
>> > Nigel - can you please point me to the link for the branch you think
>> fixes this issue, I will ask my Tech guy to implement it (is it difficult
>> to do/any instructions I can point him too)?
>> >
>> > Thanks
>> > Aaron
>> >
>> > ------------------------------
>> >
>> > Message: 2
>> > Date: Mon, 19 May 2014 09:44:42 +0530
>> > From: Nigel Babu <nigel.babu at okfn.org>
>> > To: CKAN Development Discussions <ckan-dev at lists.okfn.org>
>> > Subject: Re: [ckan-dev] Help! - data resources gone missing
>> > Message-ID:
>> >         <CALGFUD7NG8UDo1+cBeBbTGv=
>> chmYXTuMsottzdD370pdYoxGBg at mail.gmail.com>
>> > Content-Type: text/plain; charset="utf-8"
>> >
>> > Hi Alex and Aaron,
>> >
>> > I believe the latest commit in the stable branch fixes this issue. I'll
>> arrange for an email out to ckan-announce today.
>> >
>> > Nigel Babu
>> > Developer, Open Knowledge
>> >
>> >
>> > On 19 May 2014 09:10, Aaron McGlinchy <
>> McGlinchyA at landcareresearch.co.nz>wrote:
>> >
>> > >  We are also on CKAN 2.2.
>> > >
>> > >
>> > >
>> > > Like Alex the files that disappeared have had their state set to
>> > > deleted, however setting it back to active and then switching the
>> > > dataset from private-public-private did not make them appear again.
>> > > The files that disappeared on me we some large zip files, a couple of
>> jpgs, tsv, and pdf ?
>> > > not limieted to csv files as Alex experienced.
>> > >
>> > >
>> > >
>> > > I noticed that one of the files position integer setting was
>> > > duplicated, so I changed it (the xlsx file) to 12 to avoid a conflict,
>> > > but the resources still did not show up (apart from 1).  Have decided
>> > > to delete the original dataset and reload, being careful to put in the
>> > > order I want so I don?t need to try reordering which seems to trigger
>> the problem.
>> > >
>> > >
>> > >
>> > > [image: cid:image002.png at 01CF7368.09A36550]
>> > >
>> > >
>> > >
>> > > ------------------------------
>> > >
>> > >
>> > >
>> > > Message: 3
>> > >
>> > > Date: Sat, 17 May 2014 17:52:25 +1000
>> > >
>> > > From: "Alex (Maxious) Sadleir" <maxious at gmail.com>
>> > >
>> > > To: CKAN Development Discussions <ckan-dev at lists.okfn.org>
>> > >
>> > > Subject: Re: [ckan-dev] Help! - data resources gone missing
>> > >
>> > > Message-ID:
>> > >
>> > >                 <
>> > > CAL3EF=9P=JbdfZJ4DL3aVKnYWC_uZqvY=Qsy3Kqi+oJ84ROA-g at mail.gmail.com>
>> > >
>> > > Content-Type: text/plain; charset=UTF-8
>> > >
>> > >
>> > >
>> > > Haven't upgraded datapusher since the 2.2 release so fbc6dc3 on stable
>> > > branch.
>> > >
>> > >
>> > >
>> > > I see there are some fixes so will deploy and see if it is resolved,
>> > > maybe if this is a known severe issue let ckan-announce know?
>> > >
>> > >
>> > >
>> > > On Thu, May 15, 2014 at 3:38 PM, Nigel Babu <nigel.babu at okfn.org>
>> wrote:
>> > >
>> > > > This sounds terrible. Can you tell me what's the latest commit in
>> > > > the
>> > >
>> > > > datapusher version you're using? I remember hearing about this bug
>> > > > and
>> > >
>> > > > us fixing it.
>> > >
>> > > >
>> > >
>> > > > Nigel Babu
>> > >
>> > > > Developer, Open Knowledge
>> > >
>> > > >
>> > >
>> > > >
>> > >
>> > > > On 15 May 2014 08:23, Alex (Maxious) Sadleir <maxious at gmail.com>
>> wrote:
>> > >
>> > > >>
>> > >
>> > > >> I have experienced this too - my best theory is it happens to
>> > >
>> > > >> datasets with very small CSV files and there is some kind of race
>> > >
>> > > >> condition in the DataPusher updating the resources.
>> > >
>> > > >>
>> > >
>> > > >> On datasets that it occurs (7 or so small CSVs), it occurs
>> > >
>> > > >> consistently after it occurs for the first time whenever another
>> > > >> file
>> > >
>> > > >> is added or existing file edited. Interestingly the resources are
>> > > >> all
>> > >
>> > > >> marked as deleted within ms of each other and if I manually set the
>> > >
>> > > >> state back to Active in postgres, I have to set the dataset Private
>> > >
>> > > >> and Public to trigger them to reappear.
>> > >
>> > > >>
>> > >
>> > > >> On Thu, May 15, 2014 at 12:43 PM, Aaron McGlinchy
>> > >
>> > > >> <McGlinchyA at landcareresearch.co.nz> wrote:
>> > >
>> > > >> > Hi,
>> > >
>> > > >> >     I created a dataset and had uploaded a number of resources
>> > >
>> > > >> > yesterday (about 10, of various file formats).
>> > >
>> > > >> >
>> > >
>> > > >> > Today I had some more resources to add, and I edited the dataset
>> > >
>> > > >> > and began uploading the additional resources.  Whilst doing so I
>> > >
>> > > >> > used the reorder resources button to arrange them in the order I
>> > >
>> > > >> > wanted.  When I did this with the first additional resource it
>> > >
>> > > >> > worked fine, but when I added another resource and went to
>> > > >> > reorder,
>> > >
>> > > >> > the list of resources to rearrange that came up only had a single
>> > >
>> > > >> > resource (the one just loaded) and all the ones I had loaded
>> > > >> > earlier
>> > > disappeared.
>> > >
>> > > >> >
>> > >
>> > > >> > When I clicked back to the Dataset it only showed that single
>> > > resource.
>> > >
>> > > >> > However I knew the link for some of the resources I had
>> > > >> > previously
>> > >
>> > > >> > loaded so I opened that and the resource I check is still there,
>> > >
>> > > >> > just not showing up under the dataset.
>> > >
>> > > >> >
>> > >
>> > > >> > What can I do to get them back listed under the dataset?
>> > >
>> > > >> >
>> > >
>> > > >> > Thanks
>> > >
>> > > >> > Aaron
>> >
>> > ________________________________
>> >
>> > Please consider the environment before printing this email
>> > Warning: This electronic message together with any attachments is
>> confidential. If you receive it in error: (i) you must not read, use,
>> disclose, copy or retain it; (ii) please contact the sender immediately by
>> reply email and then delete the emails.
>> > The views expressed in this email may not be those of Landcare Research
>> New Zealand Limited. http://www.landcareresearch.co.nz
>> > _______________________________________________
>> > ckan-dev mailing list
>> > ckan-dev at lists.okfn.org
>> > https://lists.okfn.org/mailman/listinfo/ckan-dev
>> > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>
>> --
>> Nigel Babu
>> Developer, Open Knowledge
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20141024/0f20ecd0/attachment-0003.html>

More information about the ckan-dev mailing list