[ckan-dev] Help! - data resources gone missing (Aaron McGlinchy)

Nigel Babu nigel.babu at okfn.org
Tue Jun 10 05:16:47 UTC 2014


Hi Aaron,

Yes, it does sound like it's a bug. Can you file one against
github.com/ckan/ckan? I'll try to reproduce the issue locally meanwhile and
confirm what's going on.

On Mon, Jun 09, 2014 at 03:39:21AM +0000, Aaron McGlinchy wrote:
> Bump on my query as to whether this is a further bug (after we applied the fix for the first issue it seemed to be resolved, but similar issue arose when deleting resources).
>
> ------------------------------
>
> Message: 3
> Date: Wed, 28 May 2014 22:42:36 +0000
> From: Aaron McGlinchy <McGlinchyA at landcareresearch.co.nz>
> To: "ckan-dev at lists.okfn.org" <ckan-dev at lists.okfn.org>
> Subject: Re: [ckan-dev] Help! - data resources gone missing
> Message-ID:
>         <A924326C14714F409FB3703B815FF9FD1225F672 at HERMES.landcare.ad.landcareresearch.co.nz>
>
> Content-Type: text/plain; charset="us-ascii"
>
> We updated the datapusher to this commit d433bd8a20ffd82ea684dc0c2a63bfc11dda5ada.
>
> This seemed to fix the problem of resources disappearing during uploading of other resources.  However today I just found that when I went into a dataset that had multiple (6) resources, and decided to delete one of them (the 3rd in the list in this case), on completion of the delete resources 4,5 and 6 had disappeared as well.
>
> When I deleted the resource, I went into Manage | Edit resource, clicked on the Remove button, and then clicked the Delete button.  Possibly removing the file/link first then deleting has triggered an additional error.
>
> Does this sound like a further bug?
>
> Cheers
> Aaron
> ------------------------------
>
> Message: 2
> Date: Tue, 20 May 2014 23:39:58 +0000
> From: Aaron McGlinchy <McGlinchyA at landcareresearch.co.nz>
> To: "ckan-dev at lists.okfn.org" <ckan-dev at lists.okfn.org>
> Subject: Re: [ckan-dev] Help! - data resources gone missing
> Message-ID:
>         <A924326C14714F409FB3703B815FF9FD1225C186 at HERMES.landcare.ad.landcareresearch.co.nz>
>
> Content-Type: text/plain; charset="us-ascii"
>
> Aaargh!  It has happened again, this time I did not attempt to reorder resources (I did edit some resource descriptions, and spontaneousy all but one of the approx 5 resources I had uploaded disappeared).
>
> Nigel - can you please point me to the link for the branch you think fixes this issue, I will ask my Tech guy to implement it (is it difficult to do/any instructions I can point him too)?
>
> Thanks
> Aaron
>
> ------------------------------
>
> Message: 2
> Date: Mon, 19 May 2014 09:44:42 +0530
> From: Nigel Babu <nigel.babu at okfn.org>
> To: CKAN Development Discussions <ckan-dev at lists.okfn.org>
> Subject: Re: [ckan-dev] Help! - data resources gone missing
> Message-ID:
>         <CALGFUD7NG8UDo1+cBeBbTGv=chmYXTuMsottzdD370pdYoxGBg at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Hi Alex and Aaron,
>
> I believe the latest commit in the stable branch fixes this issue. I'll arrange for an email out to ckan-announce today.
>
> Nigel Babu
> Developer, Open Knowledge
>
>
> On 19 May 2014 09:10, Aaron McGlinchy <McGlinchyA at landcareresearch.co.nz>wrote:
>
> >  We are also on CKAN 2.2.
> >
> >
> >
> > Like Alex the files that disappeared have had their state set to
> > deleted, however setting it back to active and then switching the
> > dataset from private-public-private did not make them appear again.
> > The files that disappeared on me we some large zip files, a couple of jpgs, tsv, and pdf ?
> > not limieted to csv files as Alex experienced.
> >
> >
> >
> > I noticed that one of the files position integer setting was
> > duplicated, so I changed it (the xlsx file) to 12 to avoid a conflict,
> > but the resources still did not show up (apart from 1).  Have decided
> > to delete the original dataset and reload, being careful to put in the
> > order I want so I don?t need to try reordering which seems to trigger the problem.
> >
> >
> >
> > [image: cid:image002.png at 01CF7368.09A36550]
> >
> >
> >
> > ------------------------------
> >
> >
> >
> > Message: 3
> >
> > Date: Sat, 17 May 2014 17:52:25 +1000
> >
> > From: "Alex (Maxious) Sadleir" <maxious at gmail.com>
> >
> > To: CKAN Development Discussions <ckan-dev at lists.okfn.org>
> >
> > Subject: Re: [ckan-dev] Help! - data resources gone missing
> >
> > Message-ID:
> >
> >                 <
> > CAL3EF=9P=JbdfZJ4DL3aVKnYWC_uZqvY=Qsy3Kqi+oJ84ROA-g at mail.gmail.com>
> >
> > Content-Type: text/plain; charset=UTF-8
> >
> >
> >
> > Haven't upgraded datapusher since the 2.2 release so fbc6dc3 on stable
> > branch.
> >
> >
> >
> > I see there are some fixes so will deploy and see if it is resolved,
> > maybe if this is a known severe issue let ckan-announce know?
> >
> >
> >
> > On Thu, May 15, 2014 at 3:38 PM, Nigel Babu <nigel.babu at okfn.org> wrote:
> >
> > > This sounds terrible. Can you tell me what's the latest commit in
> > > the
> >
> > > datapusher version you're using? I remember hearing about this bug
> > > and
> >
> > > us fixing it.
> >
> > >
> >
> > > Nigel Babu
> >
> > > Developer, Open Knowledge
> >
> > >
> >
> > >
> >
> > > On 15 May 2014 08:23, Alex (Maxious) Sadleir <maxious at gmail.com> wrote:
> >
> > >>
> >
> > >> I have experienced this too - my best theory is it happens to
> >
> > >> datasets with very small CSV files and there is some kind of race
> >
> > >> condition in the DataPusher updating the resources.
> >
> > >>
> >
> > >> On datasets that it occurs (7 or so small CSVs), it occurs
> >
> > >> consistently after it occurs for the first time whenever another
> > >> file
> >
> > >> is added or existing file edited. Interestingly the resources are
> > >> all
> >
> > >> marked as deleted within ms of each other and if I manually set the
> >
> > >> state back to Active in postgres, I have to set the dataset Private
> >
> > >> and Public to trigger them to reappear.
> >
> > >>
> >
> > >> On Thu, May 15, 2014 at 12:43 PM, Aaron McGlinchy
> >
> > >> <McGlinchyA at landcareresearch.co.nz> wrote:
> >
> > >> > Hi,
> >
> > >> >     I created a dataset and had uploaded a number of resources
> >
> > >> > yesterday (about 10, of various file formats).
> >
> > >> >
> >
> > >> > Today I had some more resources to add, and I edited the dataset
> >
> > >> > and began uploading the additional resources.  Whilst doing so I
> >
> > >> > used the reorder resources button to arrange them in the order I
> >
> > >> > wanted.  When I did this with the first additional resource it
> >
> > >> > worked fine, but when I added another resource and went to
> > >> > reorder,
> >
> > >> > the list of resources to rearrange that came up only had a single
> >
> > >> > resource (the one just loaded) and all the ones I had loaded
> > >> > earlier
> > disappeared.
> >
> > >> >
> >
> > >> > When I clicked back to the Dataset it only showed that single
> > resource.
> >
> > >> > However I knew the link for some of the resources I had
> > >> > previously
> >
> > >> > loaded so I opened that and the resource I check is still there,
> >
> > >> > just not showing up under the dataset.
> >
> > >> >
> >
> > >> > What can I do to get them back listed under the dataset?
> >
> > >> >
> >
> > >> > Thanks
> >
> > >> > Aaron
>
> ________________________________
>
> Please consider the environment before printing this email
> Warning: This electronic message together with any attachments is confidential. If you receive it in error: (i) you must not read, use, disclose, copy or retain it; (ii) please contact the sender immediately by reply email and then delete the emails.
> The views expressed in this email may not be those of Landcare Research New Zealand Limited. http://www.landcareresearch.co.nz
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev

--
Nigel Babu
Developer, Open Knowledge



More information about the ckan-dev mailing list