[CKAN-support] Fwd: Problem

Fri May 23 08:16:54 UTC 2014

Begin forwarded message:

> From: Aaron McGlinchy <McGlinchyA at landcareresearch.co.nz>
> Subject: Problem
> Date: 23 May 2014 05:17:58 BST
> To: Adam McGreggor <adam.mcgreggor at okfn.org>
> Message-Id: <A924326C14714F409FB3703B815FF9FD1225E18A at HERMES.landcare.ad.landcareresearch.co.nz>
> 
> Hi Adam,
>        The issue below is a major problem for us (disappearing data resources) just ahedo us releasing CKAN to our staff.  I encountered it in my final testing using a real dataset with multiple resources of various formats.  Three times I have been loading up data and all of a sudden all the previously loaded resources disappear (they're still in the database, but they don't show up under a datasets resources anymore).
> 
> We need this resolved quickly so we can get our staff using it asap (ahead of the other modifiactions we have tal;ked about).
> 
> Nigel below seemed to think there may be a fix for this problem?  Under our support agreement days, can I get your guys to advise us how to apply the fix, or if necessary to do it for us.
> 
> Thanks
> Aaron
> 
> -----Original Message-----
> Send ckan-dev mailing list submissions to
>        ckan-dev at lists.okfn.org
> 
> Today's Topics:
> 
>   2. Re: Help! - data resources gone missing (Aaron McGlinchy)
> 
> ----------------------------------------------------------------------
> Message: 2
> Date: Tue, 20 May 2014 23:39:58 +0000
> From: Aaron McGlinchy <McGlinchyA at landcareresearch.co.nz>
> To: "ckan-dev at lists.okfn.org" <ckan-dev at lists.okfn.org>
> Subject: Re: [ckan-dev] Help! - data resources gone missing
> Message-ID:
>        <A924326C14714F409FB3703B815FF9FD1225C186 at HERMES.landcare.ad.landcareresearch.co.nz>
> 
> Content-Type: text/plain; charset="us-ascii"
> 
> Aaargh!  It has happened again, this time I did not attempt to reorder resources (I did edit some resource descriptions, and spontaneousy all but one of the approx 5 resources I had uploaded disappeared).
> 
> Nigel - can you please point me to the link for the branch you think fixes this issue, I will ask my Tech guy to implement it (is it difficult to do/any instructions I can point him too)?
> 
> Thanks
> Aaron
> 
> ------------------------------
> 
> Message: 2
> Date: Mon, 19 May 2014 09:44:42 +0530
> From: Nigel Babu <nigel.babu at okfn.org>
> To: CKAN Development Discussions <ckan-dev at lists.okfn.org>
> Subject: Re: [ckan-dev] Help! - data resources gone missing
> Message-ID:
>        <CALGFUD7NG8UDo1+cBeBbTGv=chmYXTuMsottzdD370pdYoxGBg at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
> 
> Hi Alex and Aaron,
> 
> I believe the latest commit in the stable branch fixes this issue. I'll arrange for an email out to ckan-announce today.
> 
> Nigel Babu
> Developer, Open Knowledge
> 
> 
> On 19 May 2014 09:10, Aaron McGlinchy <McGlinchyA at landcareresearch.co.nz>wrote:
> 
>> We are also on CKAN 2.2.
>> 
>> 
>> 
>> Like Alex the files that disappeared have had their state set to
>> deleted, however setting it back to active and then switching the
>> dataset from private-public-private did not make them appear again.
>> The files that disappeared on me we some large zip files, a couple of jpgs, tsv, and pdf ?
>> not limieted to csv files as Alex experienced.
>> 
>> 
>> 
>> I noticed that one of the files position integer setting was
>> duplicated, so I changed it (the xlsx file) to 12 to avoid a conflict,
>> but the resources still did not show up (apart from 1).  Have decided
>> to delete the original dataset and reload, being careful to put in the
>> order I want so I don?t need to try reordering which seems to trigger the problem.
>> 
>> 
>> 
>> [image: cid:image002.png at 01CF7368.09A36550]
>> 
>> 
>> 
>> ------------------------------
>> 
>> 
>> 
>> Message: 3
>> 
>> Date: Sat, 17 May 2014 17:52:25 +1000
>> 
>> From: "Alex (Maxious) Sadleir" <maxious at gmail.com>
>> 
>> To: CKAN Development Discussions <ckan-dev at lists.okfn.org>
>> 
>> Subject: Re: [ckan-dev] Help! - data resources gone missing
>> 
>> Message-ID:
>> 
>>                <
>> CAL3EF=9P=JbdfZJ4DL3aVKnYWC_uZqvY=Qsy3Kqi+oJ84ROA-g at mail.gmail.com>
>> 
>> Content-Type: text/plain; charset=UTF-8
>> 
>> 
>> 
>> Haven't upgraded datapusher since the 2.2 release so fbc6dc3 on stable
>> branch.
>> 
>> 
>> 
>> I see there are some fixes so will deploy and see if it is resolved,
>> maybe if this is a known severe issue let ckan-announce know?
>> 
>> 
>> 
>> On Thu, May 15, 2014 at 3:38 PM, Nigel Babu <nigel.babu at okfn.org> wrote:
>> 
>>> This sounds terrible. Can you tell me what's the latest commit in
>>> the
>> 
>>> datapusher version you're using? I remember hearing about this bug
>>> and
>> 
>>> us fixing it.
>> 
>>> 
>> 
>>> Nigel Babu
>> 
>>> Developer, Open Knowledge
>> 
>>> 
>> 
>>> 
>> 
>>> On 15 May 2014 08:23, Alex (Maxious) Sadleir <maxious at gmail.com> wrote:
>> 
>>>> 
>> 
>>>> I have experienced this too - my best theory is it happens to
>> 
>>>> datasets with very small CSV files and there is some kind of race
>> 
>>>> condition in the DataPusher updating the resources.
>> 
>>>> 
>> 
>>>> On datasets that it occurs (7 or so small CSVs), it occurs
>> 
>>>> consistently after it occurs for the first time whenever another
>>>> file
>> 
>>>> is added or existing file edited. Interestingly the resources are
>>>> all
>> 
>>>> marked as deleted within ms of each other and if I manually set the
>> 
>>>> state back to Active in postgres, I have to set the dataset Private
>> 
>>>> and Public to trigger them to reappear.
>> 
>>>> 
>> 
>>>> On Thu, May 15, 2014 at 12:43 PM, Aaron McGlinchy
>> 
>>>> <McGlinchyA at landcareresearch.co.nz> wrote:
>> 
>>>>> Hi,
>> 
>>>>>    I created a dataset and had uploaded a number of resources
>> 
>>>>> yesterday (about 10, of various file formats).
>> 
>>>>> 
>> 
>>>>> Today I had some more resources to add, and I edited the dataset
>> 
>>>>> and began uploading the additional resources.  Whilst doing so I
>> 
>>>>> used the reorder resources button to arrange them in the order I
>> 
>>>>> wanted.  When I did this with the first additional resource it
>> 
>>>>> worked fine, but when I added another resource and went to
>>>>> reorder,
>> 
>>>>> the list of resources to rearrange that came up only had a single
>> 
>>>>> resource (the one just loaded) and all the ones I had loaded
>>>>> earlier
>> disappeared.
>> 
>>>>> 
>> 
>>>>> When I clicked back to the Dataset it only showed that single
>> resource.
>> 
>>>>> However I knew the link for some of the resources I had
>>>>> previously
>> 
>>>>> loaded so I opened that and the resource I check is still there,
>> 
>>>>> just not showing up under the dataset.
>> 
>>>>> 
>> 
>>>>> What can I do to get them back listed under the dataset?
>> 
>>>>> 
>> 
>>>>> Thanks
>> 
>>>>> Aaron
>> 
>>>>> 
>> 
>>>>> ________________________________
>> 
>>>>> 
>> 
>>>>> Please consider the environment before printing this email
>> 
>>>>> Warning: This electronic message together with any attachments is
>> 
>>>>> confidential. If you receive it in error: (i) you must not read,
>> 
>>>>> use, disclose, copy or retain it; (ii) please contact the sender
>> 
>>>>> immediately by reply email and then delete the emails.
>> 
>>>>> The views expressed in this email may not be those of Landcare
>> 
>>>>> Research New Zealand Limited. http://www.landcareresearch.co.nz
>> 
>>>>> _______________________________________________
>> 
>>>>> ckan-dev mailing list
>> 
>>>>> ckan-dev at lists.okfn.org
>> 
>>>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> 
>>>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>> 
>>>> _______________________________________________
>> 
>>>> ckan-dev mailing list
>> 
>>>> ckan-dev at lists.okfn.org
>> 
>>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> 
>>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>> 
>>> 
>> 
>>> 
>> 
>>> 
>> 
>>> _______________________________________________
>> 
>>> ckan-dev mailing list
>> 
>>> ckan-dev at lists.okfn.org
>> 
>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> 
>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>> 
>>> 
>> 
>> 
>> 
>> 
>> 
>> ------------------------------
>> 
>> 
>> 
>> Subject: Digest Footer
>> 
>> 
>> 
>> _______________________________________________
>> 
>> ckan-dev mailing list
>> 
>> ckan-dev at lists.okfn.org
>> 
>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> 
>> Unsubscribe: https://lists.okfn.org/mailman/optionss/ckan-dev
>> 
>> 
>> 
>> 
>> 
>> ------------------------------
>> 
>> 
>> 
>> End of ckan-dev Digest, Vol 43, Issue 62
>> 
>> ****************************************
>> 
>> ------------------------------
>> 
>> Please consider the environment before printing this email
>> Warning: This electronic message together with any attachments is
>> confidential. If you receive it in error: (i) you must not read, use,
>> disclose, copy or retain it; (ii) please contact the sender
>> immediately by reply email and then delete the emails.
>> The views expressed in this email may not be those of Landcare
>> Research New Zealand Limited. http://www.landcareresearch.co.nz
>> 
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>> 
>> 
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <https://lists.okfn.org/pipermail/ckan-dev/attachments/20140519/a7430471/attachment.html>
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: image001.png
> Type: image/png
> Size: 111689 bytes
> Desc: not available
> URL: <https://lists.okfn.org/pipermail/ckan-dev/attachments/20140519/a7430471/attachment.png>
> 
> ------------------------------
> 
> Subject: Digest Footer
> 
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/optionss/ckan-dev
> 
> 
> ------------------------------
> 
> End of ckan-dev Digest, Vol 43, Issue 65
> ****************************************
> 
> ________________________________
> 
> Please consider the environment before printing this email
> Warning: This electronic message together with any attachments is confidential. If you receive it in error: (i) you must not read, use, disclose, copy or retain it; (ii) please contact the sender immediately by reply email and then delete the emails.
> The views expressed in this email may not be those of Landcare Research New Zealand Limited. http://www.landcareresearch.co.nz
> 
> 
> ------------------------------
> 
> Message: 3
> Date: Wed, 21 May 2014 11:48:38 +0200
> From: Sean Hammond <sean.hammond at okfn.org>
> To: ckan-dev <ckan-dev at lists.okfn.org>
> Subject: Re: [ckan-dev] Datapusher and Datastore
> Message-ID: <1400665651-sup-2539 at mistakenot>
> Content-Type: text/plain; charset=UTF-8
> 
>> If I insert a csv resource that comes from another site, using its
>> link, the datapusher will process it and add the resource content
>> inside the datastore. What if the resource content has been changed?
>> Does the datapusher handle this case or it will display the previous content?
> 
> It doesn't automatically detect chanes to remotely-hosted files. I think you can manually trigger a datapusher update though.
> 
> 
> ------------------------------
> 
> Subject: Digest Footer
> 
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/optionss/ckan-dev
> 
> 
> ------------------------------
> 
> End of ckan-dev Digest, Vol 43, Issue 72
> ****************************************
> 
> ________________________________
> 
> Please consider the environment before printing this email
> Warning: This electronic message together with any attachments is confidential. If you receive it in error: (i) you must not read, use, disclose, copy or retain it; (ii) please contact the sender immediately by reply email and then delete the emails.
> The views expressed in this email may not be those of Landcare Research New Zealand Limited. http://www.landcareresearch.co.nz