[CKAN-support] [Open Knowledge Foundation] Re: Problem

Mon May 26 00:22:15 UTC 2014

This seems to have fixed the problem, touch wood!

Thanks
Aaron

From: support at ckan.org [mailto:support at ckan.org]
Sent: Friday, 23 May 2014 8:39 p.m.
To: Aaron McGlinchy
Subject: [Open Knowledge Foundation] Re: Problem

##- Please type your reply above this line -##
[Nigel Babu]

Nigel Babu (Open Knowledge Foundation)

May 23 09:39

Hi Aaron,

As I've been saying on the mailing list, this is most definitely a bug on datapuser that we have fixed. The datapusher should be running on this commit -> d433bd8a20ffd82ea684dc0c2a63bfc11dda5ada. If it's not, see the instructions below to upgrade.

(assuming the user is root)
cd /usr/lib/ckan/datapusher/src/datapusher
git pull origin stable

Confirm with git log that the latest commit is d433bd8a20ffd82ea684dc0c2a63bfc11dda5ada. Please let us know if you run into trouble. We will be releasing 2.2.1 in the next few weeks which should have this fix.

[Adam McGre]

Adam McGreggor (Open Knowledge Foundation)

May 23 09:22

Aaron,

Thanks for getting in touch, and sorry to read that you (and others) are having problems -- particularly a stopper like that.

I'll see what information I can uncover, but hopefully you'll hear back soon from the dev-team with a fix. We can advise, or perhaps work with Michael to resolve this, as from memory, I don't think we have access to your installation.

Best wishes,

Adam

[Aaron McGl]

Aaron McGlinchy

May 23 09:17

Hi Adam,
The issue below is a major problem for us (disappearing data resources) just ahedo us releasing CKAN to our staff. I encountered it in my final testing using a real dataset with multiple resources of various formats. Three times I have been loading up data and all of a sudden all the previously loaded resources disappear (they're still in the database, but they don't show up under a datasets resources anymore).

We need this resolved quickly so we can get our staff using it asap (ahead of the other modifiactions we have tal;ked about).

Nigel below seemed to think there may be a fix for this problem? Under our support agreement days, can I get your guys to advise us how to apply the fix, or if necessary to do it for us.

Thanks
Aaron

-----Original Message-----
Send ckan-dev mailing list submissions to
ckan-dev at lists.okfn.org<mailto:ckan-dev at lists.okfn.org>

Today's Topics:

2. Re: Help! - data resources gone missing (Aaron McGlinchy)

----------------------------------------------------------------------
Message: 2
Date: Tue, 20 May 2014 23:39:58 +0000
From: Aaron McGlinchy <McGlinchyA at landcareresearch.co.nz<mailto:McGlinchyA at landcareresearch.co.nz>>
To: "ckan-dev at lists.okfn.org<mailto:ckan-dev at lists.okfn.org>" <ckan-dev at lists.okfn.org<mailto:ckan-dev at lists.okfn.org>>
Subject: Re: [ckan-dev] Help! - data resources gone missing
Message-ID:
<A924326C14714F409FB3703B815FF9FD1225C186 at HERMES.landcare.ad.landcareresearch.co.nz<mailto:A924326C14714F409FB3703B815FF9FD1225C186 at HERMES.landcare.ad.landcareresearch.co.nz>>

Content-Type: text/plain; charset="us-ascii"

Aaargh! It has happened again, this time I did not attempt to reorder resources (I did edit some resource descriptions, and spontaneousy all but one of the approx 5 resources I had uploaded disappeared).

Nigel - can you please point me to the link for the branch you think fixes this issue, I will ask my Tech guy to implement it (is it difficult to do/any instructions I can point him too)?

Thanks
Aaron

------------------------------

Message: 2
Date: Mon, 19 May 2014 09:44:42 +0530
From: Nigel Babu <nigel.babu at okfn.org<mailto:nigel.babu at okfn.org>>
To: CKAN Development Discussions <ckan-dev at lists.okfn.org<mailto:ckan-dev at lists.okfn.org>>
Subject: Re: [ckan-dev] Help! - data resources gone missing
Message-ID:
<CALGFUD7NG8UDo1+cBeBbTGv=chmYXTuMsottzdD370pdYoxGBg at mail.gmail.com<mailto:chmYXTuMsottzdD370pdYoxGBg at mail.gmail.com>>
Content-Type: text/plain; charset="utf-8"

Hi Alex and Aaron,

I believe the latest commit in the stable branch fixes this issue. I'll arrange for an email out to ckan-announce today.

Nigel Babu
Developer, Open Knowledge

On 19 May 2014 09:10, Aaron McGlinchy <McGlinchyA at landcareresearch.co.nz<mailto:McGlinchyA at landcareresearch.co.nz>>wrote:

> We are also on CKAN 2.2.
>
>
>
> Like Alex the files that disappeared have had their state set to
> deleted, however setting it back to active and then switching the
> dataset from private-public-private did not make them appear again.
> The files that disappeared on me we some large zip files, a couple of jpgs, tsv, and pdf ?
> not limieted to csv files as Alex experienced.
>
>
>
> I noticed that one of the files position integer setting was
> duplicated, so I changed it (the xlsx file) to 12 to avoid a conflict,
> but the resources still did not show up (apart from 1). Have decided
> to delete the original dataset and reload, being careful to put in the
> order I want so I don?t need to try reordering which seems to trigger the problem.
>
>
>
> [image: cid:image002.png at 01CF7368.09A36550<mailto:image002.png at 01CF7368.09A36550>]
>
>
>
> ------------------------------
>
>
>
> Message: 3
>
> Date: Sat, 17 May 2014 17:52:25 +1000
>
> From: "Alex (Maxious) Sadleir" <maxious at gmail.com<mailto:maxious at gmail.com>>
>
> To: CKAN Development Discussions <ckan-dev at lists.okfn.org<mailto:ckan-dev at lists.okfn.org>>
>
> Subject: Re: [ckan-dev] Help! - data resources gone missing
>
> Message-ID:
>
> <
> CAL3EF=9P=JbdfZJ4DL3aVKnYWC_uZqvY=Qsy3Kqi+oJ84ROA-g at mail.gmail.com<mailto:Qsy3Kqi+oJ84ROA-g at mail.gmail.com>>
>
> Content-Type: text/plain; charset=UTF-8
>
>
>
> Haven't upgraded datapusher since the 2.2 release so fbc6dc3 on stable
> branch.
>
>
>
> I see there are some fixes so will deploy and see if it is resolved,
> maybe if this is a known severe issue let ckan-announce know?
>
>
>
> On Thu, May 15, 2014 at 3:38 PM, Nigel Babu <nigel.babu at okfn.org<mailto:nigel.babu at okfn.org>> wrote:
>
>> This sounds terrible. Can you tell me what's the latest commit in
>> the
>
>> datapusher version you're using? I remember hearing about this bug
>> and
>
>> us fixing it.
>
>>
>
>> Nigel Babu
>
>> Developer, Open Knowledge
>
>>
>
>>
>
>> On 15 May 2014 08:23, Alex (Maxious) Sadleir <maxious at gmail.com<mailto:maxious at gmail.com>> wrote:
>
>>>
>
>>> I have experienced this too - my best theory is it happens to
>
>>> datasets with very small CSV files and there is some kind of race
>
>>> condition in the DataPusher updating the resources.
>
>>>
>
>>> On datasets that it occurs (7 or so small CSVs), it occurs
>
>>> consistently after it occurs for the first time whenever another
>>> file
>
>>> is added or existing file edited. Interestingly the resources are
>>> all
>
>>> marked as deleted within ms of each other and if I manually set the
>
>>> state back to Active in postgres, I have to set the dataset Private
>
>>> and Public to trigger them to reappear.
>
>>>
>
>>> On Thu, May 15, 2014 at 12:43 PM, Aaron McGlinchy
>
>>> <McGlinchyA at landcareresearch.co.nz<mailto:McGlinchyA at landcareresearch.co.nz>> wrote:
>
>>>> Hi,
>
>>>> I created a dataset and had uploaded a number of resources
>
>>>> yesterday (about 10, of various file formats).
>
>>>>
>
>>>> Today I had some more resources to add, and I edited the dataset
>
>>>> and began uploading the additional resources. Whilst doing so I
>
>>>> used the reorder resources button to arrange them in the order I
>
>>>> wanted. When I did this with the first additional resource it
>
>>>> worked fine, but when I added another resource and went to
>>>> reorder,
>
>>>> the list of resources to rearrange that came up only had a single
>
>>>> resource (the one just loaded) and all the ones I had loaded
>>>> earlier
> disappeared.
>
>>>>
>
>>>> When I clicked back to the Dataset it only showed that single
> resource.
>
>>>> However I knew the link for some of the resources I had
>>>> previously
>
>>>> loaded so I opened that and the resource I check is still there,
>
>>>> just not showing up under the dataset.
>
>>>>
>
>>>> What can I do to get them back listed under the dataset?
>
>>>>
>
>>>> Thanks
>
>>>> Aaron
>
>>>>
>
>>>> ________________________________
>
>>>>
>
>>>> Please consider the environment before printing this email
>
>>>> Warning: This electronic message together with any attachments is
>
>>>> confidential. If you receive it in error: (i) you must not read,
>
>>>> use, disclose, copy or retain it; (ii) please contact the sender
>
>>>> immediately by reply email and then delete the emails.
>
>>>> The views expressed in this email may not be those of Landcare
>
>>>> Research New Zealand Limited. http://www.landcareresearch.co.nz
>
>>>> _______________________________________________
>
>>>> ckan-dev mailing list
>
>>>> ckan-dev at lists.okfn.org<mailto:ckan-dev at lists.okfn.org>
>
>>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>
>>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>
>>> _______________________________________________
>
>>> ckan-dev mailing list
>
>>> ckan-dev at lists.okfn.org<mailto:ckan-dev at lists.okfn.org>
>
>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>
>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>
>>
>
>>
>
>>
>
>> _______________________________________________
>
>> ckan-dev mailing list
>
>> ckan-dev at lists.okfn.org<mailto:ckan-dev at lists.okfn.org>
>
>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>
>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>
>>
>
>
>
>
>
> ------------------------------
>
>
>
> Subject: Digest Footer
>
>
>
> _______________________________________________
>
> ckan-dev mailing list
>
> ckan-dev at lists.okfn.org<mailto:ckan-dev at lists.okfn.org>
>
> https://lists.okfn.org/mailman/listinfo/ckan-dev
>
> Unsubscribe: https://lists.okfn.org/mailman/optionss/ckan-dev
>
>
>
>
>
> ------------------------------
>
>
>
> End of ckan-dev Digest, Vol 43, Issue 62
>
> ****************************************
>
> ------------------------------
>
> Please consider the environment before printing this email
> Warning: This electronic message together with any attachments is
> confidential. If you receive it in error: (i) you must not read, use,
> disclose, copy or retain it; (ii) please contact the sender
> immediately by reply email and then delete the emails.
> The views expressed in this email may not be those of Landcare
> Research New Zealand Limited. http://www.landcareresearch.co.nz
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org<mailto:ckan-dev at lists.okfn.org>
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.okfn.org/pipermail/ckan-dev/attachments/20140519/a7430471/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 111689 bytes
Desc: not available
URL: <https://lists.okfn.org/pipermail/ckan-dev/attachments/20140519/a7430471/attachment.png>

------------------------------

Subject: Digest Footer

_______________________________________________
ckan-dev mailing list
ckan-dev at lists.okfn.org<mailto:ckan-dev at lists.okfn.org>
https://lists.okfn.org/mailman/listinfo/ckan-dev
Unsubscribe: https://lists.okfn.org/mailman/optionss/ckan-dev

------------------------------

End of ckan-dev Digest, Vol 43, Issue 65
****************************************

________________________________

Please consider the environment before printing this email
Warning: This electronic message together with any attachments is confidential. If you receive it in error: (i) you must not read, use, disclose, copy or retain it; (ii) please contact the sender immediately by reply email and then delete the emails.
The views expressed in this email may not be those of Landcare Research New Zealand Limited. http://www.landcareresearch.co.nz

------------------------------

Message: 3
Date: Wed, 21 May 2014 11:48:38 +0200
From: Sean Hammond <sean.hammond at okfn.org<mailto:sean.hammond at okfn.org>>
To: ckan-dev <ckan-dev at lists.okfn.org<mailto:ckan-dev at lists.okfn.org>>
Subject: Re: [ckan-dev] Datapusher and Datastore
Message-ID: <1400665651-sup-2539 at mistakenot>
Content-Type: text/plain; charset=UTF-8

> If I insert a csv resource that comes from another site, using its
> link, the datapusher will process it and add the resource content
> inside the datastore. What if the resource content has been changed?
> Does the datapusher handle this case or it will display the previous content?

It doesn't automatically detect chanes to remotely-hosted files. I think you can manually trigger a datapusher update though.

------------------------------

Subject: Digest Footer

_______________________________________________
ckan-dev mailing list
ckan-dev at lists.okfn.org<mailto:ckan-dev at lists.okfn.org>
https://lists.okfn.org/mailman/listinfo/ckan-dev
Unsubscribe: https://lists.okfn.org/mailman/optionss/ckan-dev

------------------------------

End of ckan-dev Digest, Vol 43, Issue 72
****************************************

________________________________

Please consider the environment before printing this email
Warning: This electronic message together with any attachments is confidential. If you receive it in error: (i) you must not read, use, disclose, copy or retain it; (ii) please contact the sender immediately by reply email and then delete the emails.
The views expressed in this email may not be those of Landcare Research New Zealand Limited. http://www.landcareresearch.co.nz

This email is a service from Open Knowledge Foundation. Delivered by Zendesk<http://www.zendesk.com>.
Message-Id:77WQE6Y7_537f092cd5c6b_16dd3feb17ec9e8c218454_sprut

________________________________

Please consider the environment before printing this email
Warning: This electronic message together with any attachments is confidential. If you receive it in error: (i) you must not read, use, disclose, copy or retain it; (ii) please contact the sender immediately by reply email and then delete the emails.
The views expressed in this email may not be those of Landcare Research New Zealand Limited. http://www.landcareresearch.co.nz
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.okfn.org/mailman/private/ckan-support/attachments/20140526/88214a98/attachment-0003.html>