[ckan-dev] problems upgrading to CKAN 2.2

Thu Jan 30 10:34:50 UTC 2014

I'll try to reproduce it today with some harvesting DBs to see if we
can get to the bottom of this.
If we don't by Tuesday, then yes, we might need to delay the 2.2 release.

Adrià

On 30 January 2014 10:06, David Read <david.read at hackneyworkshop.com> wrote:
> We're also suffering this same exception on DGU since upgrading to
> CKAN 2.2. We see metadata_modified bring set to null when creating
> some new datasets.
>
> I've not reproduced it on my own machine or a test machine - just on
> our live machine, which is harder to play with. We've restarted
> harvesters and queues etc but still no joy. I've not managed to get a
> handle on what exactly causes it, but will have some more time to look
> at it next week.
>
> I believe it happens occasionally, with harvested datasets. Since it's
> not just us, then it may well be worth delaying CKAN 2.2 until this is
> sorted. I wasn't aware that metadata_modified's behaviour had changed
> in CKAN 2.2, assuming it was something to do with our custom
> harvesters. It it would be good to chat about it on the dev team call
> today.
>
> BTW When it occurs I've been fixing it with some quick paster shell commands:
>
> for p in model.Session.query(model.Package).filter_by(metadata_modified=None).all():
> p.metadata_modified = p.metadata_created
> rev = model.repo.new_revision(); rev.author='admin';
> model.repo.commit_and_remove()
>
> David
>
> On 30 January 2014 08:34, Adrià Mercader <adria.mercader at okfn.org> wrote:
>> Hi Hilde,
>>
>> One possible thing that comes to mind is whether if you had harvesters
>> running at the time of doing the upgrade (by running I mean the
>> processes for gather_consumer and fetch_consumer were up). If so, and
>> you triggered a reharvest after the update, the harvesters may have
>> still been running with the 2.1 code, thus not filling the
>> metadata_modified field. Could that be the case?
>>
>> Regardless, it would be great if you could test the commands the David
>> mentioned on the previous email to help us debug this.
>>
>> Many thanks,
>>
>> Adrià
>>
>>
>> On 29 January 2014 22:06, David Raznick <david.raznick at okfn.org> wrote:
>>> Hello
>>>
>>> The only thing I can think of is an error happened with the db upgrade
>>> or that for some reason your db is not upgraded correctly.
>>>
>>> I have had a look over the upgrade script and tested it on some of our
>>> instances and it appears fine, i.e no missing metadata_modified  (you
>>> are correct that this is the issue).
>>>
>>>
>>> Is there anything odd about your db instance or any modifications you
>>> made to it?
>>>
>>> As this is critical bug, if it effects our 2.2 release, could you help
>>> us diagnose this issue?
>>>
>>> Could you run the db commands manually from line 22 to 55 located here:
>>>
>>> https://github.com/ckan/ckan/blob/master/ckan/migration/versions/069_resource_url_and_metadata_modified.py#L22
>>>
>>> The first update should fill in all the metadata_modified data and the
>>> following should just update them with later dates if needed.  Could
>>> you see if at any stage in running these commands make the
>>> metadata_modifed fields became null.
>>>
>>> David
>>>
>>>
>>> On 29 January 2014 18:36, Hildegard Gerlach
>>> <hildegard.gerlach at jrc.ec.europa.eu> wrote:
>>>> Yes, I wrote it in the beginning of the message but forgot to wrote it in the commands below. But yes, I have run it.
>>>>
>>>> Hilde
>>>>
>>>>
>>>> On 29.01.2014, at 19:31, David Raznick <david.raznick at okfn.org> wrote:
>>>>
>>>> Hello
>>>>
>>>> Did you run
>>>>
>>>> paster db upgrade
>>>>
>>>> I would run it after python setup.py develop
>>>>
>>>> David
>>>>
>>>> On 29 January 2014 17:56, Hildegard Gerlach
>>>> <hildegard.gerlach at jrc.ec.europa.eu> wrote:
>>>>> Hello,
>>>>>
>>>>> I started to upgrade a CKAN 2.1.1 version to 2.2. I did the paster db
>>>>> upgrade command.
>>>>> When I want to rebuild the search index I get the following error:
>>>>>
>>>>> File
>>>>> "/usr/local/mtester.ies.jrc.it/pyenv/src/ckan/ckan/lib/dictization/model_dictize.py",
>>>>> line 329, in package_dictize
>>>>>    result_dict['metadata_modified'] = pkg.metadata_modified.isoformat()
>>>>>
>>>>> I cannot see any data anymore on the CKAN Web site as the index is empty but
>>>>> the data is still in the database.
>>>>> It seems that there are 2 new fields in the table package: metadata_modified
>>>>> and creator_user_id
>>>>>
>>>>> The value of metadata_modified  seems to be mostly empty in the database
>>>>> records which, I think, causes the error.
>>>>>
>>>>> I tried as well to clean the database and reimport it (paster db clean,
>>>>> paster db load) and I get the same error again with the import. The database
>>>>> is already with the new fields as after the upgrade.
>>>>>
>>>>> I had done the commands in the following order:
>>>>>
>>>>> git checkout release-v2.2
>>>>> git pull
>>>>> paster db dump -c development.ini  ckantest_dump_28Jan14.sql
>>>>> pip install --upgrade -r requirements.txt
>>>>> python setup.py develop
>>>>> restart Tomcat and httpd
>>>>> paster search-index rebuild -r -c development.ini
>>>>>
>>>>> Luckily this is a test version and I can restart with an empty database and
>>>>> reharvest data. But in production it seems difficult to dare upgrading.
>>>>>
>>>>> Thanks
>>>>>
>>>>> Hilde
>>>>>
>>>>> _______________________________________________
>>>>> ckan-dev mailing list
>>>>> ckan-dev at lists.okfn.org
>>>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>>>>
>>>> _______________________________________________
>>>> ckan-dev mailing list
>>>> ckan-dev at lists.okfn.org
>>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>>>
>>>> _______________________________________________
>>>> ckan-dev mailing list
>>>> ckan-dev at lists.okfn.org
>>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>> _______________________________________________
>>> ckan-dev mailing list
>>> ckan-dev at lists.okfn.org
>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev