[ckan-dev] problems upgrading to CKAN 2.2

[David Read]

Glad to hear that Hildegard cleared up her issue with
metadata_modified and the migration.

I think I've solved our issue too with harvesting. Although I thought
I'd restarted our harvesters since upgrading, on Adria's suggestion I
checked and it turns out I hadn't. I've not seen these errors since,
so I think that has sorted it. Many thanks for the help,

David


On 30 January 2014 10:06, David Read <david.read at hackneyworkshop.com> wrote:
> We're also suffering this same exception on DGU since upgrading to
> CKAN 2.2. We see metadata_modified bring set to null when creating
> some new datasets.
>
> I've not reproduced it on my own machine or a test machine - just on
> our live machine, which is harder to play with. We've restarted
> harvesters and queues etc but still no joy. I've not managed to get a
> handle on what exactly causes it, but will have some more time to look
> at it next week.
>
> I believe it happens occasionally, with harvested datasets. Since it's
> not just us, then it may well be worth delaying CKAN 2.2 until this is
> sorted. I wasn't aware that metadata_modified's behaviour had changed
> in CKAN 2.2, assuming it was something to do with our custom
> harvesters. It it would be good to chat about it on the dev team call
> today.
>
> BTW When it occurs I've been fixing it with some quick paster shell commands:
>
> for p in model.Session.query(model.Package).filter_by(metadata_modified=None).all():
> p.metadata_modified = p.metadata_created
> rev = model.repo.new_revision(); rev.author='admin';
> model.repo.commit_and_remove()
>
> David
>
> On 30 January 2014 08:34, Adrià Mercader <adria.mercader at okfn.org> wrote:
>> Hi Hilde,
>>
>> One possible thing that comes to mind is whether if you had harvesters
>> running at the time of doing the upgrade (by running I mean the
>> processes for gather_consumer and fetch_consumer were up). If so, and
>> you triggered a reharvest after the update, the harvesters may have
>> still been running with the 2.1 code, thus not filling the
>> metadata_modified field. Could that be the case?
>>
>> Regardless, it would be great if you could test the commands the David
>> mentioned on the previous email to help us debug this.
>>
>> Many thanks,
>>
>> Adrià
>>
>>
>> On 29 January 2014 22:06, David Raznick <david.raznick at okfn.org> wrote:
>>> Hello
>>>
>>> The only thing I can think of is an error happened with the db upgrade
>>> or that for some reason your db is not upgraded correctly.
>>>
>>> I have had a look over the upgrade script and tested it on some of our
>>> instances and it appears fine, i.e no missing metadata_modified  (you
>>> are correct that this is the issue).
>>>
>>>
>>> Is there anything odd about your db instance or any modifications you
>>> made to it?
>>>
>>> As this is critical bug, if it effects our 2.2 release, could you help
>>> us diagnose this issue?
>>>
>>> Could you run the db commands manually from line 22 to 55 located here:
>>>
>>> https://github.com/ckan/ckan/blob/master/ckan/migration/versions/069_resource_url_and_metadata_modified.py#L22
>>>
>>> The first update should fill in all the metadata_modified data and the
>>> following should just update them with later dates if needed.  Could
>>> you see if at any stage in running these commands make the
>>> metadata_modifed fields became null.
>>>
>>> David
>>>
>>>
>>> On 29 January 2014 18:36, Hildegard Gerlach
>>> <hildegard.gerlach at jrc.ec.europa.eu> wrote:
>>>> Yes, I wrote it in the beginning of the message but forgot to wrote it in the commands below. But yes, I have run it.
>>>>
>>>> Hilde
>>>>
>>>>
>>>> On 29.01.2014, at 19:31, David Raznick <david.raznick at okfn.org> wrote:
>>>>
>>>> Hello
>>>>
>>>> Did you run
>>>>
>>>> paster db upgrade
>>>>
>>>> I would run it after python setup.py develop
>>>>
>>>> David
>>>>
>>>> On 29 January 2014 17:56, Hildegard Gerlach
>>>> <hildegard.gerlach at jrc.ec.europa.eu> wrote:
>>>>> Hello,
>>>>>
>>>>> I started to upgrade a CKAN 2.1.1 version to 2.2. I did the paster db
>>>>> upgrade command.
>>>>> When I want to rebuild the search index I get the following error:
>>>>>
>>>>> File
>>>>> "/usr/local/mtester.ies.jrc.it/pyenv/src/ckan/ckan/lib/dictization/model_dictize.py",
>>>>> line 329, in package_dictize
>>>>>    result_dict['metadata_modified'] = pkg.metadata_modified.isoformat()
>>>>>
>>>>> I cannot see any data anymore on the CKAN Web site as the index is empty but
>>>>> the data is still in the database.
>>>>> It seems that there are 2 new fields in the table package: metadata_modified
>>>>> and creator_user_id
>>>>>
>>>>> The value of metadata_modified  seems to be mostly empty in the database
>>>>> records which, I think, causes the error.
>>>>>
>>>>> I tried as well to clean the database and reimport it (paster db clean,
>>>>> paster db load) and I get the same error again with the import. The database
>>>>> is already with the new fields as after the upgrade.
>>>>>
>>>>> I had done the commands in the following order:
>>>>>
>>>>> git checkout release-v2.2
>>>>> git pull
>>>>> paster db dump -c development.ini  ckantest_dump_28Jan14.sql
>>>>> pip install --upgrade -r requirements.txt
>>>>> python setup.py develop
>>>>> restart Tomcat and httpd
>>>>> paster search-index rebuild -r -c development.ini
>>>>>
>>>>> Luckily this is a test version and I can restart with an empty database and
>>>>> reharvest data. But in production it seems difficult to dare upgrading.
>>>>>
>>>>> Thanks
>>>>>
>>>>> Hilde
>>>>>
>>>>> _______________________________________________
>>>>> ckan-dev mailing list
>>>>> ckan-dev at lists.okfn.org
>>>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>>>>
>>>> _______________________________________________
>>>> ckan-dev mailing list
>>>> ckan-dev at lists.okfn.org
>>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>>>
>>>> _______________________________________________
>>>> ckan-dev mailing list
>>>> ckan-dev at lists.okfn.org
>>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>> _______________________________________________
>>> ckan-dev mailing list
>>> ckan-dev at lists.okfn.org
>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev