[ckan-dev] problems upgrading to CKAN 2.2

Hildegard Gerlach hildegard.gerlach at jrc.ec.europa.eu
Thu Jan 30 14:45:15 UTC 2014 

I will try with a database dump from November to see if there is a 
difference. I will let you know.

However, I did my backup with the paster command
paster db dump

so I think I can only do the import in the same way

paster db load

Thanks
Hilde

On 1/30/2014 3:28 PM, David Raznick wrote:
> Hello
>
> This stumps me, mainly because I do not know how to reproduce.
>
> The only things I can think of now that could be the issue.
>
> * postgres 8.4,  David is your production 8.4 or 9.X?
> * using the paster db load commands (instead of using pg_dump) may in
> some way confuse the issue.
> * possibly dumping the db when you have upgraded CKAN to the new version
>
> Hilde it looks like the db has been upgraded already before you did
> the initial dump.  Is there a dump you did with the old db before you
> upgraded CKAN at all.
>
> David
>
>
>
>
>
>
>
> On 30 January 2014 11:12, Hildegard Gerlach
> <hildegard.gerlach at jrc.ec.europa.eu> wrote:
>> Hi,
>>
>> again me.
>> I created a new DB, then
>> paster db clean -c development.ini  ckantest_dump_28Jan14.sql (is needed to
>> clear the postgis tables, otherwise it claims when loading)
>> paster db load -c development.ini  ckantest_dump_28Jan14.sql
>>
>> and here at the end I get the following:
>>
>> Loaded CKAN database: ckantest_dump_28Jan14.sql
>> Upgrading DB
>> Rebuilding search index
>> Traceback (most recent call last):
>>    File "/usr/local/mtester.ies.jrc.it/pyenv/bin/paster", line 9, in <module>
>>      load_entry_point('PasteScript==1.7.5', 'console_scripts', 'paster')()
>>    File
>> "/usr/local/mtester.ies.jrc.it/pyenv/lib/python2.6/site-packages/paste/script/command.py",
>> line 104, in run
>>      invoke(command, command_name, options, args[1:])
>>    File
>> "/usr/local/mtester.ies.jrc.it/pyenv/lib/python2.6/site-packages/paste/script/command.py",
>> line 143, in invoke
>>      exit_code = runner.run(args)
>>    File
>> "/usr/local/mtester.ies.jrc.it/pyenv/lib/python2.6/site-packages/paste/script/command.py",
>> line 238, in run
>>      result = self.command()
>>    File "/usr/local/mtester.ies.jrc.it/pyenv/src/ckan/ckan/lib/cli.py", line
>> 174, in command
>>      self.load()
>>    File "/usr/local/mtester.ies.jrc.it/pyenv/src/ckan/ckan/lib/cli.py", line
>> 263, in load
>>      ckan.lib.search.rebuild()
>>    File
>> "/usr/local/mtester.ies.jrc.it/pyenv/src/ckan/ckan/lib/search/__init__.py",
>> line 190, in rebuild
>>      {'id': pkg_id}
>>    File
>> "/usr/local/mtester.ies.jrc.it/pyenv/src/ckan/ckan/logic/__init__.py", line
>> 419, in wrapped
>>      result = _action(context, data_dict, **kw)
>>    File
>> "/usr/local/mtester.ies.jrc.it/pyenv/src/ckan/ckan/logic/action/get.py",
>> line 830, in package_show
>>      package_dict = model_dictize.package_dictize(pkg, context)
>>
>>    File
>> "/usr/local/mtester.ies.jrc.it/pyenv/src/ckan/ckan/lib/dictization/model_dictize.py",
>> line 329, in package_dictize
>>      result_dict['metadata_modified'] = pkg.metadata_modified.isoformat()
>> AttributeError: 'NoneType' object has no attribute 'isoformat'
>>
>> So if I understand correctly, paster db does not only load the data, but
>> does as well the upgrade (which partly fails) and then rebuilds the search
>> index with the same problem. The metadata_modified field is mostly empty but
>> not always.
>>
>> I try again to do
>> paster db upgrade -c development.ini
>>
>> but the fields are not filled.
>>
>> How can I find out if the script you sent me is really executed ?
>> https://github.com/ckan/ckan/blob/master/ckan/migration/versions/069_resource_url_and_metadata_modified.py#L22
>>
>> I am using  PostgreSQL 8.4 on Linux RH6.
>>
>> This I can find in the logfile.
>>
>> 2014-01-30 11:50:42,402 INFO  [ckan.model] Database tables dropped
>> 2014-01-30 11:51:18,251 INFO  [ckan.model] CKAN database version upgraded:
>> 70 -> 71
>> 2014-01-30 11:51:18,334 INFO  [ckan.lib.search] Rebuilding search index...
>> 2014-01-30 11:51:18,365 INFO  [ckan.lib.search] Rebuilding the whole
>> index...
>> 2014-01-30 11:51:18,460 ERROR [ckan.lib.search] Error while indexing dataset
>> 310ef993-b947-407e-87a8-717bcbeb1911: 'NoneType' object has no attribute
>> 'isoformat'
>> 2014-01-30 12:00:42,158 INFO  [ckan.model] CKAN database version remains as:
>> 71
>> 2014-01-30 12:01:42,697 INFO  [ckan.model] CKAN database version remains as:
>> 71
>>
>> So the upgrade command does obviously not execute the update package command
>> in my case.
>>
>> Hope this helps you.
>>
>> Hilde
>>
>>
>>
>> On 1/30/2014 11:41 AM, Hildegard Gerlach wrote:
>>> Dear all,
>>>
>>> thanks to all for the info.
>>>
>>> To answer Adria's question: No, I didn't have any harvesters running when
>>> I did the upgrade last time.
>>> I am not sure if it could have been a problem that I didn't stop the httpd
>>> service and I had left the plugins for the extensions in the configuration
>>> file.
>>>
>>> What I did now is:
>>> stop httpd, disable plugins for extensions in the configuration file
>>> create a clean database, import (paster db load) my latest db dump file
>>> from 28 January just before the upgrade.
>>> This already adds the new database fields as in version 2.2, i.e. the
>>> first 4 commands from here.
>>>
>>> https://github.com/ckan/ckan/blob/master/ckan/migration/versions/069_resource_url_and_metadata_modified.py#L22
>>>
>>> Is this supposed to be so ?
>>> The metadata_modified is empty at this stage. I then executed all the
>>> update commands and the metadata_modified fields are filled.
>>>
>>> After that I can rebuild the search-index and CKAN works !
>>>
>>> I propose to do the same thing again but using the paster db upgrade after
>>> loading the database to see if it happens again.
>>>
>>> Hilde
>>>
>>>
>>>
>>> On 1/30/2014 9:34 AM, Adrià Mercader wrote:
>>>> Hi Hilde,
>>>>
>>>> One possible thing that comes to mind is whether if you had harvesters
>>>> running at the time of doing the upgrade (by running I mean the
>>>> processes for gather_consumer and fetch_consumer were up). If so, and
>>>> you triggered a reharvest after the update, the harvesters may have
>>>> still been running with the 2.1 code, thus not filling the
>>>> metadata_modified field. Could that be the case?
>>>>
>>>> Regardless, it would be great if you could test the commands the David
>>>> mentioned on the previous email to help us debug this.
>>>>
>>>> Many thanks,
>>>>
>>>> Adrià
>>>>
>>>>
>>>> On 29 January 2014 22:06, David Raznick <david.raznick at okfn.org> wrote:
>>>>> Hello
>>>>>
>>>>> The only thing I can think of is an error happened with the db upgrade
>>>>> or that for some reason your db is not upgraded correctly.
>>>>>
>>>>> I have had a look over the upgrade script and tested it on some of our
>>>>> instances and it appears fine, i.e no missing metadata_modified  (you
>>>>> are correct that this is the issue).
>>>>>
>>>>>
>>>>> Is there anything odd about your db instance or any modifications you
>>>>> made to it?
>>>>>
>>>>> As this is critical bug, if it effects our 2.2 release, could you help
>>>>> us diagnose this issue?
>>>>>
>>>>> Could you run the db commands manually from line 22 to 55 located here:
>>>>>
>>>>>
>>>>> https://github.com/ckan/ckan/blob/master/ckan/migration/versions/069_resource_url_and_metadata_modified.py#L22
>>>>>
>>>>> The first update should fill in all the metadata_modified data and the
>>>>> following should just update them with later dates if needed. Could
>>>>> you see if at any stage in running these commands make the
>>>>> metadata_modifed fields became null.
>>>>>
>>>>> David
>>>>>
>>>>>
>>>>> On 29 January 2014 18:36, Hildegard Gerlach
>>>>> <hildegard.gerlach at jrc.ec.europa.eu> wrote:
>>>>>> Yes, I wrote it in the beginning of the message but forgot to wrote it
>>>>>> in the commands below. But yes, I have run it.
>>>>>>
>>>>>> Hilde
>>>>>>
>>>>>>
>>>>>> On 29.01.2014, at 19:31, David Raznick <david.raznick at okfn.org> wrote:
>>>>>>
>>>>>> Hello
>>>>>>
>>>>>> Did you run
>>>>>>
>>>>>> paster db upgrade
>>>>>>
>>>>>> I would run it after python setup.py develop
>>>>>>
>>>>>> David
>>>>>>
>>>>>> On 29 January 2014 17:56, Hildegard Gerlach
>>>>>> <hildegard.gerlach at jrc.ec.europa.eu> wrote:
>>>>>>> Hello,
>>>>>>>
>>>>>>> I started to upgrade a CKAN 2.1.1 version to 2.2. I did the paster db
>>>>>>> upgrade command.
>>>>>>> When I want to rebuild the search index I get the following error:
>>>>>>>
>>>>>>> File
>>>>>>>
>>>>>>> "/usr/local/mtester.ies.jrc.it/pyenv/src/ckan/ckan/lib/dictization/model_dictize.py",
>>>>>>> line 329, in package_dictize
>>>>>>>      result_dict['metadata_modified'] =
>>>>>>> pkg.metadata_modified.isoformat()
>>>>>>>
>>>>>>> I cannot see any data anymore on the CKAN Web site as the index is
>>>>>>> empty but
>>>>>>> the data is still in the database.
>>>>>>> It seems that there are 2 new fields in the table package:
>>>>>>> metadata_modified
>>>>>>> and creator_user_id
>>>>>>>
>>>>>>> The value of metadata_modified  seems to be mostly empty in the
>>>>>>> database
>>>>>>> records which, I think, causes the error.
>>>>>>>
>>>>>>> I tried as well to clean the database and reimport it (paster db
>>>>>>> clean,
>>>>>>> paster db load) and I get the same error again with the import. The
>>>>>>> database
>>>>>>> is already with the new fields as after the upgrade.
>>>>>>>
>>>>>>> I had done the commands in the following order:
>>>>>>>
>>>>>>> git checkout release-v2.2
>>>>>>> git pull
>>>>>>> paster db dump -c development.ini ckantest_dump_28Jan14.sql
>>>>>>> pip install --upgrade -r requirements.txt
>>>>>>> python setup.py develop
>>>>>>> restart Tomcat and httpd
>>>>>>> paster search-index rebuild -r -c development.ini
>>>>>>>
>>>>>>> Luckily this is a test version and I can restart with an empty
>>>>>>> database and
>>>>>>> reharvest data. But in production it seems difficult to dare
>>>>>>> upgrading.
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> Hilde
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> ckan-dev mailing list
>>>>>>> ckan-dev at lists.okfn.org
>>>>>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>>>>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>>>>>>
>>>
>>> _______________________________________________
>>> ckan-dev mailing list
>>> ckan-dev at lists.okfn.org
>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev

More information about the ckan-dev mailing list