[ckan-dev] problems upgrading to CKAN 2.2
Hildegard Gerlach
hildegard.gerlach at jrc.ec.europa.eu
Thu Jan 30 15:08:36 UTC 2014
No, the result is the same. In fact, I had done the dump before
upgrading the DB.
Anyway I tried with an empty database and loaded a dump from November.
It is the paster db load command itself which does the upgrade as well
as the rebuild of the search index
paster db load -c development.ini ckantest_dump_25Nov.sql
...
Loaded CKAN database: ckantest_dump_25Nov.sql
Upgrading DB
Rebuilding search index
Traceback (most recent call last):
File "/usr/local/mtester.ies.jrc.it/pyenv/bin/paster", line 9, in
<module>
load_entry_point('PasteScript==1.7.5', 'console_scripts', 'paster')()
File
"/usr/local/mtester.ies.jrc.it/pyenv/lib/python2.6/site-packages/paste/script/command.py",
line 104, in run
invoke(command, command_name, options, args[1:])
File
"/usr/local/mtester.ies.jrc.it/pyenv/lib/python2.6/site-packages/paste/script/command.py",
line 143, in invoke
exit_code = runner.run(args)
File
"/usr/local/mtester.ies.jrc.it/pyenv/lib/python2.6/site-packages/paste/script/command.py",
line 238, in run
result = self.command()
File "/usr/local/mtester.ies.jrc.it/pyenv/src/ckan/ckan/lib/cli.py",
line 174, in command
self.load()
File "/usr/local/mtester.ies.jrc.it/pyenv/src/ckan/ckan/lib/cli.py",
line 263, in load
ckan.lib.search.rebuild()
File
"/usr/local/mtester.ies.jrc.it/pyenv/src/ckan/ckan/lib/search/__init__.py",
line 190, in rebuild
{'id': pkg_id}
File
"/usr/local/mtester.ies.jrc.it/pyenv/src/ckan/ckan/logic/__init__.py",
line 419, in wrapped
result = _action(context, data_dict, **kw)
File
"/usr/local/mtester.ies.jrc.it/pyenv/src/ckan/ckan/logic/action/get.py",
line 830, in package_show
package_dict = model_dictize.package_dictize(pkg, context)
File
"/usr/local/mtester.ies.jrc.it/pyenv/src/ckan/ckan/lib/dictization/model_dictize.py",
line 329, in package_dictize
result_dict['metadata_modified'] = pkg.metadata_modified.isoformat()
AttributeError: 'NoneType' object has no attribute 'isoformat'
Hilde
On 1/30/2014 3:45 PM, Hildegard Gerlach wrote:
> I will try with a database dump from November to see if there is a
> difference. I will let you know.
>
> However, I did my backup with the paster command
> paster db dump
>
> so I think I can only do the import in the same way
>
> paster db load
>
> Thanks
> Hilde
>
> On 1/30/2014 3:28 PM, David Raznick wrote:
>> Hello
>>
>> This stumps me, mainly because I do not know how to reproduce.
>>
>> The only things I can think of now that could be the issue.
>>
>> * postgres 8.4, David is your production 8.4 or 9.X?
>> * using the paster db load commands (instead of using pg_dump) may in
>> some way confuse the issue.
>> * possibly dumping the db when you have upgraded CKAN to the new version
>>
>> Hilde it looks like the db has been upgraded already before you did
>> the initial dump. Is there a dump you did with the old db before you
>> upgraded CKAN at all.
>>
>> David
>>
>>
>>
>>
>>
>>
>>
>> On 30 January 2014 11:12, Hildegard Gerlach
>> <hildegard.gerlach at jrc.ec.europa.eu> wrote:
>>> Hi,
>>>
>>> again me.
>>> I created a new DB, then
>>> paster db clean -c development.ini ckantest_dump_28Jan14.sql (is
>>> needed to
>>> clear the postgis tables, otherwise it claims when loading)
>>> paster db load -c development.ini ckantest_dump_28Jan14.sql
>>>
>>> and here at the end I get the following:
>>>
>>> Loaded CKAN database: ckantest_dump_28Jan14.sql
>>> Upgrading DB
>>> Rebuilding search index
>>> Traceback (most recent call last):
>>> File "/usr/local/mtester.ies.jrc.it/pyenv/bin/paster", line 9, in
>>> <module>
>>> load_entry_point('PasteScript==1.7.5', 'console_scripts',
>>> 'paster')()
>>> File
>>> "/usr/local/mtester.ies.jrc.it/pyenv/lib/python2.6/site-packages/paste/script/command.py",
>>>
>>> line 104, in run
>>> invoke(command, command_name, options, args[1:])
>>> File
>>> "/usr/local/mtester.ies.jrc.it/pyenv/lib/python2.6/site-packages/paste/script/command.py",
>>>
>>> line 143, in invoke
>>> exit_code = runner.run(args)
>>> File
>>> "/usr/local/mtester.ies.jrc.it/pyenv/lib/python2.6/site-packages/paste/script/command.py",
>>>
>>> line 238, in run
>>> result = self.command()
>>> File
>>> "/usr/local/mtester.ies.jrc.it/pyenv/src/ckan/ckan/lib/cli.py", line
>>> 174, in command
>>> self.load()
>>> File
>>> "/usr/local/mtester.ies.jrc.it/pyenv/src/ckan/ckan/lib/cli.py", line
>>> 263, in load
>>> ckan.lib.search.rebuild()
>>> File
>>> "/usr/local/mtester.ies.jrc.it/pyenv/src/ckan/ckan/lib/search/__init__.py",
>>>
>>> line 190, in rebuild
>>> {'id': pkg_id}
>>> File
>>> "/usr/local/mtester.ies.jrc.it/pyenv/src/ckan/ckan/logic/__init__.py",
>>> line
>>> 419, in wrapped
>>> result = _action(context, data_dict, **kw)
>>> File
>>> "/usr/local/mtester.ies.jrc.it/pyenv/src/ckan/ckan/logic/action/get.py",
>>>
>>> line 830, in package_show
>>> package_dict = model_dictize.package_dictize(pkg, context)
>>>
>>> File
>>> "/usr/local/mtester.ies.jrc.it/pyenv/src/ckan/ckan/lib/dictization/model_dictize.py",
>>>
>>> line 329, in package_dictize
>>> result_dict['metadata_modified'] =
>>> pkg.metadata_modified.isoformat()
>>> AttributeError: 'NoneType' object has no attribute 'isoformat'
>>>
>>> So if I understand correctly, paster db does not only load the data,
>>> but
>>> does as well the upgrade (which partly fails) and then rebuilds the
>>> search
>>> index with the same problem. The metadata_modified field is mostly
>>> empty but
>>> not always.
>>>
>>> I try again to do
>>> paster db upgrade -c development.ini
>>>
>>> but the fields are not filled.
>>>
>>> How can I find out if the script you sent me is really executed ?
>>> https://github.com/ckan/ckan/blob/master/ckan/migration/versions/069_resource_url_and_metadata_modified.py#L22
>>>
>>>
>>> I am using PostgreSQL 8.4 on Linux RH6.
>>>
>>> This I can find in the logfile.
>>>
>>> 2014-01-30 11:50:42,402 INFO [ckan.model] Database tables dropped
>>> 2014-01-30 11:51:18,251 INFO [ckan.model] CKAN database version
>>> upgraded:
>>> 70 -> 71
>>> 2014-01-30 11:51:18,334 INFO [ckan.lib.search] Rebuilding search
>>> index...
>>> 2014-01-30 11:51:18,365 INFO [ckan.lib.search] Rebuilding the whole
>>> index...
>>> 2014-01-30 11:51:18,460 ERROR [ckan.lib.search] Error while indexing
>>> dataset
>>> 310ef993-b947-407e-87a8-717bcbeb1911: 'NoneType' object has no
>>> attribute
>>> 'isoformat'
>>> 2014-01-30 12:00:42,158 INFO [ckan.model] CKAN database version
>>> remains as:
>>> 71
>>> 2014-01-30 12:01:42,697 INFO [ckan.model] CKAN database version
>>> remains as:
>>> 71
>>>
>>> So the upgrade command does obviously not execute the update package
>>> command
>>> in my case.
>>>
>>> Hope this helps you.
>>>
>>> Hilde
>>>
>>>
>>>
>>> On 1/30/2014 11:41 AM, Hildegard Gerlach wrote:
>>>> Dear all,
>>>>
>>>> thanks to all for the info.
>>>>
>>>> To answer Adria's question: No, I didn't have any harvesters
>>>> running when
>>>> I did the upgrade last time.
>>>> I am not sure if it could have been a problem that I didn't stop
>>>> the httpd
>>>> service and I had left the plugins for the extensions in the
>>>> configuration
>>>> file.
>>>>
>>>> What I did now is:
>>>> stop httpd, disable plugins for extensions in the configuration file
>>>> create a clean database, import (paster db load) my latest db dump
>>>> file
>>>> from 28 January just before the upgrade.
>>>> This already adds the new database fields as in version 2.2, i.e. the
>>>> first 4 commands from here.
>>>>
>>>> https://github.com/ckan/ckan/blob/master/ckan/migration/versions/069_resource_url_and_metadata_modified.py#L22
>>>>
>>>>
>>>> Is this supposed to be so ?
>>>> The metadata_modified is empty at this stage. I then executed all the
>>>> update commands and the metadata_modified fields are filled.
>>>>
>>>> After that I can rebuild the search-index and CKAN works !
>>>>
>>>> I propose to do the same thing again but using the paster db
>>>> upgrade after
>>>> loading the database to see if it happens again.
>>>>
>>>> Hilde
>>>>
>>>>
>>>>
>>>> On 1/30/2014 9:34 AM, Adrià Mercader wrote:
>>>>> Hi Hilde,
>>>>>
>>>>> One possible thing that comes to mind is whether if you had
>>>>> harvesters
>>>>> running at the time of doing the upgrade (by running I mean the
>>>>> processes for gather_consumer and fetch_consumer were up). If so, and
>>>>> you triggered a reharvest after the update, the harvesters may have
>>>>> still been running with the 2.1 code, thus not filling the
>>>>> metadata_modified field. Could that be the case?
>>>>>
>>>>> Regardless, it would be great if you could test the commands the
>>>>> David
>>>>> mentioned on the previous email to help us debug this.
>>>>>
>>>>> Many thanks,
>>>>>
>>>>> Adrià
>>>>>
>>>>>
>>>>> On 29 January 2014 22:06, David Raznick <david.raznick at okfn.org>
>>>>> wrote:
>>>>>> Hello
>>>>>>
>>>>>> The only thing I can think of is an error happened with the db
>>>>>> upgrade
>>>>>> or that for some reason your db is not upgraded correctly.
>>>>>>
>>>>>> I have had a look over the upgrade script and tested it on some
>>>>>> of our
>>>>>> instances and it appears fine, i.e no missing metadata_modified
>>>>>> (you
>>>>>> are correct that this is the issue).
>>>>>>
>>>>>>
>>>>>> Is there anything odd about your db instance or any modifications
>>>>>> you
>>>>>> made to it?
>>>>>>
>>>>>> As this is critical bug, if it effects our 2.2 release, could you
>>>>>> help
>>>>>> us diagnose this issue?
>>>>>>
>>>>>> Could you run the db commands manually from line 22 to 55 located
>>>>>> here:
>>>>>>
>>>>>>
>>>>>> https://github.com/ckan/ckan/blob/master/ckan/migration/versions/069_resource_url_and_metadata_modified.py#L22
>>>>>>
>>>>>>
>>>>>> The first update should fill in all the metadata_modified data
>>>>>> and the
>>>>>> following should just update them with later dates if needed. Could
>>>>>> you see if at any stage in running these commands make the
>>>>>> metadata_modifed fields became null.
>>>>>>
>>>>>> David
>>>>>>
>>>>>>
>>>>>> On 29 January 2014 18:36, Hildegard Gerlach
>>>>>> <hildegard.gerlach at jrc.ec.europa.eu> wrote:
>>>>>>> Yes, I wrote it in the beginning of the message but forgot to
>>>>>>> wrote it
>>>>>>> in the commands below. But yes, I have run it.
>>>>>>>
>>>>>>> Hilde
>>>>>>>
>>>>>>>
>>>>>>> On 29.01.2014, at 19:31, David Raznick <david.raznick at okfn.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Hello
>>>>>>>
>>>>>>> Did you run
>>>>>>>
>>>>>>> paster db upgrade
>>>>>>>
>>>>>>> I would run it after python setup.py develop
>>>>>>>
>>>>>>> David
>>>>>>>
>>>>>>> On 29 January 2014 17:56, Hildegard Gerlach
>>>>>>> <hildegard.gerlach at jrc.ec.europa.eu> wrote:
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> I started to upgrade a CKAN 2.1.1 version to 2.2. I did the
>>>>>>>> paster db
>>>>>>>> upgrade command.
>>>>>>>> When I want to rebuild the search index I get the following error:
>>>>>>>>
>>>>>>>> File
>>>>>>>>
>>>>>>>> "/usr/local/mtester.ies.jrc.it/pyenv/src/ckan/ckan/lib/dictization/model_dictize.py",
>>>>>>>>
>>>>>>>> line 329, in package_dictize
>>>>>>>> result_dict['metadata_modified'] =
>>>>>>>> pkg.metadata_modified.isoformat()
>>>>>>>>
>>>>>>>> I cannot see any data anymore on the CKAN Web site as the index is
>>>>>>>> empty but
>>>>>>>> the data is still in the database.
>>>>>>>> It seems that there are 2 new fields in the table package:
>>>>>>>> metadata_modified
>>>>>>>> and creator_user_id
>>>>>>>>
>>>>>>>> The value of metadata_modified seems to be mostly empty in the
>>>>>>>> database
>>>>>>>> records which, I think, causes the error.
>>>>>>>>
>>>>>>>> I tried as well to clean the database and reimport it (paster db
>>>>>>>> clean,
>>>>>>>> paster db load) and I get the same error again with the import.
>>>>>>>> The
>>>>>>>> database
>>>>>>>> is already with the new fields as after the upgrade.
>>>>>>>>
>>>>>>>> I had done the commands in the following order:
>>>>>>>>
>>>>>>>> git checkout release-v2.2
>>>>>>>> git pull
>>>>>>>> paster db dump -c development.ini ckantest_dump_28Jan14.sql
>>>>>>>> pip install --upgrade -r requirements.txt
>>>>>>>> python setup.py develop
>>>>>>>> restart Tomcat and httpd
>>>>>>>> paster search-index rebuild -r -c development.ini
>>>>>>>>
>>>>>>>> Luckily this is a test version and I can restart with an empty
>>>>>>>> database and
>>>>>>>> reharvest data. But in production it seems difficult to dare
>>>>>>>> upgrading.
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>>
>>>>>>>> Hilde
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> ckan-dev mailing list
>>>>>>>> ckan-dev at lists.okfn.org
>>>>>>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>>>>>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>>>>>>>
>>>>
>>>> _______________________________________________
>>>> ckan-dev mailing list
>>>> ckan-dev at lists.okfn.org
>>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>>
More information about the ckan-dev
mailing list