[ckan-dev] problems upgrading to CKAN 2.2
Hildegard Gerlach
hildegard.gerlach at jrc.ec.europa.eu
Thu Jan 30 15:37:12 UTC 2014
Thanks for the hint with the load-only, but unfortunately the problem
remains.
After the paster db load-only
the version in my database in table migrate_version is already 70. I
think it should be 68.
The new fields in the package table are already there, even if I use a
dump from November.
Hilde
On 1/30/2014 4:04 PM, Joe Tsoi wrote:
> the dump should just an sql file so you should be able to load it to
> postgres using the postgres commands. But you can try use paster db
> load-only which should not do the upgrade or solr-indexing.
>
> On 30 January 2014 14:45, Hildegard Gerlach
> <hildegard.gerlach at jrc.ec.europa.eu> wrote:
>> I will try with a database dump from November to see if there is a
>> difference. I will let you know.
>>
>> However, I did my backup with the paster command
>> paster db dump
>>
>> so I think I can only do the import in the same way
>>
>> paster db load
>>
>> Thanks
>> Hilde
>>
>>
>> On 1/30/2014 3:28 PM, David Raznick wrote:
>>> Hello
>>>
>>> This stumps me, mainly because I do not know how to reproduce.
>>>
>>> The only things I can think of now that could be the issue.
>>>
>>> * postgres 8.4, David is your production 8.4 or 9.X?
>>> * using the paster db load commands (instead of using pg_dump) may in
>>> some way confuse the issue.
>>> * possibly dumping the db when you have upgraded CKAN to the new version
>>>
>>> Hilde it looks like the db has been upgraded already before you did
>>> the initial dump. Is there a dump you did with the old db before you
>>> upgraded CKAN at all.
>>>
>>> David
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On 30 January 2014 11:12, Hildegard Gerlach
>>> <hildegard.gerlach at jrc.ec.europa.eu> wrote:
>>>> Hi,
>>>>
>>>> again me.
>>>> I created a new DB, then
>>>> paster db clean -c development.ini ckantest_dump_28Jan14.sql (is needed
>>>> to
>>>> clear the postgis tables, otherwise it claims when loading)
>>>> paster db load -c development.ini ckantest_dump_28Jan14.sql
>>>>
>>>> and here at the end I get the following:
>>>>
>>>> Loaded CKAN database: ckantest_dump_28Jan14.sql
>>>> Upgrading DB
>>>> Rebuilding search index
>>>> Traceback (most recent call last):
>>>> File "/usr/local/mtester.ies.jrc.it/pyenv/bin/paster", line 9, in
>>>> <module>
>>>> load_entry_point('PasteScript==1.7.5', 'console_scripts',
>>>> 'paster')()
>>>> File
>>>>
>>>> "/usr/local/mtester.ies.jrc.it/pyenv/lib/python2.6/site-packages/paste/script/command.py",
>>>> line 104, in run
>>>> invoke(command, command_name, options, args[1:])
>>>> File
>>>>
>>>> "/usr/local/mtester.ies.jrc.it/pyenv/lib/python2.6/site-packages/paste/script/command.py",
>>>> line 143, in invoke
>>>> exit_code = runner.run(args)
>>>> File
>>>>
>>>> "/usr/local/mtester.ies.jrc.it/pyenv/lib/python2.6/site-packages/paste/script/command.py",
>>>> line 238, in run
>>>> result = self.command()
>>>> File "/usr/local/mtester.ies.jrc.it/pyenv/src/ckan/ckan/lib/cli.py",
>>>> line
>>>> 174, in command
>>>> self.load()
>>>> File "/usr/local/mtester.ies.jrc.it/pyenv/src/ckan/ckan/lib/cli.py",
>>>> line
>>>> 263, in load
>>>> ckan.lib.search.rebuild()
>>>> File
>>>>
>>>> "/usr/local/mtester.ies.jrc.it/pyenv/src/ckan/ckan/lib/search/__init__.py",
>>>> line 190, in rebuild
>>>> {'id': pkg_id}
>>>> File
>>>> "/usr/local/mtester.ies.jrc.it/pyenv/src/ckan/ckan/logic/__init__.py",
>>>> line
>>>> 419, in wrapped
>>>> result = _action(context, data_dict, **kw)
>>>> File
>>>> "/usr/local/mtester.ies.jrc.it/pyenv/src/ckan/ckan/logic/action/get.py",
>>>> line 830, in package_show
>>>> package_dict = model_dictize.package_dictize(pkg, context)
>>>>
>>>> File
>>>>
>>>> "/usr/local/mtester.ies.jrc.it/pyenv/src/ckan/ckan/lib/dictization/model_dictize.py",
>>>> line 329, in package_dictize
>>>> result_dict['metadata_modified'] = pkg.metadata_modified.isoformat()
>>>> AttributeError: 'NoneType' object has no attribute 'isoformat'
>>>>
>>>> So if I understand correctly, paster db does not only load the data, but
>>>> does as well the upgrade (which partly fails) and then rebuilds the
>>>> search
>>>> index with the same problem. The metadata_modified field is mostly empty
>>>> but
>>>> not always.
>>>>
>>>> I try again to do
>>>> paster db upgrade -c development.ini
>>>>
>>>> but the fields are not filled.
>>>>
>>>> How can I find out if the script you sent me is really executed ?
>>>>
>>>> https://github.com/ckan/ckan/blob/master/ckan/migration/versions/069_resource_url_and_metadata_modified.py#L22
>>>>
>>>> I am using PostgreSQL 8.4 on Linux RH6.
>>>>
>>>> This I can find in the logfile.
>>>>
>>>> 2014-01-30 11:50:42,402 INFO [ckan.model] Database tables dropped
>>>> 2014-01-30 11:51:18,251 INFO [ckan.model] CKAN database version
>>>> upgraded:
>>>> 70 -> 71
>>>> 2014-01-30 11:51:18,334 INFO [ckan.lib.search] Rebuilding search
>>>> index...
>>>> 2014-01-30 11:51:18,365 INFO [ckan.lib.search] Rebuilding the whole
>>>> index...
>>>> 2014-01-30 11:51:18,460 ERROR [ckan.lib.search] Error while indexing
>>>> dataset
>>>> 310ef993-b947-407e-87a8-717bcbeb1911: 'NoneType' object has no attribute
>>>> 'isoformat'
>>>> 2014-01-30 12:00:42,158 INFO [ckan.model] CKAN database version remains
>>>> as:
>>>> 71
>>>> 2014-01-30 12:01:42,697 INFO [ckan.model] CKAN database version remains
>>>> as:
>>>> 71
>>>>
>>>> So the upgrade command does obviously not execute the update package
>>>> command
>>>> in my case.
>>>>
>>>> Hope this helps you.
>>>>
>>>> Hilde
>>>>
>>>>
>>>>
>>>> On 1/30/2014 11:41 AM, Hildegard Gerlach wrote:
>>>>> Dear all,
>>>>>
>>>>> thanks to all for the info.
>>>>>
>>>>> To answer Adria's question: No, I didn't have any harvesters running
>>>>> when
>>>>> I did the upgrade last time.
>>>>> I am not sure if it could have been a problem that I didn't stop the
>>>>> httpd
>>>>> service and I had left the plugins for the extensions in the
>>>>> configuration
>>>>> file.
>>>>>
>>>>> What I did now is:
>>>>> stop httpd, disable plugins for extensions in the configuration file
>>>>> create a clean database, import (paster db load) my latest db dump file
>>>>> from 28 January just before the upgrade.
>>>>> This already adds the new database fields as in version 2.2, i.e. the
>>>>> first 4 commands from here.
>>>>>
>>>>>
>>>>> https://github.com/ckan/ckan/blob/master/ckan/migration/versions/069_resource_url_and_metadata_modified.py#L22
>>>>>
>>>>> Is this supposed to be so ?
>>>>> The metadata_modified is empty at this stage. I then executed all the
>>>>> update commands and the metadata_modified fields are filled.
>>>>>
>>>>> After that I can rebuild the search-index and CKAN works !
>>>>>
>>>>> I propose to do the same thing again but using the paster db upgrade
>>>>> after
>>>>> loading the database to see if it happens again.
>>>>>
>>>>> Hilde
>>>>>
>>>>>
>>>>>
>>>>> On 1/30/2014 9:34 AM, Adrià Mercader wrote:
>>>>>> Hi Hilde,
>>>>>>
>>>>>> One possible thing that comes to mind is whether if you had harvesters
>>>>>> running at the time of doing the upgrade (by running I mean the
>>>>>> processes for gather_consumer and fetch_consumer were up). If so, and
>>>>>> you triggered a reharvest after the update, the harvesters may have
>>>>>> still been running with the 2.1 code, thus not filling the
>>>>>> metadata_modified field. Could that be the case?
>>>>>>
>>>>>> Regardless, it would be great if you could test the commands the David
>>>>>> mentioned on the previous email to help us debug this.
>>>>>>
>>>>>> Many thanks,
>>>>>>
>>>>>> Adrià
>>>>>>
>>>>>>
>>>>>> On 29 January 2014 22:06, David Raznick <david.raznick at okfn.org> wrote:
>>>>>>> Hello
>>>>>>>
>>>>>>> The only thing I can think of is an error happened with the db upgrade
>>>>>>> or that for some reason your db is not upgraded correctly.
>>>>>>>
>>>>>>> I have had a look over the upgrade script and tested it on some of our
>>>>>>> instances and it appears fine, i.e no missing metadata_modified (you
>>>>>>> are correct that this is the issue).
>>>>>>>
>>>>>>>
>>>>>>> Is there anything odd about your db instance or any modifications you
>>>>>>> made to it?
>>>>>>>
>>>>>>> As this is critical bug, if it effects our 2.2 release, could you help
>>>>>>> us diagnose this issue?
>>>>>>>
>>>>>>> Could you run the db commands manually from line 22 to 55 located
>>>>>>> here:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> https://github.com/ckan/ckan/blob/master/ckan/migration/versions/069_resource_url_and_metadata_modified.py#L22
>>>>>>>
>>>>>>> The first update should fill in all the metadata_modified data and the
>>>>>>> following should just update them with later dates if needed. Could
>>>>>>> you see if at any stage in running these commands make the
>>>>>>> metadata_modifed fields became null.
>>>>>>>
>>>>>>> David
>>>>>>>
>>>>>>>
>>>>>>> On 29 January 2014 18:36, Hildegard Gerlach
>>>>>>> <hildegard.gerlach at jrc.ec.europa.eu> wrote:
>>>>>>>> Yes, I wrote it in the beginning of the message but forgot to wrote
>>>>>>>> it
>>>>>>>> in the commands below. But yes, I have run it.
>>>>>>>>
>>>>>>>> Hilde
>>>>>>>>
>>>>>>>>
>>>>>>>> On 29.01.2014, at 19:31, David Raznick <david.raznick at okfn.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hello
>>>>>>>>
>>>>>>>> Did you run
>>>>>>>>
>>>>>>>> paster db upgrade
>>>>>>>>
>>>>>>>> I would run it after python setup.py develop
>>>>>>>>
>>>>>>>> David
>>>>>>>>
>>>>>>>> On 29 January 2014 17:56, Hildegard Gerlach
>>>>>>>> <hildegard.gerlach at jrc.ec.europa.eu> wrote:
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> I started to upgrade a CKAN 2.1.1 version to 2.2. I did the paster
>>>>>>>>> db
>>>>>>>>> upgrade command.
>>>>>>>>> When I want to rebuild the search index I get the following error:
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> "/usr/local/mtester.ies.jrc.it/pyenv/src/ckan/ckan/lib/dictization/model_dictize.py",
>>>>>>>>> line 329, in package_dictize
>>>>>>>>> result_dict['metadata_modified'] =
>>>>>>>>> pkg.metadata_modified.isoformat()
>>>>>>>>>
>>>>>>>>> I cannot see any data anymore on the CKAN Web site as the index is
>>>>>>>>> empty but
>>>>>>>>> the data is still in the database.
>>>>>>>>> It seems that there are 2 new fields in the table package:
>>>>>>>>> metadata_modified
>>>>>>>>> and creator_user_id
>>>>>>>>>
>>>>>>>>> The value of metadata_modified seems to be mostly empty in the
>>>>>>>>> database
>>>>>>>>> records which, I think, causes the error.
>>>>>>>>>
>>>>>>>>> I tried as well to clean the database and reimport it (paster db
>>>>>>>>> clean,
>>>>>>>>> paster db load) and I get the same error again with the import. The
>>>>>>>>> database
>>>>>>>>> is already with the new fields as after the upgrade.
>>>>>>>>>
>>>>>>>>> I had done the commands in the following order:
>>>>>>>>>
>>>>>>>>> git checkout release-v2.2
>>>>>>>>> git pull
>>>>>>>>> paster db dump -c development.ini ckantest_dump_28Jan14.sql
>>>>>>>>> pip install --upgrade -r requirements.txt
>>>>>>>>> python setup.py develop
>>>>>>>>> restart Tomcat and httpd
>>>>>>>>> paster search-index rebuild -r -c development.ini
>>>>>>>>>
>>>>>>>>> Luckily this is a test version and I can restart with an empty
>>>>>>>>> database and
>>>>>>>>> reharvest data. But in production it seems difficult to dare
>>>>>>>>> upgrading.
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>>
>>>>>>>>> Hilde
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> ckan-dev mailing list
>>>>>>>>> ckan-dev at lists.okfn.org
>>>>>>>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>>>>>>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>>>>>>>>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
More information about the ckan-dev
mailing list