[ckan-dev] problems upgrading to CKAN 2.2
Adrià Mercader
adria.mercader at okfn.org
Fri Jan 31 11:15:12 UTC 2014
Hi Hilde,
That makes perfect sense. As the new table fields were already there,
the transaction in step n. 69 was failing completely, so the field was
not filled.
Many thanks for reporting this, I'll add a notice on the changelog to
warn users of this. Note that if you want to use your latest dump, you
can run the SQL commands from line 22 to 55 of the migration file [1].
That will fill the metadata_modified field.
Changes on the extensions db tables (like the harvesting ones) are
managed by the extension themselves, independently of the CKAN core
releases (if you are curious about how it works, here's how the
harvest extension does it [2])
Adrià
[1] https://github.com/ckan/ckan/blob/master/ckan/migration/versions/069_resource_url_and_metadata_modified.py#L22
[2] https://github.com/ckan/ckanext-harvest/blob/master/ckanext/harvest/model/__init__.py#L42
On 31 January 2014 10:32, Hildegard Gerlach
<hildegard.gerlach at jrc.ec.europa.eu> wrote:
> Dear all,
>
> I understood now what was the reason for not upgrading the database
> properly. When I had first installed CKAN I had installed version 2.0. When
> I did the first upgrade (before November, I suppose) I had used the master
> branch (which was already 2.2a or something) until someone recommended later
> to always use a stable version. So I went back to version 2.1 (and obviously
> didn't downgrade the database, if that can be done at all ?). In my dump
> files from November and January I can already find the added fields e.g. in
> package (metadata_modified, creator_user_id) and version 70 in table
> migrate_version. So obviously this
>
> https://github.com/ckan/ckan/blob/master/ckan/migration/versions/069_resource_url_and_metadata_modified.py#L22
>
> was not executed any more, there was only an upgrade from 70 to 71, and I
> remained with the empty fields in metadata_modified.
>
> If I import a db dump from June after just having installed CKAN 2.0 with
> migrate_version 67, the paster db upgrade works fine.
>
> Sorry for having you occupied trying to reproduce the problem.
>
> At least to me it is now clearer how the db upgrade is done. What about
> changes in the extensions (e.g harvesting) db schema ? Is this handled as
> well with the db upgrade ?
>
> Thanks
>
> Hilde
>
>
> On 1/30/2014 3:45 PM, Hildegard Gerlach wrote:
>>
>> I will try with a database dump from November to see if there is a
>> difference. I will let you know.
>>
>> However, I did my backup with the paster command
>> paster db dump
>>
>> so I think I can only do the import in the same way
>>
>> paster db load
>>
>> Thanks
>> Hilde
>>
>> On 1/30/2014 3:28 PM, David Raznick wrote:
>>>
>>> Hello
>>>
>>> This stumps me, mainly because I do not know how to reproduce.
>>>
>>> The only things I can think of now that could be the issue.
>>>
>>> * postgres 8.4, David is your production 8.4 or 9.X?
>>> * using the paster db load commands (instead of using pg_dump) may in
>>> some way confuse the issue.
>>> * possibly dumping the db when you have upgraded CKAN to the new version
>>>
>>> Hilde it looks like the db has been upgraded already before you did
>>> the initial dump. Is there a dump you did with the old db before you
>>> upgraded CKAN at all.
>>>
>>> David
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On 30 January 2014 11:12, Hildegard Gerlach
>>> <hildegard.gerlach at jrc.ec.europa.eu> wrote:
>>>>
>>>> Hi,
>>>>
>>>> again me.
>>>> I created a new DB, then
>>>> paster db clean -c development.ini ckantest_dump_28Jan14.sql (is needed
>>>> to
>>>> clear the postgis tables, otherwise it claims when loading)
>>>> paster db load -c development.ini ckantest_dump_28Jan14.sql
>>>>
>>>> and here at the end I get the following:
>>>>
>>>> Loaded CKAN database: ckantest_dump_28Jan14.sql
>>>> Upgrading DB
>>>> Rebuilding search index
>>>> Traceback (most recent call last):
>>>> File "/usr/local/mtester.ies.jrc.it/pyenv/bin/paster", line 9, in
>>>> <module>
>>>> load_entry_point('PasteScript==1.7.5', 'console_scripts',
>>>> 'paster')()
>>>> File
>>>>
>>>> "/usr/local/mtester.ies.jrc.it/pyenv/lib/python2.6/site-packages/paste/script/command.py",
>>>> line 104, in run
>>>> invoke(command, command_name, options, args[1:])
>>>> File
>>>>
>>>> "/usr/local/mtester.ies.jrc.it/pyenv/lib/python2.6/site-packages/paste/script/command.py",
>>>> line 143, in invoke
>>>> exit_code = runner.run(args)
>>>> File
>>>>
>>>> "/usr/local/mtester.ies.jrc.it/pyenv/lib/python2.6/site-packages/paste/script/command.py",
>>>> line 238, in run
>>>> result = self.command()
>>>> File "/usr/local/mtester.ies.jrc.it/pyenv/src/ckan/ckan/lib/cli.py",
>>>> line
>>>> 174, in command
>>>> self.load()
>>>> File "/usr/local/mtester.ies.jrc.it/pyenv/src/ckan/ckan/lib/cli.py",
>>>> line
>>>> 263, in load
>>>> ckan.lib.search.rebuild()
>>>> File
>>>>
>>>> "/usr/local/mtester.ies.jrc.it/pyenv/src/ckan/ckan/lib/search/__init__.py",
>>>> line 190, in rebuild
>>>> {'id': pkg_id}
>>>> File
>>>> "/usr/local/mtester.ies.jrc.it/pyenv/src/ckan/ckan/logic/__init__.py",
>>>> line
>>>> 419, in wrapped
>>>> result = _action(context, data_dict, **kw)
>>>> File
>>>> "/usr/local/mtester.ies.jrc.it/pyenv/src/ckan/ckan/logic/action/get.py",
>>>> line 830, in package_show
>>>> package_dict = model_dictize.package_dictize(pkg, context)
>>>>
>>>> File
>>>>
>>>> "/usr/local/mtester.ies.jrc.it/pyenv/src/ckan/ckan/lib/dictization/model_dictize.py",
>>>> line 329, in package_dictize
>>>> result_dict['metadata_modified'] =
>>>> pkg.metadata_modified.isoformat()
>>>> AttributeError: 'NoneType' object has no attribute 'isoformat'
>>>>
>>>> So if I understand correctly, paster db does not only load the data, but
>>>> does as well the upgrade (which partly fails) and then rebuilds the
>>>> search
>>>> index with the same problem. The metadata_modified field is mostly empty
>>>> but
>>>> not always.
>>>>
>>>> I try again to do
>>>> paster db upgrade -c development.ini
>>>>
>>>> but the fields are not filled.
>>>>
>>>> How can I find out if the script you sent me is really executed ?
>>>>
>>>> https://github.com/ckan/ckan/blob/master/ckan/migration/versions/069_resource_url_and_metadata_modified.py#L22
>>>>
>>>> I am using PostgreSQL 8.4 on Linux RH6.
>>>>
>>>> This I can find in the logfile.
>>>>
>>>> 2014-01-30 11:50:42,402 INFO [ckan.model] Database tables dropped
>>>> 2014-01-30 11:51:18,251 INFO [ckan.model] CKAN database version
>>>> upgraded:
>>>> 70 -> 71
>>>> 2014-01-30 11:51:18,334 INFO [ckan.lib.search] Rebuilding search
>>>> index...
>>>> 2014-01-30 11:51:18,365 INFO [ckan.lib.search] Rebuilding the whole
>>>> index...
>>>> 2014-01-30 11:51:18,460 ERROR [ckan.lib.search] Error while indexing
>>>> dataset
>>>> 310ef993-b947-407e-87a8-717bcbeb1911: 'NoneType' object has no attribute
>>>> 'isoformat'
>>>> 2014-01-30 12:00:42,158 INFO [ckan.model] CKAN database version remains
>>>> as:
>>>> 71
>>>> 2014-01-30 12:01:42,697 INFO [ckan.model] CKAN database version remains
>>>> as:
>>>> 71
>>>>
>>>> So the upgrade command does obviously not execute the update package
>>>> command
>>>> in my case.
>>>>
>>>> Hope this helps you.
>>>>
>>>> Hilde
>>>>
>>>>
>>>>
>>>> On 1/30/2014 11:41 AM, Hildegard Gerlach wrote:
>>>>>
>>>>> Dear all,
>>>>>
>>>>> thanks to all for the info.
>>>>>
>>>>> To answer Adria's question: No, I didn't have any harvesters running
>>>>> when
>>>>> I did the upgrade last time.
>>>>> I am not sure if it could have been a problem that I didn't stop the
>>>>> httpd
>>>>> service and I had left the plugins for the extensions in the
>>>>> configuration
>>>>> file.
>>>>>
>>>>> What I did now is:
>>>>> stop httpd, disable plugins for extensions in the configuration file
>>>>> create a clean database, import (paster db load) my latest db dump file
>>>>> from 28 January just before the upgrade.
>>>>> This already adds the new database fields as in version 2.2, i.e. the
>>>>> first 4 commands from here.
>>>>>
>>>>>
>>>>> https://github.com/ckan/ckan/blob/master/ckan/migration/versions/069_resource_url_and_metadata_modified.py#L22
>>>>>
>>>>> Is this supposed to be so ?
>>>>> The metadata_modified is empty at this stage. I then executed all the
>>>>> update commands and the metadata_modified fields are filled.
>>>>>
>>>>> After that I can rebuild the search-index and CKAN works !
>>>>>
>>>>> I propose to do the same thing again but using the paster db upgrade
>>>>> after
>>>>> loading the database to see if it happens again.
>>>>>
>>>>> Hilde
>>>>>
>>>>>
>>>>>
>>>>> On 1/30/2014 9:34 AM, Adrià Mercader wrote:
>>>>>>
>>>>>> Hi Hilde,
>>>>>>
>>>>>> One possible thing that comes to mind is whether if you had harvesters
>>>>>> running at the time of doing the upgrade (by running I mean the
>>>>>> processes for gather_consumer and fetch_consumer were up). If so, and
>>>>>> you triggered a reharvest after the update, the harvesters may have
>>>>>> still been running with the 2.1 code, thus not filling the
>>>>>> metadata_modified field. Could that be the case?
>>>>>>
>>>>>> Regardless, it would be great if you could test the commands the David
>>>>>> mentioned on the previous email to help us debug this.
>>>>>>
>>>>>> Many thanks,
>>>>>>
>>>>>> Adrià
>>>>>>
>>>>>>
>>>>>> On 29 January 2014 22:06, David Raznick <david.raznick at okfn.org>
>>>>>> wrote:
>>>>>>>
>>>>>>> Hello
>>>>>>>
>>>>>>> The only thing I can think of is an error happened with the db
>>>>>>> upgrade
>>>>>>> or that for some reason your db is not upgraded correctly.
>>>>>>>
>>>>>>> I have had a look over the upgrade script and tested it on some of
>>>>>>> our
>>>>>>> instances and it appears fine, i.e no missing metadata_modified (you
>>>>>>> are correct that this is the issue).
>>>>>>>
>>>>>>>
>>>>>>> Is there anything odd about your db instance or any modifications you
>>>>>>> made to it?
>>>>>>>
>>>>>>> As this is critical bug, if it effects our 2.2 release, could you
>>>>>>> help
>>>>>>> us diagnose this issue?
>>>>>>>
>>>>>>> Could you run the db commands manually from line 22 to 55 located
>>>>>>> here:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> https://github.com/ckan/ckan/blob/master/ckan/migration/versions/069_resource_url_and_metadata_modified.py#L22
>>>>>>>
>>>>>>> The first update should fill in all the metadata_modified data and
>>>>>>> the
>>>>>>> following should just update them with later dates if needed. Could
>>>>>>> you see if at any stage in running these commands make the
>>>>>>> metadata_modifed fields became null.
>>>>>>>
>>>>>>> David
>>>>>>>
>>>>>>>
>>>>>>> On 29 January 2014 18:36, Hildegard Gerlach
>>>>>>> <hildegard.gerlach at jrc.ec.europa.eu> wrote:
>>>>>>>>
>>>>>>>> Yes, I wrote it in the beginning of the message but forgot to wrote
>>>>>>>> it
>>>>>>>> in the commands below. But yes, I have run it.
>>>>>>>>
>>>>>>>> Hilde
>>>>>>>>
>>>>>>>>
>>>>>>>> On 29.01.2014, at 19:31, David Raznick <david.raznick at okfn.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hello
>>>>>>>>
>>>>>>>> Did you run
>>>>>>>>
>>>>>>>> paster db upgrade
>>>>>>>>
>>>>>>>> I would run it after python setup.py develop
>>>>>>>>
>>>>>>>> David
>>>>>>>>
>>>>>>>> On 29 January 2014 17:56, Hildegard Gerlach
>>>>>>>> <hildegard.gerlach at jrc.ec.europa.eu> wrote:
>>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> I started to upgrade a CKAN 2.1.1 version to 2.2. I did the paster
>>>>>>>>> db
>>>>>>>>> upgrade command.
>>>>>>>>> When I want to rebuild the search index I get the following error:
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> "/usr/local/mtester.ies.jrc.it/pyenv/src/ckan/ckan/lib/dictization/model_dictize.py",
>>>>>>>>> line 329, in package_dictize
>>>>>>>>> result_dict['metadata_modified'] =
>>>>>>>>> pkg.metadata_modified.isoformat()
>>>>>>>>>
>>>>>>>>> I cannot see any data anymore on the CKAN Web site as the index is
>>>>>>>>> empty but
>>>>>>>>> the data is still in the database.
>>>>>>>>> It seems that there are 2 new fields in the table package:
>>>>>>>>> metadata_modified
>>>>>>>>> and creator_user_id
>>>>>>>>>
>>>>>>>>> The value of metadata_modified seems to be mostly empty in the
>>>>>>>>> database
>>>>>>>>> records which, I think, causes the error.
>>>>>>>>>
>>>>>>>>> I tried as well to clean the database and reimport it (paster db
>>>>>>>>> clean,
>>>>>>>>> paster db load) and I get the same error again with the import. The
>>>>>>>>> database
>>>>>>>>> is already with the new fields as after the upgrade.
>>>>>>>>>
>>>>>>>>> I had done the commands in the following order:
>>>>>>>>>
>>>>>>>>> git checkout release-v2.2
>>>>>>>>> git pull
>>>>>>>>> paster db dump -c development.ini ckantest_dump_28Jan14.sql
>>>>>>>>> pip install --upgrade -r requirements.txt
>>>>>>>>> python setup.py develop
>>>>>>>>> restart Tomcat and httpd
>>>>>>>>> paster search-index rebuild -r -c development.ini
>>>>>>>>>
>>>>>>>>> Luckily this is a test version and I can restart with an empty
>>>>>>>>> database and
>>>>>>>>> reharvest data. But in production it seems difficult to dare
>>>>>>>>> upgrading.
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>>
>>>>>>>>> Hilde
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> ckan-dev mailing list
>>>>>>>>> ckan-dev at lists.okfn.org
>>>>>>>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>>>>>>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>>>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> ckan-dev mailing list
>>>>> ckan-dev at lists.okfn.org
>>>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>>>
>>>>
>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
More information about the ckan-dev
mailing list