[ckan-dev] problems upgrading to CKAN 2.2

• Previous message: [ckan-dev] problems upgrading to CKAN 2.2
• Next message: [ckan-dev] problems upgrading to CKAN 2.2
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

the dump should just an sql file so you should be able to load it to
postgres using the postgres commands. But you can try use paster db
load-only which should not do the upgrade or solr-indexing.

On 30 January 2014 14:45, Hildegard Gerlach
<hildegard.gerlach at jrc.ec.europa.eu> wrote:
> I will try with a database dump from November to see if there is a
> difference. I will let you know.
>
> However, I did my backup with the paster command
> paster db dump
>
> so I think I can only do the import in the same way
>
> paster db load
>
> Thanks
> Hilde
>
>
> On 1/30/2014 3:28 PM, David Raznick wrote:
>>
>> Hello
>>
>> This stumps me, mainly because I do not know how to reproduce.
>>
>> The only things I can think of now that could be the issue.
>>
>> * postgres 8.4,  David is your production 8.4 or 9.X?
>> * using the paster db load commands (instead of using pg_dump) may in
>> some way confuse the issue.
>> * possibly dumping the db when you have upgraded CKAN to the new version
>>
>> Hilde it looks like the db has been upgraded already before you did
>> the initial dump.  Is there a dump you did with the old db before you
>> upgraded CKAN at all.
>>
>> David
>>
>>
>>
>>
>>
>>
>>
>> On 30 January 2014 11:12, Hildegard Gerlach
>> <hildegard.gerlach at jrc.ec.europa.eu> wrote:
>>>
>>> Hi,
>>>
>>> again me.
>>> I created a new DB, then
>>> paster db clean -c development.ini  ckantest_dump_28Jan14.sql (is needed
>>> to
>>> clear the postgis tables, otherwise it claims when loading)
>>> paster db load -c development.ini  ckantest_dump_28Jan14.sql
>>>
>>> and here at the end I get the following:
>>>
>>> Loaded CKAN database: ckantest_dump_28Jan14.sql
>>> Upgrading DB
>>> Rebuilding search index
>>> Traceback (most recent call last):
>>>    File "/usr/local/mtester.ies.jrc.it/pyenv/bin/paster", line 9, in
>>> <module>
>>>      load_entry_point('PasteScript==1.7.5', 'console_scripts',
>>> 'paster')()
>>>    File
>>>
>>> "/usr/local/mtester.ies.jrc.it/pyenv/lib/python2.6/site-packages/paste/script/command.py",
>>> line 104, in run
>>>      invoke(command, command_name, options, args[1:])
>>>    File
>>>
>>> "/usr/local/mtester.ies.jrc.it/pyenv/lib/python2.6/site-packages/paste/script/command.py",
>>> line 143, in invoke
>>>      exit_code = runner.run(args)
>>>    File
>>>
>>> "/usr/local/mtester.ies.jrc.it/pyenv/lib/python2.6/site-packages/paste/script/command.py",
>>> line 238, in run
>>>      result = self.command()
>>>    File "/usr/local/mtester.ies.jrc.it/pyenv/src/ckan/ckan/lib/cli.py",
>>> line
>>> 174, in command
>>>      self.load()
>>>    File "/usr/local/mtester.ies.jrc.it/pyenv/src/ckan/ckan/lib/cli.py",
>>> line
>>> 263, in load
>>>      ckan.lib.search.rebuild()
>>>    File
>>>
>>> "/usr/local/mtester.ies.jrc.it/pyenv/src/ckan/ckan/lib/search/__init__.py",
>>> line 190, in rebuild
>>>      {'id': pkg_id}
>>>    File
>>> "/usr/local/mtester.ies.jrc.it/pyenv/src/ckan/ckan/logic/__init__.py",
>>> line
>>> 419, in wrapped
>>>      result = _action(context, data_dict, **kw)
>>>    File
>>> "/usr/local/mtester.ies.jrc.it/pyenv/src/ckan/ckan/logic/action/get.py",
>>> line 830, in package_show
>>>      package_dict = model_dictize.package_dictize(pkg, context)
>>>
>>>    File
>>>
>>> "/usr/local/mtester.ies.jrc.it/pyenv/src/ckan/ckan/lib/dictization/model_dictize.py",
>>> line 329, in package_dictize
>>>      result_dict['metadata_modified'] = pkg.metadata_modified.isoformat()
>>> AttributeError: 'NoneType' object has no attribute 'isoformat'
>>>
>>> So if I understand correctly, paster db does not only load the data, but
>>> does as well the upgrade (which partly fails) and then rebuilds the
>>> search
>>> index with the same problem. The metadata_modified field is mostly empty
>>> but
>>> not always.
>>>
>>> I try again to do
>>> paster db upgrade -c development.ini
>>>
>>> but the fields are not filled.
>>>
>>> How can I find out if the script you sent me is really executed ?
>>>
>>> https://github.com/ckan/ckan/blob/master/ckan/migration/versions/069_resource_url_and_metadata_modified.py#L22
>>>
>>> I am using  PostgreSQL 8.4 on Linux RH6.
>>>
>>> This I can find in the logfile.
>>>
>>> 2014-01-30 11:50:42,402 INFO  [ckan.model] Database tables dropped
>>> 2014-01-30 11:51:18,251 INFO  [ckan.model] CKAN database version
>>> upgraded:
>>> 70 -> 71
>>> 2014-01-30 11:51:18,334 INFO  [ckan.lib.search] Rebuilding search
>>> index...
>>> 2014-01-30 11:51:18,365 INFO  [ckan.lib.search] Rebuilding the whole
>>> index...
>>> 2014-01-30 11:51:18,460 ERROR [ckan.lib.search] Error while indexing
>>> dataset
>>> 310ef993-b947-407e-87a8-717bcbeb1911: 'NoneType' object has no attribute
>>> 'isoformat'
>>> 2014-01-30 12:00:42,158 INFO  [ckan.model] CKAN database version remains
>>> as:
>>> 71
>>> 2014-01-30 12:01:42,697 INFO  [ckan.model] CKAN database version remains
>>> as:
>>> 71
>>>
>>> So the upgrade command does obviously not execute the update package
>>> command
>>> in my case.
>>>
>>> Hope this helps you.
>>>
>>> Hilde
>>>
>>>
>>>
>>> On 1/30/2014 11:41 AM, Hildegard Gerlach wrote:
>>>>
>>>> Dear all,
>>>>
>>>> thanks to all for the info.
>>>>
>>>> To answer Adria's question: No, I didn't have any harvesters running
>>>> when
>>>> I did the upgrade last time.
>>>> I am not sure if it could have been a problem that I didn't stop the
>>>> httpd
>>>> service and I had left the plugins for the extensions in the
>>>> configuration
>>>> file.
>>>>
>>>> What I did now is:
>>>> stop httpd, disable plugins for extensions in the configuration file
>>>> create a clean database, import (paster db load) my latest db dump file
>>>> from 28 January just before the upgrade.
>>>> This already adds the new database fields as in version 2.2, i.e. the
>>>> first 4 commands from here.
>>>>
>>>>
>>>> https://github.com/ckan/ckan/blob/master/ckan/migration/versions/069_resource_url_and_metadata_modified.py#L22
>>>>
>>>> Is this supposed to be so ?
>>>> The metadata_modified is empty at this stage. I then executed all the
>>>> update commands and the metadata_modified fields are filled.
>>>>
>>>> After that I can rebuild the search-index and CKAN works !
>>>>
>>>> I propose to do the same thing again but using the paster db upgrade
>>>> after
>>>> loading the database to see if it happens again.
>>>>
>>>> Hilde
>>>>
>>>>
>>>>
>>>> On 1/30/2014 9:34 AM, Adrià Mercader wrote:
>>>>>
>>>>> Hi Hilde,
>>>>>
>>>>> One possible thing that comes to mind is whether if you had harvesters
>>>>> running at the time of doing the upgrade (by running I mean the
>>>>> processes for gather_consumer and fetch_consumer were up). If so, and
>>>>> you triggered a reharvest after the update, the harvesters may have
>>>>> still been running with the 2.1 code, thus not filling the
>>>>> metadata_modified field. Could that be the case?
>>>>>
>>>>> Regardless, it would be great if you could test the commands the David
>>>>> mentioned on the previous email to help us debug this.
>>>>>
>>>>> Many thanks,
>>>>>
>>>>> Adrià
>>>>>
>>>>>
>>>>> On 29 January 2014 22:06, David Raznick <david.raznick at okfn.org> wrote:
>>>>>>
>>>>>> Hello
>>>>>>
>>>>>> The only thing I can think of is an error happened with the db upgrade
>>>>>> or that for some reason your db is not upgraded correctly.
>>>>>>
>>>>>> I have had a look over the upgrade script and tested it on some of our
>>>>>> instances and it appears fine, i.e no missing metadata_modified  (you
>>>>>> are correct that this is the issue).
>>>>>>
>>>>>>
>>>>>> Is there anything odd about your db instance or any modifications you
>>>>>> made to it?
>>>>>>
>>>>>> As this is critical bug, if it effects our 2.2 release, could you help
>>>>>> us diagnose this issue?
>>>>>>
>>>>>> Could you run the db commands manually from line 22 to 55 located
>>>>>> here:
>>>>>>
>>>>>>
>>>>>>
>>>>>> https://github.com/ckan/ckan/blob/master/ckan/migration/versions/069_resource_url_and_metadata_modified.py#L22
>>>>>>
>>>>>> The first update should fill in all the metadata_modified data and the
>>>>>> following should just update them with later dates if needed. Could
>>>>>> you see if at any stage in running these commands make the
>>>>>> metadata_modifed fields became null.
>>>>>>
>>>>>> David
>>>>>>
>>>>>>
>>>>>> On 29 January 2014 18:36, Hildegard Gerlach
>>>>>> <hildegard.gerlach at jrc.ec.europa.eu> wrote:
>>>>>>>
>>>>>>> Yes, I wrote it in the beginning of the message but forgot to wrote
>>>>>>> it
>>>>>>> in the commands below. But yes, I have run it.
>>>>>>>
>>>>>>> Hilde
>>>>>>>
>>>>>>>
>>>>>>> On 29.01.2014, at 19:31, David Raznick <david.raznick at okfn.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Hello
>>>>>>>
>>>>>>> Did you run
>>>>>>>
>>>>>>> paster db upgrade
>>>>>>>
>>>>>>> I would run it after python setup.py develop
>>>>>>>
>>>>>>> David
>>>>>>>
>>>>>>> On 29 January 2014 17:56, Hildegard Gerlach
>>>>>>> <hildegard.gerlach at jrc.ec.europa.eu> wrote:
>>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> I started to upgrade a CKAN 2.1.1 version to 2.2. I did the paster
>>>>>>>> db
>>>>>>>> upgrade command.
>>>>>>>> When I want to rebuild the search index I get the following error:
>>>>>>>>
>>>>>>>> File
>>>>>>>>
>>>>>>>>
>>>>>>>> "/usr/local/mtester.ies.jrc.it/pyenv/src/ckan/ckan/lib/dictization/model_dictize.py",
>>>>>>>> line 329, in package_dictize
>>>>>>>>      result_dict['metadata_modified'] =
>>>>>>>> pkg.metadata_modified.isoformat()
>>>>>>>>
>>>>>>>> I cannot see any data anymore on the CKAN Web site as the index is
>>>>>>>> empty but
>>>>>>>> the data is still in the database.
>>>>>>>> It seems that there are 2 new fields in the table package:
>>>>>>>> metadata_modified
>>>>>>>> and creator_user_id
>>>>>>>>
>>>>>>>> The value of metadata_modified  seems to be mostly empty in the
>>>>>>>> database
>>>>>>>> records which, I think, causes the error.
>>>>>>>>
>>>>>>>> I tried as well to clean the database and reimport it (paster db
>>>>>>>> clean,
>>>>>>>> paster db load) and I get the same error again with the import. The
>>>>>>>> database
>>>>>>>> is already with the new fields as after the upgrade.
>>>>>>>>
>>>>>>>> I had done the commands in the following order:
>>>>>>>>
>>>>>>>> git checkout release-v2.2
>>>>>>>> git pull
>>>>>>>> paster db dump -c development.ini ckantest_dump_28Jan14.sql
>>>>>>>> pip install --upgrade -r requirements.txt
>>>>>>>> python setup.py develop
>>>>>>>> restart Tomcat and httpd
>>>>>>>> paster search-index rebuild -r -c development.ini
>>>>>>>>
>>>>>>>> Luckily this is a test version and I can restart with an empty
>>>>>>>> database and
>>>>>>>> reharvest data. But in production it seems difficult to dare
>>>>>>>> upgrading.
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>>
>>>>>>>> Hilde
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> ckan-dev mailing list
>>>>>>>> ckan-dev at lists.okfn.org
>>>>>>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>>>>>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>>>>>>>
>>>>
>>>> _______________________________________________
>>>> ckan-dev mailing list
>>>> ckan-dev at lists.okfn.org
>>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>>
>>>
>>>
>>> _______________________________________________
>>> ckan-dev mailing list
>>> ckan-dev at lists.okfn.org
>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>
>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev

• Previous message: [ckan-dev] problems upgrading to CKAN 2.2
• Next message: [ckan-dev] problems upgrading to CKAN 2.2
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]