[ECODP-dev] PRODUCTION 09 upgrade
Dimitrios Mexis
dimitrios.mexis at tenforce.com
Thu Oct 10 11:35:05 UTC 2013
Thanks John
On 10/10/2013 13:05, John Glover wrote:
> Hi Dimitrios,
>
> Yes, that paragraph can remain largely unchanged (I have added a note
> to reindex again after the migration).
>
> Those paster commands to purge old data are not specifically for this
> upgrade: the first one should have been run after release 09 was
> installed (and the old archiver/datastorer/qa extensions removed), the
> second should be run regularly by CRON and so there should not be so
> much data in that table at any given time (I am making a note about
> this elsewhere however as it is not in your list of CRON task in the
> Operations Manual).
>
> Regards,
> John
>
>
> On 10 October 2013 12:39, Dimitrios Mexis
> <dimitrios.mexis at tenforce.com <mailto:dimitrios.mexis at tenforce.com>>
> wrote:
>
> Hi John
> good to know that you made it.
>
> Can you check the paragraph of the document in google that refers
> to the upgrade of 09 version, with what ever needs to be done in
> the end ?
>
>
> Thanks
> Dimitrios
>
>
> On 10/10/2013 11:52, John Glover wrote:
>> Hi Dimitrios,
>>
>> I have successfully ran through the namespace migration without
>> any problems. I have reindexed the site and it is now back up.
>>
>> First however, I made sure to run the following commands (as
>> documented previously in relation to the database size):
>>
>> paster --plugin=ckanext-ecportal ecportal purge-task-data -c <config>
>> paster --plugin=ckanext-ecportal ecportal
>> purge-package-extra-revision -c <config>
>>
>> I believe that the second command in particular is responsible
>> for the speed up in the migration (for me it took just a couple
>> of minutes). The reason is that we have to update the metadata
>> for all datasets (and all of their revisions) in the database,
>> and as no one had cleared out the old entries from the
>> package_extra_revision table, it was performing individual update
>> statements over about 2 million records (instead of the 60
>> thousand that are there now).
>>
>> Regards,
>> John
>>
>>
>> On 10 October 2013 10:12, Dimitrios Mexis
>> <dimitrios.mexis at tenforce.com
>> <mailto:dimitrios.mexis at tenforce.com>> wrote:
>>
>> Ok.
>>
>> Thanks.
>> Dimitrios
>>
>>
>> On 10/10/2013 10:08, John Glover wrote:
>>> Ok thanks. Could you please not run any of the steps again
>>> for now so I can have a look at the database?
>>>
>>> Thanks,
>>> John
>>>
>>>
>>> On 10 October 2013 10:06, Dimitrios Mexis
>>> <dimitrios.mexis at tenforce.com
>>> <mailto:dimitrios.mexis at tenforce.com>> wrote:
>>>
>>> John no ,
>>> no I found it "killed"
>>>
>>> Regards
>>> Dimitrios
>>>
>>>
>>> On 10/10/2013 10:04, John Glover wrote:
>>>> Hi Dimitrios,
>>>>
>>>> Did the namespace migration step finish the last time
>>>> you ran it or did you kill it?
>>>>
>>>> Thanks,
>>>> John
>>>>
>>>>
>>>> On 10 October 2013 09:53, Dimitrios Mexis
>>>> <dimitrios.mexis at tenforce.com
>>>> <mailto:dimitrios.mexis at tenforce.com>> wrote:
>>>>
>>>> Hi Glen,
>>>>
>>>> Ok ight now I am feeding again the database with
>>>> data (pg_restore ).
>>>> The back up is under $USERHOME/ckan/backups called,
>>>> PO-release09.tgz
>>>>
>>>> We will try to rename this publisher that caused
>>>> issues before we start the procedure.
>>>>
>>>> regards
>>>> Dimitrios
>>>>
>>>>
>>>>
>>>> On 10/10/2013 9:46, John Glover wrote:
>>>>> Hi Dimitrios,
>>>>>
>>>>> Did you have to kill the migration step again? Can
>>>>> you tell me the path to the database dump on the
>>>>> postgres machine so I can try loading and running
>>>>> it myself a few times?
>>>>>
>>>>> Thanks,
>>>>> John
>>>>>
>>>>>
>>>>> On 10 October 2013 09:12, Dimitrios Mexis
>>>>> <dimitrios.mexis at tenforce.com
>>>>> <mailto:dimitrios.mexis at tenforce.com>> wrote:
>>>>>
>>>>> Good morning.
>>>>>
>>>>> Here can you explain the issue ? How do we
>>>>> de-block it ?
>>>>>
>>>>>
>>>>> [root at tffvirt-rh-template ~]# screen -x 25060
>>>>> +++ unset pydoc
>>>>> +++ '[' -n '' ']'
>>>>> +++ '[' -n '' ']'
>>>>> +++ '[' -n /bin/bash -o -n '' ']'
>>>>> +++ hash -r
>>>>> +++ '[' -n '' ']'
>>>>> +++ unset VIRTUAL_ENV
>>>>> +++ '[' '!' nondestructive = nondestructive ']'
>>>>> +++
>>>>> VIRTUAL_ENV=/applications/ecodp/users/ecodp/ckan/ecportal
>>>>> +++ export VIRTUAL_ENV
>>>>> +++
>>>>> _OLD_VIRTUAL_PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
>>>>> +++
>>>>> PATH=/applications/ecodp/users/ecodp/ckan/ecportal/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
>>>>> +++ export PATH
>>>>> +++ '[' -n '' ']'
>>>>> +++ '[' -z '' ']'
>>>>> +++ _OLD_VIRTUAL_PS1=
>>>>> +++ '[' x '!=' x ']'
>>>>> ++++ basename
>>>>> /applications/ecodp/users/ecodp/ckan/ecportal
>>>>> +++ '[' ecportal = __ ']'
>>>>> ++++ basename
>>>>> /applications/ecodp/users/ecodp/ckan/ecportal
>>>>> +++ PS1='(ecportal)'
>>>>> +++ export PS1
>>>>> +++ alias 'pydoc=python -m pydoc'
>>>>> +++ '[' -n /bin/bash -o -n '' ']'
>>>>> +++ hash -r
>>>>> ++ cd
>>>>> /applications/ecodp/users/ecodp//ckan/ecportal/src/ckan
>>>>> + set -x
>>>>> + paster --plugin=ckan search-index rebuild -r
>>>>> -c
>>>>> /applications/ecodp/users/ecodp//ckan/conf/ecportal.ini
>>>>> 2013-10-09 12:40:58,527 INFO [ckan.lib.search]
>>>>> Rebuilding search index...
>>>>> 2013-10-09 12:40:58,946 INFO [ckan.lib.search]
>>>>> Rebuilding the whole index...
>>>>> 2013-10-09 12:59:56,800 INFO [ckan.lib.search]
>>>>> Finished rebuilding search index.
>>>>> 2013-10-09 12:59:59,174 INFO [ckan.lib.search]
>>>>> Commited pending changes on the search index
>>>>> + paster --plugin=ckanext-ecportal ecportal
>>>>> migrate-odp-namespace -c
>>>>> /applications/ecodp/users/ecodp//ckan/conf/ecportal.ini
>>>>> ./ckan_upgrade_09to01.sh: line 21: 5031
>>>>> Killed paster --plugin=ckanext-ecportal
>>>>> ecportal migrate-odp-namespace -c $CKAN_INI
>>>>> + paster --plugin=ckanext-ecportal ecportal
>>>>> update-all-vocabs -c
>>>>> /applications/ecodp/users/ecodp//ckan/conf/ecportal.ini
>>>>> + paster --plugin=ckanext-ecportal ecportal
>>>>> update-publishers -c
>>>>> /applications/ecodp/users/ecodp//ckan/conf/ecportal.ini
>>>>> Traceback (most recent call last):
>>>>> File
>>>>> "/applications/ecodp/users/ecodp/ckan/ecportal/bin/paster",
>>>>> line 9, in <module>
>>>>> load_entry_point('PasteScript==1.7.5',
>>>>> 'console_scripts', 'paster')()
>>>>> File
>>>>> "/applications/ecodp/users/ecodp/ckan/ecportal/lib/python2.6/site-packages/paste/script/command.py",
>>>>> line 104, in run
>>>>> invoke(command, command_name, options, args[1:])
>>>>> File
>>>>> "/applications/ecodp/users/ecodp/ckan/ecportal/lib/python2.6/site-packages/paste/script/command.py",
>>>>> line 143, in invoke
>>>>> exit_code = runner.run(args)
>>>>> File
>>>>> "/applications/ecodp/users/ecodp/ckan/ecportal/lib/python2.6/site-packages/paste/script/command.py",
>>>>> line 238, in run
>>>>> result = self.command()
>>>>> File
>>>>> "/applications/ecodp/users/ecodp/ckan/ecportal/src/ckanext-ecportal/ckanext/ecportal/commands.py",
>>>>> line 109, in command
>>>>> self.update_publishers(file_path)
>>>>> File
>>>>> "/applications/ecodp/users/ecodp/ckan/ecportal/src/ckanext-ecportal/ckanext/ecportal/commands.py",
>>>>> line 527, in update_publishers
>>>>> self._add_publishers(new_publishers)
>>>>> File
>>>>> "/applications/ecodp/users/ecodp/ckan/ecportal/src/ckanext-ecportal/ckanext/ecportal/commands.py",
>>>>> line 213, in _add_publishers
>>>>> plugins.toolkit.get_action('group_create')(context,
>>>>> group)
>>>>> File
>>>>> "/applications/ecodp/users/ecodp/ckan/ecportal/src/ckan/ckan/logic/action/create.py",
>>>>> line 538, in group_create
>>>>> raise ValidationError(errors)
>>>>> ckan.logic.ValidationError: {'Name': 'Group
>>>>> name already exists in database'}
>>>>> + paster --plugin=ckanext-ecportal ecportal
>>>>> import-csv-translations -c
>>>>> /applications/ecodp/users/ecodp//ckan/conf/ecportal.ini
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.okfn.org/mailman/private/ecodp-dev/attachments/20131010/fa151af6/attachment.html>
More information about the ecodp-dev
mailing list