[ECODP-dev] PRODUCTION 09 upgrade

Dimitrios Mexis dimitrios.mexis at tenforce.com
Thu Oct 10 11:35:05 UTC 2013


Thanks John

On 10/10/2013 13:05, John Glover wrote:
> Hi Dimitrios,
>
> Yes, that paragraph can remain largely unchanged (I have added a note 
> to reindex again after the migration).
>
> Those paster commands to purge old data are not specifically for this 
> upgrade: the first one should have been run after release 09 was 
> installed (and the old archiver/datastorer/qa extensions removed), the 
> second should be run regularly by CRON and so there should not be so 
> much data in that table at any given time (I am making a note about 
> this elsewhere however as it is not in your list of CRON task in the 
> Operations Manual).
>
> Regards,
> John
>
>
> On 10 October 2013 12:39, Dimitrios Mexis 
> <dimitrios.mexis at tenforce.com <mailto:dimitrios.mexis at tenforce.com>> 
> wrote:
>
>     Hi John
>     good to know that you made it.
>
>     Can you check the paragraph of the document in google that refers
>     to the upgrade of 09 version, with what ever needs to be done in
>     the end ?
>
>
>     Thanks
>     Dimitrios
>
>
>     On 10/10/2013 11:52, John Glover wrote:
>>     Hi Dimitrios,
>>
>>     I have successfully ran through the namespace migration without
>>     any problems. I have reindexed the site and it is now back up.
>>
>>     First however, I made sure to run the following commands (as
>>     documented previously in relation to the database size):
>>
>>     paster --plugin=ckanext-ecportal ecportal purge-task-data -c <config>
>>     paster --plugin=ckanext-ecportal ecportal
>>     purge-package-extra-revision -c <config>
>>
>>     I believe that the second command in particular is responsible
>>     for the speed up in the migration (for me it took just a couple
>>     of minutes). The reason is that we have to update the metadata
>>     for all datasets (and all of their revisions) in the database,
>>     and as no one had cleared out the old entries from the
>>     package_extra_revision table, it was performing individual update
>>     statements over about 2 million records (instead of the 60
>>     thousand that are there now).
>>
>>     Regards,
>>     John
>>
>>
>>     On 10 October 2013 10:12, Dimitrios Mexis
>>     <dimitrios.mexis at tenforce.com
>>     <mailto:dimitrios.mexis at tenforce.com>> wrote:
>>
>>         Ok.
>>
>>         Thanks.
>>         Dimitrios
>>
>>
>>         On 10/10/2013 10:08, John Glover wrote:
>>>         Ok thanks. Could you please not run any of the steps again
>>>         for now so I can have a look at the database?
>>>
>>>         Thanks,
>>>         John
>>>
>>>
>>>         On 10 October 2013 10:06, Dimitrios Mexis
>>>         <dimitrios.mexis at tenforce.com
>>>         <mailto:dimitrios.mexis at tenforce.com>> wrote:
>>>
>>>             John no ,
>>>             no I found it "killed"
>>>
>>>             Regards
>>>             Dimitrios
>>>
>>>
>>>             On 10/10/2013 10:04, John Glover wrote:
>>>>             Hi Dimitrios,
>>>>
>>>>             Did the namespace migration step finish the last time
>>>>             you ran it or did you kill it?
>>>>
>>>>             Thanks,
>>>>             John
>>>>
>>>>
>>>>             On 10 October 2013 09:53, Dimitrios Mexis
>>>>             <dimitrios.mexis at tenforce.com
>>>>             <mailto:dimitrios.mexis at tenforce.com>> wrote:
>>>>
>>>>                 Hi Glen,
>>>>
>>>>                 Ok ight now I am feeding again the database with
>>>>                 data (pg_restore ).
>>>>                 The back up is under $USERHOME/ckan/backups called,
>>>>                 PO-release09.tgz
>>>>
>>>>                 We will try to rename this publisher that caused
>>>>                 issues before we start the procedure.
>>>>
>>>>                 regards
>>>>                 Dimitrios
>>>>
>>>>
>>>>
>>>>                 On 10/10/2013 9:46, John Glover wrote:
>>>>>                 Hi Dimitrios,
>>>>>
>>>>>                 Did you have to kill the migration step again? Can
>>>>>                 you tell me the path to the database dump on the
>>>>>                 postgres machine so I can try loading and running
>>>>>                 it myself a few times?
>>>>>
>>>>>                 Thanks,
>>>>>                 John
>>>>>
>>>>>
>>>>>                 On 10 October 2013 09:12, Dimitrios Mexis
>>>>>                 <dimitrios.mexis at tenforce.com
>>>>>                 <mailto:dimitrios.mexis at tenforce.com>> wrote:
>>>>>
>>>>>                     Good morning.
>>>>>
>>>>>                     Here can you explain the issue ? How do we
>>>>>                     de-block it ?
>>>>>
>>>>>
>>>>>                     [root at tffvirt-rh-template ~]# screen -x 25060
>>>>>                     +++ unset pydoc
>>>>>                     +++ '[' -n '' ']'
>>>>>                     +++ '[' -n '' ']'
>>>>>                     +++ '[' -n /bin/bash -o -n '' ']'
>>>>>                     +++ hash -r
>>>>>                     +++ '[' -n '' ']'
>>>>>                     +++ unset VIRTUAL_ENV
>>>>>                     +++ '[' '!' nondestructive = nondestructive ']'
>>>>>                     +++
>>>>>                     VIRTUAL_ENV=/applications/ecodp/users/ecodp/ckan/ecportal
>>>>>                     +++ export VIRTUAL_ENV
>>>>>                     +++
>>>>>                     _OLD_VIRTUAL_PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
>>>>>                     +++
>>>>>                     PATH=/applications/ecodp/users/ecodp/ckan/ecportal/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
>>>>>                     +++ export PATH
>>>>>                     +++ '[' -n '' ']'
>>>>>                     +++ '[' -z '' ']'
>>>>>                     +++ _OLD_VIRTUAL_PS1=
>>>>>                     +++ '[' x '!=' x ']'
>>>>>                     ++++ basename
>>>>>                     /applications/ecodp/users/ecodp/ckan/ecportal
>>>>>                     +++ '[' ecportal = __ ']'
>>>>>                     ++++ basename
>>>>>                     /applications/ecodp/users/ecodp/ckan/ecportal
>>>>>                     +++ PS1='(ecportal)'
>>>>>                     +++ export PS1
>>>>>                     +++ alias 'pydoc=python -m pydoc'
>>>>>                     +++ '[' -n /bin/bash -o -n '' ']'
>>>>>                     +++ hash -r
>>>>>                     ++ cd
>>>>>                     /applications/ecodp/users/ecodp//ckan/ecportal/src/ckan
>>>>>                     + set -x
>>>>>                     + paster --plugin=ckan search-index rebuild -r
>>>>>                     -c
>>>>>                     /applications/ecodp/users/ecodp//ckan/conf/ecportal.ini
>>>>>                     2013-10-09 12:40:58,527 INFO [ckan.lib.search]
>>>>>                     Rebuilding search index...
>>>>>                     2013-10-09 12:40:58,946 INFO [ckan.lib.search]
>>>>>                     Rebuilding the whole index...
>>>>>                     2013-10-09 12:59:56,800 INFO [ckan.lib.search]
>>>>>                     Finished rebuilding search index.
>>>>>                     2013-10-09 12:59:59,174 INFO [ckan.lib.search]
>>>>>                     Commited pending changes on the search index
>>>>>                     + paster --plugin=ckanext-ecportal ecportal
>>>>>                     migrate-odp-namespace -c
>>>>>                     /applications/ecodp/users/ecodp//ckan/conf/ecportal.ini
>>>>>                     ./ckan_upgrade_09to01.sh: line 21:  5031
>>>>>                     Killed paster --plugin=ckanext-ecportal
>>>>>                     ecportal migrate-odp-namespace -c $CKAN_INI
>>>>>                     + paster --plugin=ckanext-ecportal ecportal
>>>>>                     update-all-vocabs -c
>>>>>                     /applications/ecodp/users/ecodp//ckan/conf/ecportal.ini
>>>>>                     + paster --plugin=ckanext-ecportal ecportal
>>>>>                     update-publishers -c
>>>>>                     /applications/ecodp/users/ecodp//ckan/conf/ecportal.ini
>>>>>                     Traceback (most recent call last):
>>>>>                       File
>>>>>                     "/applications/ecodp/users/ecodp/ckan/ecportal/bin/paster",
>>>>>                     line 9, in <module>
>>>>>                     load_entry_point('PasteScript==1.7.5',
>>>>>                     'console_scripts', 'paster')()
>>>>>                       File
>>>>>                     "/applications/ecodp/users/ecodp/ckan/ecportal/lib/python2.6/site-packages/paste/script/command.py",
>>>>>                     line 104, in run
>>>>>                     invoke(command, command_name, options, args[1:])
>>>>>                       File
>>>>>                     "/applications/ecodp/users/ecodp/ckan/ecportal/lib/python2.6/site-packages/paste/script/command.py",
>>>>>                     line 143, in invoke
>>>>>                         exit_code = runner.run(args)
>>>>>                       File
>>>>>                     "/applications/ecodp/users/ecodp/ckan/ecportal/lib/python2.6/site-packages/paste/script/command.py",
>>>>>                     line 238, in run
>>>>>                         result = self.command()
>>>>>                       File
>>>>>                     "/applications/ecodp/users/ecodp/ckan/ecportal/src/ckanext-ecportal/ckanext/ecportal/commands.py",
>>>>>                     line 109, in command
>>>>>                     self.update_publishers(file_path)
>>>>>                       File
>>>>>                     "/applications/ecodp/users/ecodp/ckan/ecportal/src/ckanext-ecportal/ckanext/ecportal/commands.py",
>>>>>                     line 527, in update_publishers
>>>>>                     self._add_publishers(new_publishers)
>>>>>                       File
>>>>>                     "/applications/ecodp/users/ecodp/ckan/ecportal/src/ckanext-ecportal/ckanext/ecportal/commands.py",
>>>>>                     line 213, in _add_publishers
>>>>>                     plugins.toolkit.get_action('group_create')(context,
>>>>>                     group)
>>>>>                       File
>>>>>                     "/applications/ecodp/users/ecodp/ckan/ecportal/src/ckan/ckan/logic/action/create.py",
>>>>>                     line 538, in group_create
>>>>>                         raise ValidationError(errors)
>>>>>                     ckan.logic.ValidationError: {'Name': 'Group
>>>>>                     name already exists in database'}
>>>>>                     + paster --plugin=ckanext-ecportal ecportal
>>>>>                     import-csv-translations -c
>>>>>                     /applications/ecodp/users/ecodp//ckan/conf/ecportal.ini
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.okfn.org/mailman/private/ecodp-dev/attachments/20131010/fa151af6/attachment.html>


More information about the ecodp-dev mailing list