[ECODP-dev] PRODUCTION 09 upgrade

[John Glover]

Hi Dimitrios,

I have successfully ran through the namespace migration without any
problems. I have reindexed the site and it is now back up.

First however, I made sure to run the following commands (as documented
previously in relation to the database size):

paster --plugin=ckanext-ecportal ecportal purge-task-data -c <config>
paster --plugin=ckanext-ecportal ecportal purge-package-extra-revision -c
<config>

I believe that the second command in particular is responsible for the
speed up in the migration (for me it took just a couple of minutes). The
reason is that we have to update the metadata for all datasets (and all of
their revisions) in the database, and as no one had cleared out the old
entries from the package_extra_revision table, it was performing individual
update statements over about 2 million records (instead of the 60 thousand
that are there now).

Regards,
John


On 10 October 2013 10:12, Dimitrios Mexis <dimitrios.mexis at tenforce.com>wrote:

>  Ok.
>
> Thanks.
> Dimitrios
>
>
> On 10/10/2013 10:08, John Glover wrote:
>
> Ok thanks. Could you please not run any of the steps again for now so I
> can have a look at the database?
>
>  Thanks,
> John
>
>
>  On 10 October 2013 10:06, Dimitrios Mexis <dimitrios.mexis at tenforce.com>wrote:
>
>>  John no ,
>> no I found it "killed"
>>
>> Regards
>> Dimitrios
>>
>>
>> On 10/10/2013 10:04, John Glover wrote:
>>
>> Hi Dimitrios,
>>
>>  Did the namespace migration step finish the last time you ran it or did
>> you kill it?
>>
>>  Thanks,
>> John
>>
>>
>> On 10 October 2013 09:53, Dimitrios Mexis <dimitrios.mexis at tenforce.com>wrote:
>>
>>>  Hi Glen,
>>>
>>> Ok ight now I am feeding again the database with data (pg_restore ).
>>> The back up is under $USERHOME/ckan/backups called, PO-release09.tgz
>>>
>>> We will try to rename this publisher that caused issues before we start
>>> the procedure.
>>>
>>> regards
>>> Dimitrios
>>>
>>>
>>>
>>> On 10/10/2013 9:46, John Glover wrote:
>>>
>>> Hi Dimitrios,
>>>
>>>  Did you have to kill the migration step again? Can you tell me the
>>> path to the database dump on the postgres machine so I can try loading and
>>> running it myself a few times?
>>>
>>>  Thanks,
>>> John
>>>
>>>
>>> On 10 October 2013 09:12, Dimitrios Mexis <dimitrios.mexis at tenforce.com>wrote:
>>>
>>>>  Good morning.
>>>>
>>>> Here can you explain the issue ? How do we de-block it ?
>>>>
>>>>
>>>> [root at tffvirt-rh-template ~]# screen -x 25060
>>>> +++ unset pydoc
>>>> +++ '[' -n '' ']'
>>>> +++ '[' -n '' ']'
>>>> +++ '[' -n /bin/bash -o -n '' ']'
>>>> +++ hash -r
>>>> +++ '[' -n '' ']'
>>>> +++ unset VIRTUAL_ENV
>>>> +++ '[' '!' nondestructive = nondestructive ']'
>>>> +++ VIRTUAL_ENV=/applications/ecodp/users/ecodp/ckan/ecportal
>>>> +++ export VIRTUAL_ENV
>>>> +++
>>>> _OLD_VIRTUAL_PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
>>>> +++
>>>> PATH=/applications/ecodp/users/ecodp/ckan/ecportal/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
>>>> +++ export PATH
>>>> +++ '[' -n '' ']'
>>>> +++ '[' -z '' ']'
>>>> +++ _OLD_VIRTUAL_PS1=
>>>> +++ '[' x '!=' x ']'
>>>> ++++ basename /applications/ecodp/users/ecodp/ckan/ecportal
>>>> +++ '[' ecportal = __ ']'
>>>> ++++ basename /applications/ecodp/users/ecodp/ckan/ecportal
>>>> +++ PS1='(ecportal)'
>>>> +++ export PS1
>>>> +++ alias 'pydoc=python -m pydoc'
>>>> +++ '[' -n /bin/bash -o -n '' ']'
>>>> +++ hash -r
>>>> ++ cd /applications/ecodp/users/ecodp//ckan/ecportal/src/ckan
>>>> + set -x
>>>> + paster --plugin=ckan search-index rebuild -r -c
>>>> /applications/ecodp/users/ecodp//ckan/conf/ecportal.ini
>>>> 2013-10-09 12:40:58,527 INFO  [ckan.lib.search] Rebuilding search
>>>> index...
>>>> 2013-10-09 12:40:58,946 INFO  [ckan.lib.search] Rebuilding the whole
>>>> index...
>>>> 2013-10-09 12:59:56,800 INFO  [ckan.lib.search] Finished rebuilding
>>>> search index.
>>>> 2013-10-09 12:59:59,174 INFO  [ckan.lib.search] Commited pending
>>>> changes on the search index
>>>> + paster --plugin=ckanext-ecportal ecportal migrate-odp-namespace -c
>>>> /applications/ecodp/users/ecodp//ckan/conf/ecportal.ini
>>>> ./ckan_upgrade_09to01.sh: line 21:  5031 Killed                  paster
>>>> --plugin=ckanext-ecportal ecportal migrate-odp-namespace -c $CKAN_INI
>>>> + paster --plugin=ckanext-ecportal ecportal update-all-vocabs -c
>>>> /applications/ecodp/users/ecodp//ckan/conf/ecportal.ini
>>>> + paster --plugin=ckanext-ecportal ecportal update-publishers -c
>>>> /applications/ecodp/users/ecodp//ckan/conf/ecportal.ini
>>>> Traceback (most recent call last):
>>>>   File "/applications/ecodp/users/ecodp/ckan/ecportal/bin/paster", line
>>>> 9, in <module>
>>>>     load_entry_point('PasteScript==1.7.5', 'console_scripts',
>>>> 'paster')()
>>>>   File
>>>> "/applications/ecodp/users/ecodp/ckan/ecportal/lib/python2.6/site-packages/paste/script/command.py",
>>>> line 104, in run
>>>>     invoke(command, command_name, options, args[1:])
>>>>   File
>>>> "/applications/ecodp/users/ecodp/ckan/ecportal/lib/python2.6/site-packages/paste/script/command.py",
>>>> line 143, in invoke
>>>>     exit_code = runner.run(args)
>>>>   File
>>>> "/applications/ecodp/users/ecodp/ckan/ecportal/lib/python2.6/site-packages/paste/script/command.py",
>>>> line 238, in run
>>>>     result = self.command()
>>>>   File
>>>> "/applications/ecodp/users/ecodp/ckan/ecportal/src/ckanext-ecportal/ckanext/ecportal/commands.py",
>>>> line 109, in command
>>>>     self.update_publishers(file_path)
>>>>   File
>>>> "/applications/ecodp/users/ecodp/ckan/ecportal/src/ckanext-ecportal/ckanext/ecportal/commands.py",
>>>> line 527, in update_publishers
>>>>     self._add_publishers(new_publishers)
>>>>   File
>>>> "/applications/ecodp/users/ecodp/ckan/ecportal/src/ckanext-ecportal/ckanext/ecportal/commands.py",
>>>> line 213, in _add_publishers
>>>>     plugins.toolkit.get_action('group_create')(context, group)
>>>>   File
>>>> "/applications/ecodp/users/ecodp/ckan/ecportal/src/ckan/ckan/logic/action/create.py",
>>>> line 538, in group_create
>>>>     raise ValidationError(errors)
>>>> ckan.logic.ValidationError: {'Name': 'Group name already exists in
>>>> database'}
>>>> + paster --plugin=ckanext-ecportal ecportal import-csv-translations -c
>>>> /applications/ecodp/users/ecodp//ckan/conf/ecportal.ini
>>>>
>>>
>>>
>>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.okfn.org/mailman/private/ecodp-dev/attachments/20131010/a8ee7ffd/attachment.html>