[ECODP-dev] PRODUCTION 09 upgrade

Thu Oct 10 11:05:37 UTC 2013

Hi Dimitrios,

Yes, that paragraph can remain largely unchanged (I have added a note to
reindex again after the migration).

Those paster commands to purge old data are not specifically for this
upgrade: the first one should have been run after release 09 was installed
(and the old archiver/datastorer/qa extensions removed), the second should
be run regularly by CRON and so there should not be so much data in that
table at any given time (I am making a note about this elsewhere however as
it is not in your list of CRON task in the Operations Manual).

Regards,
John

On 10 October 2013 12:39, Dimitrios Mexis <dimitrios.mexis at tenforce.com>wrote:

>  Hi John
> good to know that you made it.
>
> Can you check the paragraph of the document in google that refers to the
> upgrade of 09 version, with what ever needs to be done in the end ?
>
>
> Thanks
> Dimitrios
>
>
> On 10/10/2013 11:52, John Glover wrote:
>
> Hi Dimitrios,
>
>  I have successfully ran through the namespace migration without any
> problems. I have reindexed the site and it is now back up.
>
>  First however, I made sure to run the following commands (as documented
> previously in relation to the database size):
>
>  paster --plugin=ckanext-ecportal ecportal purge-task-data -c <config>
>  paster --plugin=ckanext-ecportal ecportal purge-package-extra-revision
> -c <config>
>
>  I believe that the second command in particular is responsible for the
> speed up in the migration (for me it took just a couple of minutes). The
> reason is that we have to update the metadata for all datasets (and all of
> their revisions) in the database, and as no one had cleared out the old
> entries from the package_extra_revision table, it was performing individual
> update statements over about 2 million records (instead of the 60 thousand
> that are there now).
>
>  Regards,
> John
>
>
> On 10 October 2013 10:12, Dimitrios Mexis <dimitrios.mexis at tenforce.com>wrote:
>
>>  Ok.
>>
>> Thanks.
>> Dimitrios
>>
>>
>> On 10/10/2013 10:08, John Glover wrote:
>>
>> Ok thanks. Could you please not run any of the steps again for now so I
>> can have a look at the database?
>>
>>  Thanks,
>> John
>>
>>
>>  On 10 October 2013 10:06, Dimitrios Mexis <dimitrios.mexis at tenforce.com>wrote:
>>
>>>  John no ,
>>> no I found it "killed"
>>>
>>> Regards
>>> Dimitrios
>>>
>>>
>>> On 10/10/2013 10:04, John Glover wrote:
>>>
>>> Hi Dimitrios,
>>>
>>>  Did the namespace migration step finish the last time you ran it or
>>> did you kill it?
>>>
>>>  Thanks,
>>> John
>>>
>>>
>>> On 10 October 2013 09:53, Dimitrios Mexis <dimitrios.mexis at tenforce.com>wrote:
>>>
>>>>  Hi Glen,
>>>>
>>>> Ok ight now I am feeding again the database with data (pg_restore ).
>>>> The back up is under $USERHOME/ckan/backups called, PO-release09.tgz
>>>>
>>>> We will try to rename this publisher that caused issues before we start
>>>> the procedure.
>>>>
>>>> regards
>>>> Dimitrios
>>>>
>>>>
>>>>
>>>> On 10/10/2013 9:46, John Glover wrote:
>>>>
>>>> Hi Dimitrios,
>>>>
>>>>  Did you have to kill the migration step again? Can you tell me the
>>>> path to the database dump on the postgres machine so I can try loading and
>>>> running it myself a few times?
>>>>
>>>>  Thanks,
>>>> John
>>>>
>>>>
>>>> On 10 October 2013 09:12, Dimitrios Mexis <dimitrios.mexis at tenforce.com
>>>> > wrote:
>>>>
>>>>>  Good morning.
>>>>>
>>>>> Here can you explain the issue ? How do we de-block it ?
>>>>>
>>>>>
>>>>> [root at tffvirt-rh-template ~]# screen -x 25060
>>>>> +++ unset pydoc
>>>>> +++ '[' -n '' ']'
>>>>> +++ '[' -n '' ']'
>>>>> +++ '[' -n /bin/bash -o -n '' ']'
>>>>> +++ hash -r
>>>>> +++ '[' -n '' ']'
>>>>> +++ unset VIRTUAL_ENV
>>>>> +++ '[' '!' nondestructive = nondestructive ']'
>>>>> +++ VIRTUAL_ENV=/applications/ecodp/users/ecodp/ckan/ecportal
>>>>> +++ export VIRTUAL_ENV
>>>>> +++
>>>>> _OLD_VIRTUAL_PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
>>>>> +++
>>>>> PATH=/applications/ecodp/users/ecodp/ckan/ecportal/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
>>>>> +++ export PATH
>>>>> +++ '[' -n '' ']'
>>>>> +++ '[' -z '' ']'
>>>>> +++ _OLD_VIRTUAL_PS1=
>>>>> +++ '[' x '!=' x ']'
>>>>> ++++ basename /applications/ecodp/users/ecodp/ckan/ecportal
>>>>> +++ '[' ecportal = __ ']'
>>>>> ++++ basename /applications/ecodp/users/ecodp/ckan/ecportal
>>>>> +++ PS1='(ecportal)'
>>>>> +++ export PS1
>>>>> +++ alias 'pydoc=python -m pydoc'
>>>>> +++ '[' -n /bin/bash -o -n '' ']'
>>>>> +++ hash -r
>>>>> ++ cd /applications/ecodp/users/ecodp//ckan/ecportal/src/ckan
>>>>> + set -x
>>>>> + paster --plugin=ckan search-index rebuild -r -c
>>>>> /applications/ecodp/users/ecodp//ckan/conf/ecportal.ini
>>>>> 2013-10-09 12:40:58,527 INFO  [ckan.lib.search] Rebuilding search
>>>>> index...
>>>>> 2013-10-09 12:40:58,946 INFO  [ckan.lib.search] Rebuilding the whole
>>>>> index...
>>>>> 2013-10-09 12:59:56,800 INFO  [ckan.lib.search] Finished rebuilding
>>>>> search index.
>>>>> 2013-10-09 12:59:59,174 INFO  [ckan.lib.search] Commited pending
>>>>> changes on the search index
>>>>> + paster --plugin=ckanext-ecportal ecportal migrate-odp-namespace -c
>>>>> /applications/ecodp/users/ecodp//ckan/conf/ecportal.ini
>>>>> ./ckan_upgrade_09to01.sh: line 21:  5031 Killed
>>>>> paster --plugin=ckanext-ecportal ecportal migrate-odp-namespace -c $CKAN_INI
>>>>> + paster --plugin=ckanext-ecportal ecportal update-all-vocabs -c
>>>>> /applications/ecodp/users/ecodp//ckan/conf/ecportal.ini
>>>>> + paster --plugin=ckanext-ecportal ecportal update-publishers -c
>>>>> /applications/ecodp/users/ecodp//ckan/conf/ecportal.ini
>>>>> Traceback (most recent call last):
>>>>>   File "/applications/ecodp/users/ecodp/ckan/ecportal/bin/paster",
>>>>> line 9, in <module>
>>>>>     load_entry_point('PasteScript==1.7.5', 'console_scripts',
>>>>> 'paster')()
>>>>>   File
>>>>> "/applications/ecodp/users/ecodp/ckan/ecportal/lib/python2.6/site-packages/paste/script/command.py",
>>>>> line 104, in run
>>>>>     invoke(command, command_name, options, args[1:])
>>>>>   File
>>>>> "/applications/ecodp/users/ecodp/ckan/ecportal/lib/python2.6/site-packages/paste/script/command.py",
>>>>> line 143, in invoke
>>>>>     exit_code = runner.run(args)
>>>>>   File
>>>>> "/applications/ecodp/users/ecodp/ckan/ecportal/lib/python2.6/site-packages/paste/script/command.py",
>>>>> line 238, in run
>>>>>     result = self.command()
>>>>>   File
>>>>> "/applications/ecodp/users/ecodp/ckan/ecportal/src/ckanext-ecportal/ckanext/ecportal/commands.py",
>>>>> line 109, in command
>>>>>     self.update_publishers(file_path)
>>>>>   File
>>>>> "/applications/ecodp/users/ecodp/ckan/ecportal/src/ckanext-ecportal/ckanext/ecportal/commands.py",
>>>>> line 527, in update_publishers
>>>>>     self._add_publishers(new_publishers)
>>>>>   File
>>>>> "/applications/ecodp/users/ecodp/ckan/ecportal/src/ckanext-ecportal/ckanext/ecportal/commands.py",
>>>>> line 213, in _add_publishers
>>>>>     plugins.toolkit.get_action('group_create')(context, group)
>>>>>   File
>>>>> "/applications/ecodp/users/ecodp/ckan/ecportal/src/ckan/ckan/logic/action/create.py",
>>>>> line 538, in group_create
>>>>>     raise ValidationError(errors)
>>>>> ckan.logic.ValidationError: {'Name': 'Group name already exists in
>>>>> database'}
>>>>> + paster --plugin=ckanext-ecportal ecportal import-csv-translations -c
>>>>> /applications/ecodp/users/ecodp//ckan/conf/ecportal.ini
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.okfn.org/mailman/private/ecodp-dev/attachments/20131010/3277f2b2/attachment.html>