[ECODP-dev] CKAN release notes for 09.00

Mon Jun 17 21:28:41 UTC 2013

Hi John,

The goal is to hand over the work to PO in the end.
We already handed over the update of the coorporate body NAL since dec 2012.
So we must ensure that this functionality keeps on working. So is
there a path dependency for these files?
Do they have to be in a specific location?
If not, can we then move them to the proper rpm location /ckan/data
and configure CKAN so that they are found for the upload procedure.

For the enlargement of the handover: REST calls are not the best
"operational" way. The way through a prepared file like the json's and
then trigger an update of the system is much easier to explain and to
manage for them. It is closer to their way of working and it allows to
transfer data from test to production is a clean way.
It works for the coorporate body NAL. Does it hence work to for the others?

I hope I have clarified it sufficiently now.

kind regards,

Bert

2013/6/17 John Glover <john.glover at okfn.org>:
> Hi Bert,
>
> Yes I would not expect the live database to be upgraded in place, and the
> procedure that you describe sounds fine to me.
>
>> => how is that related to the json's I pointed you in the /ckan/data
>> directory. Does the process imports them from there? If not, how do we
>> configure CKAN to take the data from there.
>
> I don't follow your question about the vocabularies. I mentioned in a
> previous email that these can be updated using our API, copied again here
> for reference:
>
> Those JSON files were not intended to be the official way of updating the
> vocabularies. We use them to do the initial database setup, but I did not
> even know that you had your own copy of them in the RPM. However, as all of
> this information is stored in the database, it can be updated using the API.
>
> To view or update vocabularies, see:
> http://docs.ckan.org/en/ckan-1.8/apiv3.html#ckan.logic.action.get.vocabulary_list
> http://docs.ckan.org/en/ckan-1.8/apiv3.html#ckan.logic.action.get.vocabulary_show
> http://docs.ckan.org/en/ckan-1.8/apiv3.html#ckan.logic.action.create.vocabulary_create
> http://docs.ckan.org/en/ckan-1.8/apiv3.html#ckan.logic.action.update.vocabulary_update
>
> To view or update translations, see:
> http://docs.ckan.org/en/ckan-1.8/apiv3.html#ckan.logic.action.get.term_translation_show
> http://docs.ckan.org/en/ckan-1.8/apiv3.html#ckan.logic.action.update.term_translation_update_many
>
>> => For the Operations Manual. How does PO can update the translations
>> themselves?
>> does this affects the management of the publishers with their
>> translations?
>
> I am also not sure what you mean here, are you asking how the PO can update
> the vocabulary translations? This would be the same as the answer above,
> using the API.
>
> Regards,
> John
>
>
> On 13 June 2013 23:26, Bert Van Nuffelen <bert.van.nuffelen at tenforce.com>
> wrote:
>>
>> Hi john,
>>
>> thanks for the release notes and the associated instructions.
>>
>> Some remarks.
>> The CKAN database update will not happen on the "life production
>> database".
>> As this is a complete reinstall from zero (because of the rpm
>> packaging) the process to support migration is:
>>    * install a release 00.09.00 in test
>>    * migrate the production db (version v.1.7) to the release 00.09.00 in
>> test
>>    * upgrade the content to the v.1.8
>>    => this process will be repeated for a new production system
>> whenever PO accepts this release and migration path.
>>    So if you see any issues here, let us know.
>>
>> On the translations for the vocabularies.
>> => how is that related to the json's I pointed you in the /ckan/data
>> directory. Does the process imports them from there? If not, how do we
>> configure CKAN to take the data from there.
>> => For the Operations Manual. How does PO can update the translations
>> themselves?
>>
>> does this affects the management of the publishers with their
>> translations?
>>
>> FYI, the EU ODP system is installed on 3 machines so "copying" cannot be
>> done.
>>
>> Kind regards,
>>
>> Bert
>>
>>
>>
>> 2013/6/13 John Glover <john.glover at okfn.org>:
>> > Hello,
>> >
>> > I am currently writing the release notes for release 09.00. The latest
>> > draft
>> > of the setup instructions is given below, it would be good if you could
>> > verify that this process works on your acceptance server (starting with
>> > the
>> > CKAN 1.7 package and database). I am still working on the changelog so
>> > you
>> > should get the full version early next week.
>> >
>> > Let me know of any problems.
>> >
>> > Regards,
>> > John
>> >
>> > -----
>> >
>> > Release 09.00
>> > =============
>> >
>> > Unlike previous releases, the RPM is created by TenForce.
>> > Nothing has been added to the odp Git repository by the OKF.
>> >
>> > The CKAN codebase is available at:
>> > https://github.com/okfn/ckan
>> > branch: release-v1.8.1-ecportal
>> > commit hash: 1143f5c074dfffdd1409da289a609e7cf323567d
>> >
>> > The CKAN EC Portal Extension is available at:
>> > https://github.com/okfn/ckanext-ecportal
>> > branch: master
>> > commit hash: e3b7f35df98baec1412fa584ec8227e9d404ad3f
>> >
>> > Note: As this is only a draft of the release notes, both of thesse
>> > commit hashes may change.
>> >
>> >
>> > Installation
>> > ============
>> >
>> > * Install the new RPM.
>> >
>> > * Upgrade the database for CKAN 1.8.
>> >
>> > /applications/ecodp/users/ecodp/ckan/ecportal/bin/paster --plugin=ckan
>> > db
>> > upgrade -c /applications/ecodp/users/ecodp/ckan/conf/ecportal.ini
>> >
>> > This command should return:
>> > <date> INFO  [ckan.model] CKAN database version upgraded: 57 -> 59
>> >
>> > * Add the database tables for the new search cloud feature.
>> >
>> > /applications/ecodp/users/ecodp/ckan/ecportal/bin/paster
>> > --plugin=ckanext-ecportal ecportal searchcloud-install-tables -c
>> > /applications/ecodp/users/ecodp/ckan/conf/ecportal.ini
>> >
>> > This command should return:
>> > Creating the search_query table ...
>> > done.
>> > Creating the search_popular_latest table ...
>> > done.
>> > Creating the search_popular_approved table ...
>> > done.
>> > Creating the search_query_date index ...
>> > done.
>> >
>> > * Import the new translations of the fixed vocabularies.
>> >
>> > /applications/ecodp/users/ecodp/ckan/ecportal/bin/paster
>> > --plugin=ckanext-ecportal ecportal import-csv-translations -c
>> > /applications/ecodp/users/ecodp/ckan/conf/ecportal.ini
>> >
>> > This command does not return anything.
>> >
>> > * Make sure that Solr is using the latest schema.xml file from the
>> > ckanext
>> > ecportal extension.
>> > This is at:
>> >
>> > /applications/ecodp/users/ecodp/ckan/ecportal/src/ckanext-ecportal/ckanext/ecportal/solr
>> > This can be copied to the solr conf directory:
>> >
>> > cp
>> >
>> > /applications/ecodp/users/ecodp/ckan/ecportal/src/ckanext-ecportal/ckanext/ecportal/solr/*
>> > /applications/ecodp/users/ecodp/solr/solr/conf
>> >
>> > * Restart tomcat.
>> >
>> > * Restart Apache.
>> >
>> > * Rebuild the search index (may take a while to run).
>> >
>> > /applications/ecodp/users/ecodp/ckan/ecportal/bin/paster --plugin=ckan
>> > search-index -r rebuild -c
>> > /applications/ecodp/users/ecodp/ckan/conf/ecportal.ini
>> >
>> >
>> > CRON jobs
>> > ---------
>> >
>> > * The following should be run to update the list of recently searched
>> > terms:
>> >
>> > /applications/ecodp/users/ecodp/ckan/ecportal/bin/paster
>> > --plugin=ckanext-ecportal ecportal
>> > searchcloud-generate-unapproved-search-list -c
>> > /applications/ecodp/users/ecodp/ckan/conf/ecportal.ini
>> >
>> > We recommend running this daily at midnight.
>> >
>> > * The following should be run to update the tracking information for the
>> > previous day:
>> >
>> > /applications/ecodp/users/ecodp/ckan/ecportal/bin/paster --plugin=ckan
>> > tracking update -c
>> > /applications/ecodp/users/ecodp/ckan/conf/ecportal.ini
>> >
>> > We recommend running this daily at 1 minute past midnight.
>> >
>> >
>> > Database cleanup
>> > ----------------
>> >
>> > When investigating the issues with the rdf import failing, we noticed
>> > that
>> > some database tables had grown quite large and can be cleaned up in this
>> > release.
>> >
>> > 1. The package_extra_revision table contains data on a lot of old
>> > revisions that are not currently needed. These can be removed with the
>> > following
>> > paster command:
>> >
>> > /applications/ecodp/users/ecodp/ckan/ecportal/bin/paster
>> > --plugin=ckanext-ecportal ecportal purge-package-extra-revision -c
>> > /applications/ecodp/users/ecodp/ckan/conf/ecportal.ini
>> >
>> > This command should return:
>> > <date> WARNI [root] {'revision_rows_deleted': <number of rows>}
>> >
>> > This table is in constant use, so this command could be scheduled to be
>> > run
>> > regularly. It should not need to be run too frequently however, once
>> > every
>> > week or two should be fine.
>> >
>> >
>> > 2. Some unused CKAN extensions (qa and datastorer) were installed in
>> > previous
>> > releases. These have been removed in this release, and so the tables
>> > that
>> > they were writing to can now be cleared. To do this, run:
>> >
>> > /applications/ecodp/users/ecodp/ckan/ecportal/bin/paster
>> > --plugin=ckanext-ecportal ecportal purge-task-data -c
>> > /applications/ecodp/users/ecodp/ckan/conf/ecportal.ini
>> >
>> > This command should return:
>> > <date> WARNI [root] {'celery_rows_deleted': <number of rows>,
>> > 'task_status_rows_deleted': <number of rows>}
>> >
>> > The tables cleared by this command should not be used at all with the
>> > code
>> > in
>> > this release, so this command will only have to be run once.
>> >
>> >
>> > _______________________________________________
>> > Ecodp-dev mailing list
>> > Ecodp-dev at lists.okfn.org
>> > http://lists.okfn.org/mailman/listinfo/ecodp-dev
>> >
>>
>>
>>
>> --
>> Bert Van Nuffelen
>>
>> Semantic Technologies Software Architect at TenForce
>> www.tenforce.be
>>
>> Bert.Van.Nuffelen at tenforce.com
>> Office: +32 (0)16 31 48 60
>> Mobile:+32 479 06 24 26
>> skype: bert.van.nuffelen
>>
>> _______________________________________________
>> Ecodp-dev mailing list
>> Ecodp-dev at lists.okfn.org
>> http://lists.okfn.org/mailman/listinfo/ecodp-dev
>
>
>
> _______________________________________________
> Ecodp-dev mailing list
> Ecodp-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ecodp-dev
>

-- 
Bert Van Nuffelen

Semantic Technologies Software Architect at TenForce
www.tenforce.be

Bert.Van.Nuffelen at tenforce.com
Office: +32 (0)16 31 48 60
Mobile:+32 479 06 24 26
skype: bert.van.nuffelen