[ECODP-dev] CKAN release notes for 09.00

Wed Jun 19 08:39:17 UTC 2013

Hello,

Updated release notes (with new commit hashes and a changelog) are given
below.

Regards,
John

----

Release 09.00
=============

Unlike previous releases, the RPM is created by TenForce.
Nothing has been added to the odp Git repository by the OKF.

The CKAN codebase is available at:
https://github.com/okfn/ckan
branch: release-v1.8.1-ecportal
commit hash: 39986dec52ba01b9ce0489c7961bb23d32895a9a

The CKAN EC Portal Extension is available at:
https://github.com/okfn/ckanext-ecportal
branch: master
commit hash: 7d0293d07ac837d4402418a68f8feced04d58738

Note: As this is only a draft of the release notes, both of thesse
commit hashes may change.

Installation
============

* Install the new RPM.

* Upgrade the database for CKAN 1.8.

/applications/ecodp/users/ecodp/ckan/ecportal/bin/paster --plugin=ckan db
upgrade -c /applications/ecodp/users/ecodp/ckan/conf/ecportal.ini

This command should return:
<date> INFO  [ckan.model] CKAN database version upgraded: 57 -> 59

* Add the database tables for the new search cloud feature.

/applications/ecodp/users/ecodp/ckan/ecportal/bin/paster
--plugin=ckanext-ecportal ecportal searchcloud-install-tables -c
/applications/ecodp/users/ecodp/ckan/conf/ecportal.ini

This command should return:
Creating the search_query table ...
done.
Creating the search_popular_latest table ...
done.
Creating the search_popular_approved table ...
done.
Creating the search_query_date index ...
done.

* Import the new translations of the fixed vocabularies.

/applications/ecodp/users/ecodp/ckan/ecportal/bin/paster
--plugin=ckanext-ecportal ecportal import-csv-translations -c
/applications/ecodp/users/ecodp/ckan/conf/ecportal.ini

This command does not return anything.

* Make sure that Solr is using the latest schema.xml file from the ckanext
ecportal extension.
This is at:
/applications/ecodp/users/ecodp/ckan/ecportal/src/ckanext-ecportal/ckanext/ecportal/solr
This can be copied to the solr conf directory:

cp
/applications/ecodp/users/ecodp/ckan/ecportal/src/ckanext-ecportal/ckanext/ecportal/solr/*
/applications/ecodp/users/ecodp/solr/solr/conf

* Restart tomcat.

* Restart Apache.

* Rebuild the search index.
/applications/ecodp/users/ecodp/ckan/ecportal/bin/paster --plugin=ckan
search-index -r rebuild -c
/applications/ecodp/users/ecodp/ckan/conf/ecportal.ini

This command should return:
<date> INFO  [ckan.lib.search] Rebuilding search index...
<date> INFO  [ckan.lib.search] Rebuilding the whole index...
<date> INFO  [ckan.lib.search] Finished rebuilding search index.
<date> INFO  [ckan.lib.search] Commited pending changes on the search index

CRON jobs
---------

* The following should be run to update the list of recently searched terms:

/applications/ecodp/users/ecodp/ckan/ecportal/bin/paster
--plugin=ckanext-ecportal ecportal
searchcloud-generate-unapproved-search-list -c
/applications/ecodp/users/ecodp/ckan/conf/ecportal.ini

We recommend running this daily at midnight.

* The following should be run to update the tracking information for the
previous day:

/applications/ecodp/users/ecodp/ckan/ecportal/bin/paster --plugin=ckan
tracking update -c /applications/ecodp/users/ecodp/ckan/conf/ecportal.ini

We recommend running this daily at 1 minute past midnight.

Database cleanup
----------------

When investigating the issues with the rdf import failing, we noticed that
some database tables had grown quite large and can be cleaned up in this
release.

1. The package_extra_revision table contains data on a lot of old
revisions that are not currently needed. These can be removed with the
following
paster command:

/applications/ecodp/users/ecodp/ckan/ecportal/bin/paster
--plugin=ckanext-ecportal ecportal purge-package-extra-revision -c
/applications/ecodp/users/ecodp/ckan/conf/ecportal.ini

This command should return:
<date> WARNI [root] {'revision_rows_deleted': <number of rows>}

This table is in constant use, so this command could be scheduled to be run
regularly. It should not need to be run too frequently however, once every
week or two should be fine.

2. Some unused CKAN extensions (qa and datastorer) were installed in
previous
releases. These have been removed in this release, and so the tables that
they were writing to can now be cleared. To do this, run:

/applications/ecodp/users/ecodp/ckan/ecportal/bin/paster
--plugin=ckanext-ecportal ecportal purge-task-data -c
/applications/ecodp/users/ecodp/ckan/conf/ecportal.ini

This command should return:
<date> WARNI [root] {'celery_rows_deleted': <number of rows>,
'task_status_rows_deleted': <number of rows>}

The tables cleared by this command should not be used at all with the code
in
this release, so this command will only have to be run once.

Changelog
=========

CKAN Core
---------

* Upgraded to CKAN version 1.8.1.

* Allow search operator behavior to change between matching all terms, any
terms or
  exact phrase.

Security fixes:

* Upgraded repoze.who to version 2.1 so cookies can be marked as "httponly"
and "secure".

* Allow beaker session to be used to store login information, so a user's
session can
  be invalidated as soon as they click "logout".

Bug fixes and performance improvements:

* Speed up search indexing.

* Fixed bug where storage controller and rdf-export expected a different
value to
  be stored in ckan.site_url (one expected the /data root to be present,
  one did not).

CKAN EC Portal Extension
------------------------

* Upgraded extension to work with CKAN 1.8.x.

* Improved sort order for non-ASCII characters in search results.

* Search queries are now logged.

* Add searchcloud feature to home page (cloud of popular search terms).

* Show most viewed datasets (with view counts) on home page.

* Add support for changing the search operator (AND, OR, exact phrase
matching).

* Add "Suggest a dataset" box.

* Add support for additional languages.

* Updated translations (UI and fixed vocabularies).

* Only list publishers that have released datasets (even for logged-in
users) at /publisher.

* Update header logos.

* Update metadata labels on dataset page.

* Some "Additional Information" fields are no longer shown on the resource
page.

* Removed (unused) Javascript relating to DataExplorer from resource page.

* Added paster commands for removing old items from the
package_extra_revision,
  kombu_messages and task_status tables.

Solr
----

* Added a new schema in order to support additional languages.

Other
-----

The follow extensions were removed (as they are not currently used):

* ckanext-archiver
* ckanext-datastorer
* ckanext-qa

On 17 June 2013 23:28, Bert Van Nuffelen <bert.van.nuffelen at tenforce.com>wrote:

> Hi John,
>
> The goal is to hand over the work to PO in the end.
> We already handed over the update of the coorporate body NAL since dec
> 2012.
> So we must ensure that this functionality keeps on working. So is
> there a path dependency for these files?
> Do they have to be in a specific location?
> If not, can we then move them to the proper rpm location /ckan/data
> and configure CKAN so that they are found for the upload procedure.
>
> For the enlargement of the handover: REST calls are not the best
> "operational" way. The way through a prepared file like the json's and
> then trigger an update of the system is much easier to explain and to
> manage for them. It is closer to their way of working and it allows to
> transfer data from test to production is a clean way.
> It works for the coorporate body NAL. Does it hence work to for the others?
>
> I hope I have clarified it sufficiently now.
>
> kind regards,
>
> Bert
>
>
>
> 2013/6/17 John Glover <john.glover at okfn.org>:
> > Hi Bert,
> >
> > Yes I would not expect the live database to be upgraded in place, and the
> > procedure that you describe sounds fine to me.
> >
> >> => how is that related to the json's I pointed you in the /ckan/data
> >> directory. Does the process imports them from there? If not, how do we
> >> configure CKAN to take the data from there.
> >
> > I don't follow your question about the vocabularies. I mentioned in a
> > previous email that these can be updated using our API, copied again here
> > for reference:
> >
> > Those JSON files were not intended to be the official way of updating the
> > vocabularies. We use them to do the initial database setup, but I did not
> > even know that you had your own copy of them in the RPM. However, as all
> of
> > this information is stored in the database, it can be updated using the
> API.
> >
> > To view or update vocabularies, see:
> >
> http://docs.ckan.org/en/ckan-1.8/apiv3.html#ckan.logic.action.get.vocabulary_list
> >
> http://docs.ckan.org/en/ckan-1.8/apiv3.html#ckan.logic.action.get.vocabulary_show
> >
> http://docs.ckan.org/en/ckan-1.8/apiv3.html#ckan.logic.action.create.vocabulary_create
> >
> http://docs.ckan.org/en/ckan-1.8/apiv3.html#ckan.logic.action.update.vocabulary_update
> >
> > To view or update translations, see:
> >
> http://docs.ckan.org/en/ckan-1.8/apiv3.html#ckan.logic.action.get.term_translation_show
> >
> http://docs.ckan.org/en/ckan-1.8/apiv3.html#ckan.logic.action.update.term_translation_update_many
> >
> >> => For the Operations Manual. How does PO can update the translations
> >> themselves?
> >> does this affects the management of the publishers with their
> >> translations?
> >
> > I am also not sure what you mean here, are you asking how the PO can
> update
> > the vocabulary translations? This would be the same as the answer above,
> > using the API.
> >
> > Regards,
> > John
> >
> >
> > On 13 June 2013 23:26, Bert Van Nuffelen <bert.van.nuffelen at tenforce.com
> >
> > wrote:
> >>
> >> Hi john,
> >>
> >> thanks for the release notes and the associated instructions.
> >>
> >> Some remarks.
> >> The CKAN database update will not happen on the "life production
> >> database".
> >> As this is a complete reinstall from zero (because of the rpm
> >> packaging) the process to support migration is:
> >>    * install a release 00.09.00 in test
> >>    * migrate the production db (version v.1.7) to the release 00.09.00
> in
> >> test
> >>    * upgrade the content to the v.1.8
> >>    => this process will be repeated for a new production system
> >> whenever PO accepts this release and migration path.
> >>    So if you see any issues here, let us know.
> >>
> >> On the translations for the vocabularies.
> >> => how is that related to the json's I pointed you in the /ckan/data
> >> directory. Does the process imports them from there? If not, how do we
> >> configure CKAN to take the data from there.
> >> => For the Operations Manual. How does PO can update the translations
> >> themselves?
> >>
> >> does this affects the management of the publishers with their
> >> translations?
> >>
> >> FYI, the EU ODP system is installed on 3 machines so "copying" cannot be
> >> done.
> >>
> >> Kind regards,
> >>
> >> Bert
> >>
> >>
> >>
> >> 2013/6/13 John Glover <john.glover at okfn.org>:
> >> > Hello,
> >> >
> >> > I am currently writing the release notes for release 09.00. The latest
> >> > draft
> >> > of the setup instructions is given below, it would be good if you
> could
> >> > verify that this process works on your acceptance server (starting
> with
> >> > the
> >> > CKAN 1.7 package and database). I am still working on the changelog so
> >> > you
> >> > should get the full version early next week.
> >> >
> >> > Let me know of any problems.
> >> >
> >> > Regards,
> >> > John
> >> >
> >> > -----
> >> >
> >> > Release 09.00
> >> > =============
> >> >
> >> > Unlike previous releases, the RPM is created by TenForce.
> >> > Nothing has been added to the odp Git repository by the OKF.
> >> >
> >> > The CKAN codebase is available at:
> >> > https://github.com/okfn/ckan
> >> > branch: release-v1.8.1-ecportal
> >> > commit hash: 1143f5c074dfffdd1409da289a609e7cf323567d
> >> >
> >> > The CKAN EC Portal Extension is available at:
> >> > https://github.com/okfn/ckanext-ecportal
> >> > branch: master
> >> > commit hash: e3b7f35df98baec1412fa584ec8227e9d404ad3f
> >> >
> >> > Note: As this is only a draft of the release notes, both of thesse
> >> > commit hashes may change.
> >> >
> >> >
> >> > Installation
> >> > ============
> >> >
> >> > * Install the new RPM.
> >> >
> >> > * Upgrade the database for CKAN 1.8.
> >> >
> >> > /applications/ecodp/users/ecodp/ckan/ecportal/bin/paster --plugin=ckan
> >> > db
> >> > upgrade -c /applications/ecodp/users/ecodp/ckan/conf/ecportal.ini
> >> >
> >> > This command should return:
> >> > <date> INFO  [ckan.model] CKAN database version upgraded: 57 -> 59
> >> >
> >> > * Add the database tables for the new search cloud feature.
> >> >
> >> > /applications/ecodp/users/ecodp/ckan/ecportal/bin/paster
> >> > --plugin=ckanext-ecportal ecportal searchcloud-install-tables -c
> >> > /applications/ecodp/users/ecodp/ckan/conf/ecportal.ini
> >> >
> >> > This command should return:
> >> > Creating the search_query table ...
> >> > done.
> >> > Creating the search_popular_latest table ...
> >> > done.
> >> > Creating the search_popular_approved table ...
> >> > done.
> >> > Creating the search_query_date index ...
> >> > done.
> >> >
> >> > * Import the new translations of the fixed vocabularies.
> >> >
> >> > /applications/ecodp/users/ecodp/ckan/ecportal/bin/paster
> >> > --plugin=ckanext-ecportal ecportal import-csv-translations -c
> >> > /applications/ecodp/users/ecodp/ckan/conf/ecportal.ini
> >> >
> >> > This command does not return anything.
> >> >
> >> > * Make sure that Solr is using the latest schema.xml file from the
> >> > ckanext
> >> > ecportal extension.
> >> > This is at:
> >> >
> >> >
> /applications/ecodp/users/ecodp/ckan/ecportal/src/ckanext-ecportal/ckanext/ecportal/solr
> >> > This can be copied to the solr conf directory:
> >> >
> >> > cp
> >> >
> >> >
> /applications/ecodp/users/ecodp/ckan/ecportal/src/ckanext-ecportal/ckanext/ecportal/solr/*
> >> > /applications/ecodp/users/ecodp/solr/solr/conf
> >> >
> >> > * Restart tomcat.
> >> >
> >> > * Restart Apache.
> >> >
> >> > * Rebuild the search index (may take a while to run).
> >> >
> >> > /applications/ecodp/users/ecodp/ckan/ecportal/bin/paster --plugin=ckan
> >> > search-index -r rebuild -c
> >> > /applications/ecodp/users/ecodp/ckan/conf/ecportal.ini
> >> >
> >> >
> >> > CRON jobs
> >> > ---------
> >> >
> >> > * The following should be run to update the list of recently searched
> >> > terms:
> >> >
> >> > /applications/ecodp/users/ecodp/ckan/ecportal/bin/paster
> >> > --plugin=ckanext-ecportal ecportal
> >> > searchcloud-generate-unapproved-search-list -c
> >> > /applications/ecodp/users/ecodp/ckan/conf/ecportal.ini
> >> >
> >> > We recommend running this daily at midnight.
> >> >
> >> > * The following should be run to update the tracking information for
> the
> >> > previous day:
> >> >
> >> > /applications/ecodp/users/ecodp/ckan/ecportal/bin/paster --plugin=ckan
> >> > tracking update -c
> >> > /applications/ecodp/users/ecodp/ckan/conf/ecportal.ini
> >> >
> >> > We recommend running this daily at 1 minute past midnight.
> >> >
> >> >
> >> > Database cleanup
> >> > ----------------
> >> >
> >> > When investigating the issues with the rdf import failing, we noticed
> >> > that
> >> > some database tables had grown quite large and can be cleaned up in
> this
> >> > release.
> >> >
> >> > 1. The package_extra_revision table contains data on a lot of old
> >> > revisions that are not currently needed. These can be removed with the
> >> > following
> >> > paster command:
> >> >
> >> > /applications/ecodp/users/ecodp/ckan/ecportal/bin/paster
> >> > --plugin=ckanext-ecportal ecportal purge-package-extra-revision -c
> >> > /applications/ecodp/users/ecodp/ckan/conf/ecportal.ini
> >> >
> >> > This command should return:
> >> > <date> WARNI [root] {'revision_rows_deleted': <number of rows>}
> >> >
> >> > This table is in constant use, so this command could be scheduled to
> be
> >> > run
> >> > regularly. It should not need to be run too frequently however, once
> >> > every
> >> > week or two should be fine.
> >> >
> >> >
> >> > 2. Some unused CKAN extensions (qa and datastorer) were installed in
> >> > previous
> >> > releases. These have been removed in this release, and so the tables
> >> > that
> >> > they were writing to can now be cleared. To do this, run:
> >> >
> >> > /applications/ecodp/users/ecodp/ckan/ecportal/bin/paster
> >> > --plugin=ckanext-ecportal ecportal purge-task-data -c
> >> > /applications/ecodp/users/ecodp/ckan/conf/ecportal.ini
> >> >
> >> > This command should return:
> >> > <date> WARNI [root] {'celery_rows_deleted': <number of rows>,
> >> > 'task_status_rows_deleted': <number of rows>}
> >> >
> >> > The tables cleared by this command should not be used at all with the
> >> > code
> >> > in
> >> > this release, so this command will only have to be run once.
> >> >
> >> >
> >> > _______________________________________________
> >> > Ecodp-dev mailing list
> >> > Ecodp-dev at lists.okfn.org
> >> > http://lists.okfn.org/mailman/listinfo/ecodp-dev
> >> >
> >>
> >>
> >>
> >> --
> >> Bert Van Nuffelen
> >>
> >> Semantic Technologies Software Architect at TenForce
> >> www.tenforce.be
> >>
> >> Bert.Van.Nuffelen at tenforce.com
> >> Office: +32 (0)16 31 48 60
> >> Mobile:+32 479 06 24 26
> >> skype: bert.van.nuffelen
> >>
> >> _______________________________________________
> >> Ecodp-dev mailing list
> >> Ecodp-dev at lists.okfn.org
> >> http://lists.okfn.org/mailman/listinfo/ecodp-dev
> >
> >
> >
> > _______________________________________________
> > Ecodp-dev mailing list
> > Ecodp-dev at lists.okfn.org
> > http://lists.okfn.org/mailman/listinfo/ecodp-dev
> >
>
>
>
> --
> Bert Van Nuffelen
>
> Semantic Technologies Software Architect at TenForce
> www.tenforce.be
>
> Bert.Van.Nuffelen at tenforce.com
> Office: +32 (0)16 31 48 60
> Mobile:+32 479 06 24 26
> skype: bert.van.nuffelen
>
> _______________________________________________
> Ecodp-dev mailing list
> Ecodp-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ecodp-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.okfn.org/mailman/private/ecodp-dev/attachments/20130619/ad085c7f/attachment.html>