[ckan-changes] commit/ckanext-qa: John Glover: Update readme and setup.py for new QA version (depends on latest CKAN and ckanext-archiver). Remove setup.cfg, test.ini and who.ini as tests should now be run from the CKAN directory.
Bitbucket
commits-noreply at bitbucket.org
Wed Nov 2 15:50:25 UTC 2011
1 new commit in ckanext-qa:
https://bitbucket.org/okfn/ckanext-qa/changeset/50bfd2839375/
changeset: 50bfd2839375
user: John Glover
date: 2011-11-02 16:48:20
summary: Update readme and setup.py for new QA version (depends on latest CKAN and ckanext-archiver). Remove setup.cfg, test.ini and who.ini as tests should now be run from the CKAN directory.
affected #: 6 files
diff -r e3621f68950963bb8529d618a6a96015a1b5f1a1 -r 50bfd2839375aefaad107299cd2f65c1c9b1ff07 README.rst
--- a/README.rst
+++ b/README.rst
@@ -2,7 +2,6 @@
================================
-
The ckanext-qa extension will check each of your package resources and give
these resources an openness score based Tim Berners-Lee's five stars of openness
(http://lab.linkeddata.deri.ie/2010/star-scheme-by-example)
@@ -12,26 +11,17 @@
Once you have run the qa commands (see 'Using The QA Extension' below),
resources and packages will have a set of openness key's stores in their
extra properties.
-This process will also set the hash value and content_length for each
-individual resource.
Installation
------------
-Install the plugin using pip. You can either download it, then
-from the ckanext-qa directory, run
+Install the plugin using pip. Download it, then from the ckanext-qa directory, run
::
$ pip install -e ./
-Or, you can install it directly from the OKFN bitbucket repository:
-
-::
-
- $ pip install -e hg+https://bitbucket.org/okfn/ckanext-qa#egg=ckanext-qa
-
This will register a plugin entry point, so you can now add the following
to the ``[app:main]`` section of your CKAN .ini file:
@@ -46,90 +36,29 @@
Configuration
-------------
-Create a directory for the downloads:
-
-::
-
- sudo mkdir -p /var/lib/ckan/dgu/qa/download
- sudo chown www-data:ckan /var/lib/ckan/dgu/qa/download/
- sudo chmod g+w /var/lib/ckan/dgu/qa/download
-
-Add this a config option containing the path to this directory to your CKAN .ini file:
-
-::
-
- ckan.qa_archive = /var/lib/ckan/dgu/qa/download
-
-If you plan to use a local webstore to make processed resources available online,
-then you must also set the webstore url in the CKAN .ini file.
-
-(eg: if using the datapreview plugin. See the sections 'Using The QA Extension'
-and 'Webstore Integration' for more information).
-
-::
-
- ckan.webstore_url = http://test-webstore.ckan.net
-
-The paster ``archive`` and ``qa`` commands (see 'Using The QA Extension below) make
-some changes to both packages and resources. All changes will be performed as a
-user called ``okfn_maintenance``, so this user must have permission to edit all
-packages and resources. The easiest way to do this is to make the user a system administrator
-for your ckan instance.
-
-To do this, from your ckan directory run:
-
-::
-
- paster sysadmin -c <path/to/your.ini> add okfn_maintenance
-
-You can create cron jobs for each of the QA commands:
-
-::
-
- # m h dom mon dow command
- 0 0 1 * * paster --plugin="ckanext-qa" archive update --config=/etc/ckan/dgu/dgu.ini
- 0 0 1 * * paster --plugin="ckanext-qa" qa update --config=/etc/ckan/dgu/dgu.ini
- 0 0 1 * * paster --plugin="ckanext-qa" process update --config=/etc/ckan/dgu/dgu.ini
+The QA extension now depends on the CKAN Archiver extension and CKAN 1.5 (with Celery), but no additional
+configuration is needed.
Using The QA Extension
----------------------
-The QA process is currently broken down into two main steps:
+**QA**: analyze the results of the archiving step and calculating resource/package openness ratings.
-* **Archive**: Attempt to download and save all resources.
-* **QA**: analyze the results of the archiving step and calculating resource/package
- openness ratings.
-
-Additionally, a useful third step can be performed:
-
-* **Process** archived data, parsing content and making it available
- online using a REST API. This allows archived data to be easily viewed
- and manipulated by users, and in particular this is required
- if using the ckan datapreview extension.
-
-Each of these three steps can be performed by running the associated ``paster`` command
+This step can be performed by running the associated ``paster`` command
from the ckanext-qa directory.
::
- $ paster archive update|clean [package name/id] [--limit=N] --config=../ckan/development.ini
+ $ paster qa update|clean [package name/id] --config=<path to ckan config file>
+
+``update`` or ``clean`` will either update the package resources or remove everything changed by
+the QA Extension respectively.
- $ paster qa update|clean [package name/id] [--limit=N] --config=../ckan/development.ini
-
- $ paster process update|clean [package name/id] --config=../ckan/development.ini
-
-For each command you must specify either ``update`` or ``clean`` as subcommand, which will either
-download/update/process the package resources or remove everything changed by the QA Extension
-respectively.
-
-Each command can be run on just a single package by giving the package ``name`` or ``ID`` after the
+The command can be run on just a single package by giving the package ``name`` or ``ID`` after the
``update/clean`` subcommand. If no package name is given, the database is scanned
for a list of all packages and the command is run on each one.
-An additional ``limit`` parameter can specified for the ``archive`` and ``qa`` commands, which
-will stop the command after it has processed ``N`` packages.
-
After you run the ``archive`` and ``qa`` commands, the QA results can be viewed
at
@@ -138,43 +67,6 @@
http://your-ckan-instance/qa
-Webstore Integration
---------------------
-
-**Webstore Overview**
-
-The webstore is a RESTful data store for tabular and table-like data.
-It can be used as a dynamic storage for table data, allowing filtered,
-partial or full retrieval and format conversion.
-For more information see http://github.com/okfn/webstore
-
-
-**Use With QA**
-
-By using the webstore, it is possible to make archived resources accessible
-using a RESTful API. This is done by using the ``process`` paster command.
-When ``process`` is run, it goes through each resource that has been downloaded
-and attempts to parse it and put it in the webstore database.
-This data can then be used by other applications, such as the ckanext-datapreview extension.
-
-**Configuring A Webstore For Use With The QA Extension**
-
-It is recommended that you use the same directory for the webstore that you
-use for QA archiving. To do this, make sure that the ``SQLITE_DIR`` config
-value in the webstore application is set to the same value as the
-``ckan.qa_archive`` config value. For example, you could hardcode this value into
-the webstore configuration options, or add the following to the webstore WSGI file:
-
-::
-
- from webstore.web import app as application
- application.config['SQLITE_DIR'] = '/path/to/qa_archive'
-
-It is possible to use other directories but this would
-currently require reconfiguring paths in the ``commands/process.py`` file
-and making sure that the web server has read/write access to the directories.
-
-
API Access
----------
@@ -204,8 +96,9 @@
$ pip install nose mock
-Then, run nosetests from the ckanext-qa directory
+Then, run nosetests from the ckan directory
::
- $ nosetests --ckan
+ $ nosetests --ckan <path to ckanext-qa>/tests
+
diff -r e3621f68950963bb8529d618a6a96015a1b5f1a1 -r 50bfd2839375aefaad107299cd2f65c1c9b1ff07 ckanext/qa/__init__.py
--- a/ckanext/qa/__init__.py
+++ b/ckanext/qa/__init__.py
@@ -6,4 +6,4 @@
# import pkgutil
# __path__ = pkgutil.extend_path(__path__, __name__)
-__version__ = '0.1'
+__version__ = '0.3'
diff -r e3621f68950963bb8529d618a6a96015a1b5f1a1 -r 50bfd2839375aefaad107299cd2f65c1c9b1ff07 setup.cfg
--- a/setup.cfg
+++ /dev/null
@@ -1,2 +0,0 @@
-[nosetests]
-with-pylons=test.ini
diff -r e3621f68950963bb8529d618a6a96015a1b5f1a1 -r 50bfd2839375aefaad107299cd2f65c1c9b1ff07 setup.py
--- a/setup.py
+++ b/setup.py
@@ -1,23 +1,19 @@
from setuptools import setup, find_packages
-
-try:
- from ckanext.qa import __version__
-except:
- __version__ = '0.2a'
+from ckanext.qa import __version__
setup(
name='ckanext-qa',
version=__version__,
- description="Quality assurance plugin for CKAN",
- long_description="""\
+ description="Quality Assurance plugin for CKAN",
+ long_description="""
""",
classifiers=[], # Get strings from http://pypi.python.org/pypi?%3Aaction=list_classifiers
keywords='',
- author='CKAN Team (Open Knowledge Foundation)',
- author_email='ckan at okfn.org',
+ author='Open Knowledge Foundation',
+ author_email='info at okfn.org',
url='http://ckan.org/wiki/Extensions',
license='mit',
- packages=find_packages(exclude=['ez_setup', 'examples', 'tests']),
+ packages=find_packages(exclude=['tests']),
namespace_packages=['ckanext', 'ckanext.qa'],
include_package_data=True,
zip_safe=False,
@@ -28,16 +24,14 @@
'nose',
'mock',
],
- test_suite = 'nose.collector',
entry_points=\
"""
- [ckan.plugins]
- # Add plugins here, eg
- qa=ckanext.qa.plugin:QA
[paste.paster_command]
package-scores = ckanext.qa.commands.package_score:PackageScore
- archive = ckanext.qa.commands.archive:Archive
- process = ckanext.qa.commands.process:Process
qa = ckanext.qa.commands.qa:QA
+
+ [ckan.plugins]
+ qa=ckanext.qa.plugin:QA
""",
)
+
diff -r e3621f68950963bb8529d618a6a96015a1b5f1a1 -r 50bfd2839375aefaad107299cd2f65c1c9b1ff07 test.ini
--- a/test.ini
+++ /dev/null
@@ -1,128 +0,0 @@
-# ckanext-qa testing environment configuration
-
-[DEFAULT]
-debug = false
-
-[server:main]
-use = egg:Paste#http
-host = 0.0.0.0
-port = 5000
-
-[app:main]
-use = egg:ckan
-
-ckan.qa_archive = %(here)s/test_archive
-
-# Here we hard-code the database and a flag to make default tests run fast.
-faster_db_test_hacks = True
-sqlalchemy.url = sqlite:///%(here)s/tests/test.db
-
-ckan.cache_validation_enabled = True
-ckan.cache_enabled = False
-ckan.cache.default_expires = 200
-
-package_form = standard
-carrot_messaging_library = queue
-ckan.site_url = http://test.ckan.net
-package_new_return_url = http://localhost/package/<NAME>?test=new
-package_edit_return_url = http://localhost/package/<NAME>?test=edit
-
-ckan.extra_resource_fields = alt_url
-
-# disable this so we can test all types of indexing
-ckan.build_search_index_synchronously = false
-
-# Add additional test specific configuration options as necessary.
-auth.blacklist = 83.222.23.234
-search_backend = sql
-
-# Change API key HTTP header to something non-standard.
-apikey_header_name = X-Non-Standard-CKAN-API-Key
-
-# use <strong> so we can check that html is *not* escaped
-ckan.template_footer_end = <strong>TEST TEMPLATE_FOOTER_END TEST</strong>
-
-full_stack = true
-cache_dir = %(here)s/data
-beaker.session.key = ckan
-beaker.session.secret = l5Y9J+JZsnXHLd+9Df+W+Inaf
-app_instance_uuid = {ba835a3e-76d8-4e0c-b71f-1baafb2d11dc}
-
-# repoze.who config
-who.config_file = %(here)s/who.ini
-who.log_level = warning
-who.log_file = %(cache_dir)s/who_log.ini
-
-# cache to persistent files
-beaker.cache.type = file
-
-# CKAN QoS monitoring
-ckan.enable_call_timing = false
-
-# Package form to use
-package_form = standard
-
-## Update the search index synchronously (i.e. in-process rather than
-## out-of-process as would be case if using AMQP framework)
-## Set to false to disable, true to enable
-## Default enabled (and enabled if option entirely absent)
-## NOTE this is mutually exclusive with ckan.async_notifier
-ckan.build_search_index_synchronously = true
-
-## Title of site (using in several places including templates and <title> tag
-ckan.site_title = CKAN
-
-## Logo image to use (replaces site_title string on front page if defined)
-ckan.site_logo = http://assets.okfn.org/p/ckan/img/ckan_logo_largetext.png
-
-## Site tagline / description (used on front page)
-ckan.site_description =
-
-## Used in creating some absolute urls (such as rss feeds, css files) and
-## dump filenames
-ckan.site_url =
-
-## Favicon (default is the CKAN software favicon)
-ckan.favicon = http://assets.okfn.org/p/ckan/img/ckan.ico
-
-# Directory for logs (produced by cron scripts associated with ckan)
-ckan.log_dir = %(here)s/log
-
-# Directory for JSON/CSV dumps (must match setting in apache config)
-ckan.dump_dir = %(here)s/dump
-
-# Directory for SQL database backups
-ckan.backup_dir = %(here)s/backup
-
-# Logging configuration
-[loggers]
-keys = root, ckan, sqlalchemy
-
-[handlers]
-keys = console
-
-[formatters]
-keys = generic
-
-[logger_root]
-level = INFO
-handlers = console
-
-[logger_ckan]
-qualname = ckan
-handlers =
-level = INFO
-
-[logger_sqlalchemy]
-handlers =
-qualname = sqlalchemy.engine
-level = WARN
-
-[handler_console]
-class = StreamHandler
-args = (sys.stdout,)
-level = NOTSET
-formatter = generic
-
-[formatter_generic]
-format = %(asctime)s %(levelname)-5.5s [%(name)s] %(message)s
diff -r e3621f68950963bb8529d618a6a96015a1b5f1a1 -r 50bfd2839375aefaad107299cd2f65c1c9b1ff07 who.ini
--- a/who.ini
+++ /dev/null
@@ -1,20 +0,0 @@
-[plugin:friendlyform]
-use = repoze.who.plugins.friendlyform:FriendlyFormPlugin
-login_form_url= /user/login
-login_handler_path = /login_generic
-logout_handler_path = /user/logout
-rememberer_name = auth_tkt
-post_login_url = /user/logged_in
-post_logout_url = /user/logged_out
-
-[general]
-request_classifier = repoze.who.classifiers:default_request_classifier
-
-[identifiers]
-plugins = friendlyform;browser
-
-[authenticators]
-plugins = ckan.lib.authenticator:UsernamePasswordAuthenticator
-
-[challengers]
-plugins = friendlyform;browser
Repository URL: https://bitbucket.org/okfn/ckanext-qa/
--
This is a commit notification from bitbucket.org. You are receiving
this because you have the service enabled, addressing the recipient of
this email.
More information about the ckan-changes
mailing list