[ckan-changes] commit/ckan: 12 new changesets
Bitbucket
commits-noreply at bitbucket.org
Thu Jul 28 16:21:11 UTC 2011
12 new changesets in ckan:
http://bitbucket.org/okfn/ckan/changeset/acab05d0278c/
changeset: acab05d0278c
user: Anna PS
date: 2011-07-11 18:14:54
summary: [doc,refactor][l]: First, incomplete round of docs changes
affected #: 114 files (133.7 KB)
Diff too large to display.
http://bitbucket.org/okfn/ckan/changeset/10609c7f2c7b/
changeset: 10609c7f2c7b
user: Anna PS
date: 2011-07-26 20:32:02
summary: Make second enormous docs refactor
affected #: 130 files (133.4 KB)
Diff too large to display.
http://bitbucket.org/okfn/ckan/changeset/8d61109aa8c2/
changeset: 8d61109aa8c2
user: Anna PS
date: 2011-07-27 21:24:46
summary: Make round of docs changes following ckan-dev feedback
affected #: 21 files (38.1 KB)
--- a/README.txt Tue Jul 26 19:32:02 2011 +0100
+++ b/README.txt Wed Jul 27 20:24:46 2011 +0100
@@ -1,5 +1,10 @@
README
++++++
+Welcome to CKAN.
-TBA: Update with details of Reference and Administration manuals.
\ No newline at end of file
+For installation instructions, see http://docs.ckan.org/install-from-source.html
+
+For background information and developer notes, see http://wiki.ckan.net
+
+For help during installation, contact ckan-dev at lists.okfn.org
\ No newline at end of file
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/doc/_templates/layout.html Wed Jul 27 20:24:46 2011 +0100
@@ -0,0 +1,20 @@
+{% extends '!layout.html' %}
+
+{% block footer %}
+ <div class="footer">
+ {% if hasdoc('copyright') %}
+ {% trans path=pathto('copyright'), copyright=copyright|e %}© <a href="{{ path }}">Copyright</a> {{ copyright }}.{% endtrans %}
+ {% else %}
+ {% trans copyright=copyright|e %}© Copyright {{ copyright }}.{% endtrans %}
+ {% endif %}
+ {% if last_updated %}
+ {% trans last_updated=last_updated|e %}Last updated on {{ last_updated }}.{% endtrans %}
+ {% endif %}
+ {% if show_sphinx %}
+ {% trans sphinx_version=sphinx_version|e %}Created using <a href="http://sphinx.pocoo.org/">Sphinx</a> {{ sphinx_version }}.{% endtrans %}
+ {% endif %}
+ <br/>
+ Please contact the <a href="http://lists.okfn.org/mailman/listinfo/ckan-dev">ckan-dev mailing list</a>
+ with any questions.
+ </div>
+{% endblock %}
\ No newline at end of file
--- a/doc/about.rst Tue Jul 26 19:32:02 2011 +0100
+++ b/doc/about.rst Wed Jul 27 20:24:46 2011 +0100
@@ -6,9 +6,9 @@
CKAN is an open source project and contributions are welcome!
-We discuss large changes and new features on the `ckan-discuss mailing list <http://lists.okfn.org/mailman/listinfo/ckan-discuss>`_.
+We discuss large changes and new features on the `ckan-discuss <http://lists.okfn.org/mailman/listinfo/ckan-discuss>`_ mailing list, and technical issues on the `ckan-dev <http://lists.okfn.org/mailman/listinfo/ckan-dev>`_ mailing list. Please join us there.
-Proposed changes should be made on a personal CKAN fork (e.g. on BitBucket), then merges should be requested with the mainline via the ckan-discuss list.
+You can find developer resources and links to our ticketing system on the `CKAN wiki <http://wiki.ckan.net/Main_Page>`_.
Acknowledgements
----------------
@@ -20,7 +20,7 @@
* `Pylons <http://pylonshq.com/>`_
* CKAN logo: "angry hamster" by http://www.maedelmaedel.com/ and
http://www.villainous.biz/
- * `famfamfam.com silk icons <http://www.famfamfam.com/lab/icons/silk/>`_
+ * `FamFamFam silk icons <http://www.famfamfam.com/lab/icons/silk/>`_
Copying and Licence
-------------------
--- a/doc/buildbot.rst Tue Jul 26 19:32:02 2011 +0100
+++ b/doc/buildbot.rst Wed Jul 27 20:24:46 2011 +0100
@@ -1,9 +1,11 @@
-============
-Use Buildbot
-============
+================
+Install Buildbot
+================
This section provides information for CKAN core developers setting up buildbot on an Ubuntu Lucid machine.
+If you simply want to check the status of the latest CKAN builds, visit http://buildbot.okfn.org/.
+
Apt Installs
============
--- a/doc/conf.py Tue Jul 26 19:32:02 2011 +0100
+++ b/doc/conf.py Wed Jul 27 20:24:46 2011 +0100
@@ -29,7 +29,7 @@
extensions = ['sphinx.ext.autodoc']
# Add any paths that contain templates here, relative to this directory.
-templates_path = ['.templates']
+templates_path = ['_templates']
# The suffix of source filenames.
source_suffix = '.rst'
@@ -42,7 +42,9 @@
# General information about the project.
project = u'CKAN (Comprehensive Knowledge Archive Network)'
+project_short_name = u'CKAN'
copyright = u'2009, Open Knowledge Foundation'
+html_show_sphinx = False
# The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the
@@ -111,7 +113,7 @@
html_title = "%s v%s Administration Guide" % (project, release)
# A shorter title for the navigation bar. Default is the same as html_title.
-#html_short_title = None
+html_short_title = "%s Admin Guide" % (project_short_name)
# The name of an image file (relative to this directory) to place at the top
# of the sidebar.
--- a/doc/configuration.rst Tue Jul 26 19:32:02 2011 +0100
+++ b/doc/configuration.rst Wed Jul 27 20:24:46 2011 +0100
@@ -34,6 +34,39 @@
Front-End Settings
------------------
+
+.. index::
+ single: site_description
+
+site_description
+^^^^^^^^^^^^^^^^
+
+Example::
+
+ ckan.site_description=
+
+Default value: (none)
+
+This is for a description, or tag line for the site, as displayed in the header of the CKAN web interface.
+
+.. index::
+ single: site_logo
+
+site_logo
+^^^^^^^^^
+
+Example::
+
+ ckan.site_logo=/images/ckan_logo_fullname_long.png
+
+Default value: (none)
+
+This sets the logo used in the title bar.
+
+.. index::
+ single: site_url
+
+
.. index::
single: package_hide_extras
@@ -68,6 +101,22 @@
3. A visible RDF link on the page. e.g. `<a href="http://semantic.ckan.net/record/b410e678-8a96-40cf-8e46-e8bd4bf02684.rdf">`
+.. index::
+ single: dumps_url, dumps_format
+
+dumps_url & dumps_format
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+Example::
+
+ ckan.dumps_url = http://ckan.net/dump/
+ ckan.dumps_format = CSV/JSON
+
+If there is a page which allows you to download a dump of the entire catalogue then specify the URL and the format here, so that it can be advertised in the web interface. ``dumps_format`` is just a string for display.
+
+For more information on using dumpfiles, see :doc:`database_dumps`.
+
+
Cache Settings
--------------
@@ -138,31 +187,6 @@
Setting this option to False turns off OpenID for login.
-Licensing Settings
-------------------
-
-.. index::
- single: licenses_group_url
-
-licenses_group_url
-^^^^^^^^^^^^^^^^^^
-
-A url pointing to a JSON file containing a list of licence objects. This list
-determines the licences offered by the system to users, for example when
-creating or editing a package.
-
-This is entirely optional - by default, the system will use the CKAN list of
-licences available in the `Python licenses package <http://pypi.python.org/pypi/licenses>`_.
-
-More details about the CKAN license objects - including the licence format and some
-example licence lists - can be found at the `Open Licenses Service
-<http://licenses.opendefinition.org/>`_.
-
-Examples::
-
- licenses_group_url = file:///path/to/my/local/json-list-of-licenses.js
- licenses_group_url = http://licenses.opendefinition.org/2.0/ckan_original
-
.. _config-i18n:
@@ -252,6 +276,29 @@
The ``<NAME>`` string is replaced with the name of the package edited. Full details of this process are given in :doc:`form-integration`.
+.. index::
+ single: licenses_group_url
+
+licenses_group_url
+^^^^^^^^^^^^^^^^^^
+
+A url pointing to a JSON file containing a list of licence objects. This list
+determines the licences offered by the system to users, for example when
+creating or editing a package.
+
+This is entirely optional - by default, the system will use the CKAN list of
+licences available in the `Python licenses package <http://pypi.python.org/pypi/licenses>`_.
+
+More details about the CKAN license objects - including the licence format and some
+example licence lists - can be found at the `Open Licenses Service
+<http://licenses.opendefinition.org/>`_.
+
+Examples::
+
+ licenses_group_url = file:///path/to/my/local/json-list-of-licenses.js
+ licenses_group_url = http://licenses.opendefinition.org/2.0/ckan_original
+
+
Messaging Settings
------------------
@@ -359,34 +406,6 @@
This sets the name of the site, as displayed in the CKAN web interface.
.. index::
- single: site_description
-
-site_description
-^^^^^^^^^^^^^^^^
-
-Example::
-
- ckan.site_description=
-
-Default value: (none)
-
-This is for a description, or tag line for the site, as displayed in the header of the CKAN web interface.
-
-.. index::
- single: site_logo
-
-site_logo
-^^^^^^^^^
-
-Example::
-
- ckan.site_logo=/images/ckan_logo_fullname_long.png
-
-Default value: (none)
-
-This sets the logo used in the title bar.
-
-.. index::
single: site_url
site_url
@@ -460,24 +479,6 @@
Format as a space-separated list of the extension names. The extension name is the key in the [ckan.plugins] section of the extension's ``setup.py``. For more information on extensions, see :doc:`extensions`.
-Dumpfile Settings
------------------
-
-.. index::
- single: dumps_url, dumps_format
-
-dumps_url & dumps_format
-^^^^^^^^^^^^^^^^^^^^^^^^
-
-Example::
-
- ckan.dumps_url = http://ckan.net/dump/
- ckan.dumps_format = CSV/JSON
-
-If there is a page which allows you to download a dump of the entire catalogue then specify the URL and the format here, so that it can be advertised in the web interface. ``dumps_format`` is just a string for display.
-
-For more information on using dumpfiles, see :doc:`database_dumps`.
-
Directory Settings
------------------
--- a/doc/database_dumps.rst Tue Jul 26 19:32:02 2011 +0100
+++ b/doc/database_dumps.rst Wed Jul 27 20:24:46 2011 +0100
@@ -8,17 +8,38 @@
Creating a Dump
-----------------
-Your dump script needs to run the ``paster`` command. If you are using a Python environment, as part of a development installation, it should also enable the environment.
+We provide two ``paster`` methods to create dumpfiles.
-For example, you could create ``/home/okfn/var/srvc/ckan.net/dump.sh`` as follows::
+* ``db simple-dump-json`` - A simple dumpfile, useful to create a public listing of the packages with no user information. All packages are dumped, including deleted packages and ones with strict authorization.
+* ``db dump`` - A more complicated dumpfile, useful for backups. Replicates the database completely, including users, their personal info and API keys, and hence should be kept private.
+
+For more information on paster, see :doc:`paster`.
+
+Using db simple-dump-json
++++++++++++++++++++++++++
+
+If you are using a Python environment, as part of a development installation, first enable the environment::
. /home/okfn/var/srvc/ckan.net/pyenv/bin/activate || exit 1
- paster --plugin=ckan db simple-dump-json /home/okfn/var/srvc/ckan.net/dumps/ckan.net-daily.json --config=/home/okfn/var/srvc/ckan.net/ckan.net.ini
- gzip /home/okfn/var/srvc/ckan.net/dumps/ckan.net-daily.json
+
+Then create and zip the dumpfile::
+
+ paster --plugin=ckan db simple-dump-json /var/srvc/ckan/dumps/ckan.net-daily.json --config=/etc/ckan/std/std.ini
+ gzip /var/srvc/ckan/dumps/ckan.net-daily.json
Change ``simple-dump-json`` to ``simple-dump-csv`` if you want CSV format instead of JSON.
-These dump functions dump the entire database as it is stored in CKAN, omitting user account details.
+Using db dump
++++++++++++++
+
+If you are using a Python environment, as part of a development installation, first enable the environment::
+
+ . /var/srvc/ckan/pyenv/bin/activate || exit 1
+
+Then create and zip the dumpfile::
+
+ paster --plugin=ckan db dump /var/srvc/ckan/dumps/ckan.net-daily --config=/etc/ckan/std/std.ini
+ gzip /var/srvc/ckan/dumps/ckan.net-daily
Daily Dumps
-----------
@@ -38,7 +59,7 @@
Some simple additions to the Apache config can serve the files to users in a directory listing.
-To do this, add these lines to your virtual host config (e.g. `/etc/apache2/sites-enabled/ckan.net`)::
+To do this, add these lines to your virtual host config (e.g. ``/etc/apache2/sites-enabled/ckan.net``)::
Alias /dump/ /home/okfn/var/srvc/ckan.net/dumps/
--- a/doc/developer-install.rst Tue Jul 26 19:32:02 2011 +0100
+++ /dev/null Thu Jan 01 00:00:00 1970 +0000
@@ -1,48 +0,0 @@
-======================
-Developer Installation
-======================
-
-This section marks the start of the second half of this manual, covering advanced usage of CKAN.
-
-To do a developer install, please first follow the standard :doc:`install` guide, using Debian packages to get a working instance of CKAN.
-
-After that you'll need to set up and enter a virtual Python environment, as follows:
-
-::
-
- sudo apt-get install virtualenv pip mercurial
- virtualenv /home/ubuntu/pyenv
- . /home/ubuntu/pyenv/bin/activate
-
-To develop against any dependencies, you first need a developer install of CKAN itself. You can install CKAN like this:
-
-::
-
- pip install -e hg+http://bitbucket.org/okfn/ckan#egg=ckan
-
-You can now install any of the developer versions of the CKAN dependencies you want to work on like this (using the appropriate URL):
-
-::
-
- pip install -e hg+http://bitbucket.org/okfn/<dependency-name>@<version>#egg=<egg-name>
-
-The dependency you've installed will appear in ``/home/ubuntu/pyenv/src/`` where you can work on it.
-
-To test your changes you'll need to use the ``paster serve`` command from the ``ckan`` directory:
-
-::
-
- cd /home/ubuntu/pyenv/src/ckan
- . ../../bin/activate
- paster make-config ckan development.ini
-
-Then make any changes to the ``development.ini`` file that you need before continuing:
-
-::
-
- paster db upgrade
- paster serve --reload
-
-Your new CKAN developer install (together with any new dependencies you have installed in developer mode) will be running on http://localhost:5000/
-
-After installing CKAN, you should make sure that your deployment passes the tests, as described in :doc:`test`.
--- a/doc/extensions.rst Tue Jul 26 19:32:02 2011 +0100
+++ b/doc/extensions.rst Wed Jul 27 20:24:46 2011 +0100
@@ -2,14 +2,29 @@
Add Extensions
==============
-The CKAN software can be customised with 'extensions'. These are a simple way to extend core CKAN functions.
+This is where it gets interesting! The CKAN software can be customised with 'extensions'. These are a simple way to extend core CKAN functions.
Extensions allow you to customise CKAN for your own requirements, without interfering with the basic CKAN system.
-Popular CKAN extensions are listed on the `CKAN wiki <http://wiki.ckan.net/Main_Page>`_. All CKAN extensions can be found at `OKFN's bitbucket page <https://bitbucket.org/okfn/>`_, prefaced with ``ckanext-``.
+.. warning:: This is an advanced topic. At the moment, you need to have prepared your system to work with extensions, as described in :doc:`prepare-extensions`. We are working to make the most popular extensions more easily available as Debian packages.
-.. warning:: This is an advanced topic. At the moment, you need to have installed a developer version of CKAN to work with extensions, as described in :doc:`developer-install`. If you need help, contact the `ckan-discuss mailing list <http://lists.okfn.org/mailman/listinfo/ckan-discuss>`_. We are working to make the most popular extensions more easily available as Debian packages.
+Finding Extensions
+------------------
+Some popular extensions include:
+
+* `ckanext-admin <https://bitbucket.org/okfn/ckanext-admin>`_: Admin web interface for CKAN.
+* `ckanext-apps <https://bitbucket.org/okfn/ckanext-apps>`_: Apps and ideas catalogue extension for CKAN.
+* `ckanext-deliverance <https://bitbucket.org/okfn/ckanext-deliverance>`_: Extends CKAN to use the Deliverance HTTP proxy, which can request and render web pages from * an external site (e.g. a CMS like Drupal or Wordpress).
+* `ckanext-disqus <https://bitbucket.org/okfn/ckanext-disqus>`_: Allows users to comment on package pages with Disqus.
+* `ckanext-follower <https://bitbucket.org/okfn/ckanext-follower>`_: Allow users to follow packages.
+* `ckanext-googleanalytics <https://bitbucket.org/okfn/ckanext-googleanalytics>`_: Integrates Google Analytics data into CKAN. Gives download stats on package pages, list * of most popular packages, etc.
+* `ckanext-qa <https://bitbucket.org/okfn/ckanext-qa>`_: Provides link checker, 5 stars of openness and other Quality Assurance features.
+* `ckanext-rdf <https://bitbucket.org/okfn/ckanext-rdf>`_: Consolidated handling of RDF export and import for CKAN.
+* `ckanext-stats <https://bitbucket.org/okfn/ckanext-stats>`_: Statistics (and visuals) about the datasets in a CKAN instance.
+* `ckanext-wordpresser <https://bitbucket.org/okfn/ckanext-wordpresser>`_: CKAN plugin / WSGI middleware for combining CKAN with a Wordpress site.
+
+Many CKAN extensions are listed on the `CKAN wiki <http://wiki.ckan.net/Main_Page>`_. All CKAN extensions can be found at `OKFN's bitbucket page <https://bitbucket.org/okfn/>`_, prefaced with ``ckanext-``.
Installing an Extension
-----------------------
@@ -26,6 +41,7 @@
Prefix the source URL with the repo type (``hg+`` for Mercurial, ``git+`` for Git).
+ The dependency you've installed will appear in the ``src/`` directory under your Python environment.
2. Add the names of any plugin implementations the extension uses to the CKAN
config file. You can find these in the plugin's ``setup.py`` file under ``[ckan.plugins]``.
--- a/doc/index.rst Tue Jul 26 19:32:02 2011 +0100
+++ b/doc/index.rst Wed Jul 27 20:24:46 2011 +0100
@@ -4,8 +4,9 @@
This Administration Guide covers how to set up and manage `CKAN <http://ckan.org>`_ (Comprehensive Knowledge Archive Network) software.
-* The first half of the Guide covers installation and common administration tasks, including theming and authorization.
-* The second half (from :doc:`developer-install` onwards) covers advanced tasks, including extensions and forms.
+* The first two sections cover your two options for installing CKAN: package or source install.
+* The rest of the first half of the Guide, up to :doc:`authorization`, cover setup and basic admin.
+* The second half of the Guide, from :doc:`prepare-extensions` onwards, covers advanced tasks, including extensions and forms.
For high-level information on what CKAN is, see the `CKAN website <http://ckan.org>`_.
@@ -14,23 +15,24 @@
.. toctree::
:maxdepth: 2
- preparation
- install
+ install-from-package
+ install-from-source
+ post-installation
theming
loading_data
paster
authorization
+ prepare-extensions
+ extensions
+ plugins
+ forms
+ form-integration
database_dumps
upgrade
i18n
configuration
api
- developer-install
test
- extensions
- plugins
- forms
- form-integration
buildbot
about
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/doc/install-from-package.rst Wed Jul 27 20:24:46 2011 +0100
@@ -0,0 +1,428 @@
+==============================
+Option 1: Package Installation
+==============================
+
+This section describes how to install CKAN from packages. This is the recommended and by far the easiest way to install CKAN.
+
+It requires you to use Ubuntu 10.04. If you don't have access to Ubuntu 10.04, we provide support for two options:
+
+* :ref:`using-virtualbox`. This is suitable if you want to host your CKAN instance on a machine running any other OS.
+* :ref:`using-amazon`. This is suitable if you want to host your CKAN instance in the cloud, on a readymade Ubuntu OS.
+
+.. note:: We recommend you use this method unless you are an experimental user, a core CKAN developer, or you have no access to Ubuntu 10.04 through either of the methods above, in which
+
+For support during installation, please contact `the ckan-dev mailing list <http://lists.okfn.org/mailman/listinfo/ckan-dev>`_.
+
+Prepare your System
+--------------------
+
+CKAN runs on Ubuntu 10.04 (either 64-bit or 32-bit). If you are already using Ubuntu 10.04, you can continue straight to :doc:`install`.
+
+However, if you're not, you can either use VirtualBox to set up an Ubuntu VM, or an Amazon EC2 instance.
+
+.. _using-virtualbox:
+
+Option A: Using VirtualBox
+++++++++++++++++++++++++++
+
+This option is suitable if you want to install CKAN on a machine running an OS other than Ubuntu 10.04. `VirtualBox <http://www.virtualbox.org>`_ lets you set up a virtual machine to run Ubuntu 10.04.
+
+Pre-requisites and Downloads
+****************************
+
+First, check your machine meets `the pre-requisites for VirtualBox <http://www.virtualbox.org/wiki/End-user_documentation>`_. These include a fairly recent processor and some spare memory.
+
+Then download the installation files.
+
+* `Download the VirtualBox installer <http://www.virtualbox.org/wiki/Downloads>`_.
+* `Download the Ubuntu image <http://www.ubuntu.com/download/ubuntu/download>`_ - make sure you choose Ubuntu 10.04, and 64-bit or 32-bit as appropriate for your current operating system.
+
+Install VirtualBox
+******************
+
+.. note::
+
+ This tutorial is for a Mac, but you can find instructions for installing VirtualBox on any OS `in the VirtualBox Manual <http://www.virtualbox.org/manual/ch02.html>`_.
+
+To install, double-click on the VirtualBox installer:
+
+.. image:: images/virtualbox1-package.png
+ :width: 807px
+ :alt: The VirtualBox installer - getting started
+
+Click Continue to begin the installation process. Enter your password when required, and wait for the installation to finish.
+
+Create Your Virtual Machine
+***************************
+
+Go to Applications and open VirtualBox, then click New:
+
+.. image:: images/virtualbox4-newvm.png
+ :width: 807px
+ :alt: The VirtualBox installer - the New Virtual Machine Wizard
+
+Give your VM a name - we'll call ours ``ubuntu_ckan``. Under **OS Type**, choose **Linux** and **Ubuntu** (or **Ubuntu 64-bit** if you plan to install 64-bit Ubuntu).
+
+.. image:: images/virtualbox5-vmtype.png
+ :width: 807px
+ :alt: The VirtualBox installer - choosing your operating system
+
+Leave the memory size as 512MB, and choose **Create new hard disk**. This will open a new wizard:
+
+.. image:: images/virtualbox6-vmloc.png
+ :width: 807px
+ :alt: The VirtualBox installer - creating a new hard disk
+
+You can leave the defaults unchanged here too - click **Continue**, and then **Done**, and **Done** again, to create a new VM.
+
+Next, choose your VM from the left-hand menu, and click **Start**:
+
+.. image:: images/virtualbox7-startvm.png
+ :width: 807px
+ :alt: Starting your new VM
+
+This will open the First Run Wizard:
+
+.. image:: images/virtualbox8-firstrun.png
+ :width: 807px
+ :alt: The VirtualBox First Run Wizard
+
+After clicking **Continue**, you'll see **Select Installation Media**. This is where we need to tell our VM to boot from Ubuntu. Click on the file icon, and find your Ubuntu ``.iso`` file:
+
+.. image:: images/virtualbox9-iso.png
+ :width: 807px
+ :alt: When you get to Select Installation Media, choose your Ubuntu .iso file
+
+Click **Done**, wait for a few seconds, and you will see your Ubuntu VM booting.
+
+Set Up Ubuntu
+*************
+
+During boot, you will be asked if you want to try Ubuntu, or install it. Choose **Install Ubuntu**:
+
+.. image:: images/virtualbox11-ubuntu.png
+ :width: 807px
+ :alt: Booting Ubuntu - choose the Install Ubuntu option
+
+You can then follow the usual Ubuntu installation process.
+
+After Ubuntu is installed, from the main menu, choose **System > Administration > Update Manager**. You'll be asked if you want to install updates - say yes.
+
+When all the updates have been downloaded and installed, you'll be prompted to reboot Ubuntu.
+
+At this point, you can begin installing CKAN.
+
+.. _using-amazon:
+
+Option B: Using Amazon EC2
+++++++++++++++++++++++++++
+
+If you prefer to run your CKAN package install in the cloud, you can use an Amazon EC2 instance, which is a fairly cheap and lightweight way to set up a server.
+
+Create an Amazon Account
+************************
+
+If you don't already have an Amazon AWS account you'll need to create one first. You can `create an Amazon AWS account for EC2 here <http://aws.amazon.com/ec2/>`_.
+
+Configure EC2
+*************
+
+Once you have an EC2 account, you'll need to configure settings for your CKAN instance.
+
+Start by logging into your `Amazon AWS Console <https://console.aws.amazon.com/s3/home>`_ and click on the EC2 tab.
+
+Select the region you want to run your CKAN instance in - the security group you set up is region-specific. In this tutorial, we use EU West, so it will be easier to follow if you do too.
+
+.. image :: images/1.png
+
+Set up a Security Group
+^^^^^^^^^^^^^^^^^^^^^^^
+
+Click the **Security Groups** link in the **My Resources** section in the right-hand side of the dashboard.
+
+.. image :: images/2.png
+ :width: 807px
+
+Create a security group called ``web_test`` that gives access to ports 22, 80 and 5000 as shown below. This is needed so that you'll actually be able to access your server once it is created. You can't change these settings once the instance is running, so you need to do so now.
+
+.. image :: images/3a.png
+ :width: 807px
+
+.. image :: images/3b.png
+ :width: 807px
+
+Create a Keypair
+^^^^^^^^^^^^^^^^
+
+Now create a new keypair ``ckan_test`` to access your instance:
+
+.. image :: images/4.png
+ :width: 807px
+
+When you click **Create**, your browser will prompt you to save a keypair called ``ckan_test.pem``:
+
+.. image :: images/5.png
+ :width: 807px
+
+In this tutorial, we save the keypair in ``~/Downloads/ckan_test.pem``, but you should save it
+somewhere safe.
+
+.. note :: If you plan to boot your EC2 instance from the command line, you need to remember where you've put this file.
+
+
+Boot the EC2 Image
+******************
+
+CKAN requires Ubuntu 10.04 to run (either the i386 or amd64
+architectures). Luckily Canonical provide a `range of suitable images <http://uec-images.ubuntu.com/releases/10.04/release/>`_.
+
+The cheapest EC2 instance is the micro one, but that isn't very powerful, so in this tutorial,
+we'll use the 32-bit small version.
+
+We're in ``eu-west-1`` and we'll use an instance-only image (i.e. all the data will be lost when you shut it down) so we need the `ami-3693a542 <https://console.aws.amazon.com/ec2/home?region=eu-west-1#launchAmi=ami-3693a542>`_ AMI.
+
+.. note ::
+
+ There are more recent Ubuntu images at http://cloud.ubuntu.com/ami/ but we need the older 10.04 LTS release.
+
+At this point, you can either boot this image from the AWS
+console or launch it from the command line.
+
+
+Option 1: Boot the EC2 Image AMI via the AWS Console
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+From the EC2 dashboard, choose **Launch instance >**:
+
+.. image :: images/2.png
+ :width: 807px
+ :alt: Choose launch instance from the EC2 dashboard
+
+Now work through the wizard as shown in the following screenshots.
+
+In the first step search for ``ami-3693a542`` and select it from the results (it may take a few seconds for Amazon to find it).
+
+.. warning ::
+
+ No image other than ``ami-3693a542`` will work with CKAN.
+
+.. image :: images/i1.png
+ :width: 807px
+ :alt: Search for image ami-3693a542
+
+You can keep the defaults for all of the following screens:
+
+.. image :: images/i2.png
+ :width: 807px
+ :alt: Keep the defaults while setting up your instance
+.. image :: images/i3.png
+ :width: 807px
+ :alt: Keep the defaults while setting up your instance
+.. image :: images/i4.png
+ :width: 807px
+ :alt: Keep the defaults while setting up your instance
+.. image :: images/i5.png
+ :width: 807px
+ :alt: Keep the defaults while setting up your instance
+
+Choose the ``web_test`` security group you created earlier:
+
+.. image :: images/i6.png
+ :width: 807px
+ :alt: Choose the web_test security group you created earlier
+
+Then finish the wizard:
+
+.. image :: images/i7.png
+ :width: 807px
+ :alt: Finish the wizard
+
+Finally click the **View your instances on the Instances page** link:
+
+.. image :: images/i8.png
+ :width: 807px
+ :alt: View your instance
+
+After a few seconds you'll see your instance has booted. Now skip to :ref:`log-in-to-instance`.
+
+Option 2: Boot the EC2 Image AMI from the Command Line
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+[You can skip this section if you've just booted from the AWS console and go straight to :ref:`log-in-to-instance`]
+
+To boot from the command line you still need the same information but you enter it in one command. I'll show you now.
+
+Install The EC2 Tools Locally
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you are on Linux, you can just install the tools like this:
+
+::
+
+ sudo apt-get install ec2-ami-tools
+ sudo apt-get install ec2-api-tools
+
+If you are on Windows or Mac you'll need to `download them from the Amazon website <http://aws.amazon.com/developertools/351>`_.
+
+Once the software is installed you can use the files you've just downloaded to do create your instance.
+
+Get Security Certificates
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Next click on the **Account** link, right at the top of the screen, and you'll see this screen:
+
+.. image :: images/6.png
+ :width: 807px
+ :alt: The Account screen
+
+From this screen choose **Security Credentials** from the left hand side. Once
+the page has loaded scroll down and you'll see the **Access Credentials**
+section. Click on the **X.509 Certificate** tab:
+
+.. image :: images/7.png
+ :width: 807px
+ :alt: The Access Credentials screen
+
+Here you'll be able to create an X.509 certificate and private key.
+
+.. tip ::
+
+ You can only have two X.509 certificates at any given time, so you might need
+ to inactivate an old one first and then delete it before you are allowed to
+ create a new one, as in the screenshot above.
+
+Once you click the **Create New Certificate** link you get a popup which allows
+you to download the certificate and private key - do this. Once again, ours are in
+``~/Downloads``, but you should save it somewhere safe.
+
+.. image :: images/8.png
+ :width: 807px
+ :alt: Download your certificate
+
+.. tip ::
+
+ Amazon will only give you a private key file once when you create it so
+ although you can always go back to get a copy of the certificate, you can only
+ get the private key once. Make sure you save it in a safe place.
+
+You now have:
+
+* Your private key (``pk-[ID].pem``)
+* Your certificate file (``cert-[ID].pem``)
+* Your new keypair (``ckan-test.pem``)
+
+The private key and the certificate files have the same name in the ``ID`` part.
+
+Create an Ubuntu Instance
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Once the tools are installed, run this command:
+
+::
+
+ ec2-run-instances ami-3693a542 --instance-type m1.small --region eu-west-1 --group web_test \
+ --key ckan_test \
+ --private-key ~/Downloads/pk-[ID].pem \
+ --cert ~/Downloads/cert-[ID].pem
+
+
+.. note ::
+
+ The ``--key`` argument is the name of the keypair (``ckan_test``), not the certificate
+ itself (``ckan_test.pem``).
+
+.. warning ::
+
+ Amazon charge you for a minimum of one hour usage, so you shouldn't create and
+ destroy lots of EC2 instances unless you want to be charged a lot.
+
+.. _log-in-to-instance:
+
+Log in to the Instance
+**********************
+
+Once your instance has booted, you will need to find out its public DNS. Give it
+a second or two for the instance to load then browse to the running instance in
+the AWS console. If you tick your instance you'll be able to find the public
+DNS by scrolling down to the bottom of the **Description** tag.
+
+.. image :: images/8a.png
+ :width: 807px
+ :alt: Find the public DNS
+
+Here you can see that our public DNS is
+``ec2-79-125-86-107.eu-west-1.compute.amazonaws.com``. The private DNS only works
+from other EC2 instances so isn't any use to us.
+
+Once you've found your instance's public DNS, ensure the key has the correct permissions:
+
+::
+
+ chmod 0600 "ckan_test.pem"
+
+You can then log in like this:
+
+::
+
+ ssh -i ~/Downloads/ckan_test.pem ubuntu at ec2-46-51-149-132.eu-west-1.compute.amazonaws.com
+
+The first time you connect you'll see this, choose ``yes``:
+
+::
+
+ RSA key fingerprint is 6c:7e:8d:a6:a5:49:75:4d:9e:05:2e:50:26:c9:4a:71.
+ Are you sure you want to continue connecting (yes/no)? yes
+ Warning: Permanently added 'ec2-79-125-86-107.eu-west-1.compute.amazonaws.com,79.125.86.107' (RSA) to the list of known hosts.
+
+When you log in you'll see a welcome message. You can now proceed to :ref:`run-package-installer`.
+
+
+.. note ::
+
+ If this is a test install of CKAN, when you have finished using CKAN, you can shut down your EC2 instance through the AWS console.
+
+.. warning ::
+
+ Shutting down your EC2 instance will lose all your data. Also, Amazon charge you for a minimum usage of one hour, so don't create and destroy lots of EC2 instances unless you want to be charged a lot!
+
+
+.. _run-package-installer:
+
+Run the Package Installer
+-------------------------
+
+On your Ubuntu 10.04 system, open a terminal window and switch to the root user:
+
+::
+
+ sudo -s
+
+Install the CKAN packages as follows:
+
+::
+
+ echo 'deb http://apt.okfn.org/ubuntu_ckan-std_dev lucid universe' > /etc/apt/sources.list.d/okfn.list
+ wget -qO- http://apt.okfn.org/packages.okfn.key | sudo apt-key add -
+ apt-get update
+ apt-get install ckan-std
+
+Wait for the output to finish, then create your CKAN instance:
+
+::
+
+ ckan-std-install
+
+If you are using Amazon EC2, you will additionally need to set the hostname of your server. To do this, run the command below, replacing ``ec2-46-51-149-132.eu-west-1.compute.amazonaws.com`` with the public DNS of your EC2 instance. Leave the ``/`` at the end, as it is part of the ``sed`` command. Then restart Apache. You can skip this if installing on VirtualBox or a local server.
+
+::
+
+ sudo sed -e "s/ServerAlias \(.*\)/ServerAlias ec2-46-51-149-132.eu-west-1.compute.amazonaws.com/" \
+ -i /etc/apache2/sites-available/std.common
+ sudo /etc/init.d/apache2 restart
+
+Finally visit your CKAN instance - either at your Amazon EC2 hostname, or at http://localhost. You'll be redirected to the login screen because you won't have set up any permissions yet, so the welcome screen will look something like this.
+
+.. image :: images/9.png
+ :width: 807px
+
+You can now proceed to :doc:`post-installation`.
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/doc/install-from-source.rst Wed Jul 27 20:24:46 2011 +0100
@@ -0,0 +1,301 @@
+=============================
+Option 2: Install from Source
+=============================
+
+This section describes how to install CKAN from source. This removes the requirement for Ubuntu 10.04 that exists with :doc:`install-from-package`.
+
+.. warning:: Installing from source is complex and recommended only for users with no access to Ubuntu 10.04 and core CKAN developers.
+
+For support during installation, please contact `the ckan-dev mailing list <http://lists.okfn.org/mailman/listinfo/ckan-dev>`_.
+
+Install the Source
+------------------
+
+These are instructions to get developing with CKAN.
+
+Before you start, it may be worth `checking CKAN has passed the auto build and
+tests <http://buildbot.okfn.org/waterfall>`_.
+
+
+1. Ensure the required packages are installed.
+
+ ===================== ===============================================
+ Package Description
+ ===================== ===============================================
+ mercurial Source control
+ python-dev Python interpreter v2.5 - v2.7 and dev headers
+ postgresql PostgreSQL database
+ libpq-dev PostgreSQL library
+ python-psycopg2 PostgreSQL python module
+ libxml2-dev XML library development files
+ libxslt-dev XSLT library development files
+ python-virtualenv Python virtual environments
+ wget Command line tool for downloading from the web
+ build-essential Tools for building source code
+ git-core Git source control (for getting MarkupSafe src)
+ subversion Subversion source control (for pyutilib)
+ ===================== ===============================================
+
+ If you have access to ``apt-get``, you can install these packages as follows:
+
+ ::
+
+ sudo apt-get install build-essential libxml2-dev libxslt-dev
+ sudo apt-get install wget mercurial postgresql libpq-dev git-core
+ sudo apt-get install python-dev python-psycopg2 python-virtualenv
+ sudo apt-get install subversion
+
+2. Create a Python virtual environment.
+
+ In your home directory run the command below. It is currently important to
+ call your virtual environment ``pyenv`` so that the automated deployment tools
+ work correctly.
+
+ ::
+
+ cd ~
+ virtualenv pyenv
+
+ .. tip ::
+
+ If you don't have a ``python-virtualenv`` package in your distribution
+ you can get a ``virtualenv.py`` script from within the
+ `virtualenv source distribution <http://pypi.python.org/pypi/virtualenv/>`_
+ and then run ``python virtualenv.py pyenv`` instead.
+
+ To help with automatically installing CKAN dependencies we use a tool
+ called ``pip``. Make sure you have activated your environment (see step 3)
+ and then install it from an activated shell like this:
+
+ ::
+
+ easy_install pip
+
+3. Activate your virtual environment.
+
+ To work with CKAN it is best to adjust your shell settings so that your
+ shell uses the virtual environment you just created. You can do this like
+ so:
+
+ ::
+
+ . pyenv/bin/activate
+
+ When your shell is activated you will see the prompt change to something
+ like this:
+
+ ::
+
+ (pyenv)[ckan at host ~/]$
+
+ An activated shell looks in your virtual environment first when choosing
+ which commands to run. If you enter ``python`` now it will actually
+ run ``~/pyenv/bin/python`` which is what you want.
+
+4. Install CKAN code and required Python packages into the new environment.
+
+ First you'll need to install CKAN. For the latest version run:
+
+ ::
+
+ pip install --ignore-installed -e hg+http://bitbucket.org/okfn/ckan#egg=ckan
+
+ CKAN has a set of dependencies it requires which you should install too:
+
+ ::
+
+ pip install --ignore-installed -r pyenv/src/ckan/requires/lucid_missing.txt -r pyenv/src/ckan/requires/lucid_conflict.txt
+
+ The ``--ignore-installed`` option ensures ``pip`` installs software into
+ this virtual environment even if it is already present on the system.
+
+ If you are using Ubuntu Lucid you can install the rest of the dependencies
+ from the system versions like this:
+
+ ::
+
+ sudo apt-get install python-psycopg2 python-lxml python-sphinx
+ sudo apt-get install python-pylons python-formalchemy python-repoze.who
+ sudo apt-get install python-repoze.who-plugins python-tempita python-zope.interface
+
+ If you are not using Ubuntu Lucid you'll still need to install all the
+ dependencies that would have been met in the ``apt-get install`` command
+ at the start. You can do so like this:
+
+ ::
+
+ pip install --ignore-installed -r pyenv/src/ckan/requires/lucid_present.txt
+
+ This will take a **long** time. Particularly the install of the ``lxml``
+ package.
+
+ At this point you will need to deactivate and then re-activate your
+ virtual environment to ensure that all the scripts point to the correct
+ locations:
+
+ ::
+
+ deactivate
+ . pyenv/bin/activate
+
+5. Setup a PostgreSQL database.
+
+ List existing databases:
+
+ ::
+
+ psql -l
+
+ It is advisable to ensure that the encoding of databases is 'UTF8', or
+ internationalisation may be a problem. Since changing the encoding of PostgreSQL
+ may mean deleting existing databases, it is suggested that this is fixed before
+ continuing with the CKAN install.
+
+ Next you'll need to create a database user if one doesn't already exist.
+
+ .. tip ::
+
+ If you choose a database name, user or password which are different from those
+ suggested below then you'll need to update the configuration file you'll create in
+ the next step.
+
+ Here we choose ``ckantest`` as the database and ``ckanuser`` as the user:
+
+ ::
+
+ sudo -u postgres createuser -S -D -R -P ckantest
+
+ It should prompt you for a new password for the CKAN data in the database.
+ It is suggested you enter ``pass`` for the password.
+
+ Now create the database, which we'll call ``ckantest`` (the last argument):
+
+ ::
+
+ sudo -u postgres createdb -O ckantest ckantest
+
+6. Create a CKAN config file.
+
+ Make sure you are in an activated environment (see step 3) so that Python
+ Paste and other modules are put on the python path (your command prompt will
+ start with ``(pyenv)`` if you have) then change into the ``ckan`` directory
+ which will have been created when you installed CKAN in step 4 and create the
+ config file ``development.ini`` using Paste:
+
+ ::
+
+ cd pyenv/src/ckan
+ paster make-config ckan development.ini
+
+ You can give your config file a different name but the tests will expect you
+ to have used ``development.ini`` so it is strongly recommended you use this
+ name, at least to start with.
+
+ If you used a different database name or password when creating the database
+ in step 5 you'll need to now edit ``development.ini`` and change the
+ ``sqlalchemy.url`` line, filling in the database name, user and password you used.
+
+ ::
+
+ sqlalchemy.url = postgresql://ckantest:pass@localhost/ckantest
+
+ If you're using a remote host with password authentication rather than SSL authentication, use::
+
+ sqlalchemy.url = postgresql://<user>:<password>@<remotehost>/ckan?sslmode=disable
+
+ .. caution ::
+
+ Advanced users: If you are using CKAN's fab file capability you currently need to create
+ your config file as ``pyenv/ckan.net.ini`` so you will probably have
+ ignored the advice about creating a ``development.ini`` file in the
+ ``pyenv/src/ckan`` directory. This is fine but CKAN probably won't be
+ able to find your ``who.ini`` file. To fix this edit ``pyenv/ckan.net.ini``,
+ search for the line ``who.config_file = %(here)s/who.ini`` and change it
+ to ``who.config_file = who.ini``.
+
+ We are moving to a new deployment system where this incompatibility
+ will be fixed.
+
+7. Create database tables.
+
+ Now that you have a configuration file that has the correct settings for
+ your database, you'll need to create the tables. Make sure you are still in an
+ activated environment with ``(pyenv)`` at the front of the command prompt and
+ then from the ``pyenv/src/ckan`` directory run this command:
+
+ ::
+
+ paster db init
+
+ You should see ``Initialising DB: SUCCESS``. If you are not in the
+ ``pyenv/src/ckan`` directory or you don't have an activated shell, the command
+ will not work.
+
+ If the command prompts for a password it is likely you haven't set up the
+ database configuration correctly in step 6.
+
+8. Create the cache directory.
+
+ You need to create the Pylon's cache directory specified by 'cache_dir'
+ in the config file.
+
+ (from the ``pyenv/src/ckan`` directory):
+
+ ::
+
+ mkdir data
+
+
+9. Run the CKAN webserver.
+
+ NB If you've started a new shell, you'll have to activate the environment
+ again first - see step 3.
+
+ (from the ``pyenv/src/ckan`` directory):
+
+ ::
+
+ paster serve development.ini
+
+10. Point your web browser at: http://127.0.0.1:5000/
+
+ The CKAN homepage should load.
+
+.. _run-tests:
+
+Run Tests
+---------
+
+After completing your source installation of CKAN, you should check that tests run. Tests should be re-run (and amended if necessary) before checking in changes.
+
+Make sure you've created a config file at ``pyenv/ckan/development.ini``. Then activate the Python environment::
+
+ . pyenv/bin/activate
+
+Install nose into your virtual environment::
+
+ pip install --ignore-installed nose
+
+At this point you will need to deactivate and then re-activate your
+virtual environment to ensure that all the scripts point to the correct
+locations:
+
+::
+
+ deactivate
+ . pyenv/bin/activate
+
+Then run the quick development tests::
+
+ cd pyenv/src/ckan
+ nosetests ckan/tests --ckan
+
+You *must* run the tests from the CKAN directory as shown above, otherwise the
+``--ckan`` plugin won't work correctly.
+
+.. caution ::
+
+ By default, the test run is 'quick and dirty' - only good enough as an initial check.
+ See :doc:`test` for information on full developer tests.
+
+You can now proceed to :doc:`post-installation`.
--- a/doc/install.rst Tue Jul 26 19:32:02 2011 +0100
+++ /dev/null Thu Jan 01 00:00:00 1970 +0000
@@ -1,98 +0,0 @@
-============
-Installation
-============
-
-After you have set up your Ubuntu 10.04 system, you can begin installing CKAN.
-
-.. note:: CKAN currently requires Ubuntu 10.04. Before starting, you should set up Ubuntu 10.04 using VirtualBox or Amazon EC2. See :doc:`preparation`.
-
-Run the Package Installer
-=========================
-
-On your Ubuntu 10.04 system, open a terminal window and switch to the root user:
-
-::
-
- sudo -s
-
-Install the CKAN packages as follows:
-
-::
-
- echo 'deb http://apt.okfn.org/ubuntu_ckan-std_dev lucid universe' > /etc/apt/sources.list.d/okfn.list
- wget -qO- http://apt.okfn.org/packages.okfn.key | sudo apt-key add -
- apt-get update
- apt-get install ckan-std
-
-Wait for the output to finish, then create your CKAN instance:
-
-::
-
- ckan-std-install
-
-If you are using Amazon EC2, you will additionally need to set the hostname of your server. To do this, run the command below, replacing ``ec2-46-51-149-132.eu-west-1.compute.amazonaws.com`` with the public DNS of your EC2 instance. Leave the ``/`` at the end, as it is part of the ``sed`` command. Then restart Apache. You can skip this if installing on VirtualBox or a local server.
-
-::
-
- sudo sed -e "s/ServerAlias \(.*\)/ServerAlias ec2-46-51-149-132.eu-west-1.compute.amazonaws.com/" \
- -i /etc/apache2/sites-available/std.common
- sudo /etc/init.d/apache2 restart
-
-Finally visit your CKAN instance - either at your Amazon EC2 hostname, or at http://localhost. You'll be redirected to the login screen because you won't have set up any permissions yet, so the welcome screen will look something like this.
-
-.. image :: images/9.png
- :width: 807px
-
-.. _create-admin-user:
-
-Create an Admin User
-====================
-
-By default, CKAN has a set of locked-down permissions. To begin
-working with it you need to set up a user and some permissions.
-
-First create an admin account from the command line (you must be root, ``sudo -s``):
-
-::
-
- paster --plugin=ckan user add admin --config=/etc/ckan/std/std.ini
-
-When prompted, enter a password - this is the password you will use to log in to CKAN. In the resulting output, note that you will also get assigned a CKAN API key.
-
-.. note :: This command is your first introduction to some important CKAN concepts.
- * paster is the script used to run CKAN commands.
- * std.ini is the CKAN config file. You can change options in this file to configure CKAN.
-
-For exploratory purposes, you might was well make the ``admin`` user a
-sysadmin. You obviously wouldn't give most users these rights as they would
-then be able to do anything. You can make the ``admin`` user a sysadmin like
-this:
-
-::
-
- paster --plugin=ckan sysadmin add admin --config=/etc/ckan/std/std.ini
-
-You can now login to the CKAN frontend with the username ``admin`` and the password you set up.
-
-.. _create-test-data:
-
-Load Test Data
-==============
-
-Finally, it can be handy to have some test data to start with. You can get test data like this:
-
-::
-
- paster --plugin=ckan create-test-data --config=/etc/ckan/std/std.ini
-
-You now have a CKAN instance that you can log in to, with some test data to check everything
-works.
-
-You can now proceed to :doc:`theming`.
-
-Deployment Notes
-================
-
-Standard production deployment of CKAN is Apache with modwsgi.
-
-However CKAN has been successfully deployed via a variety of other methods including Apache reverse proxy + paster, nginx reverse proxy + paster, and nginx + uwsgi.
\ No newline at end of file
--- a/doc/loading_data.rst Tue Jul 26 19:32:02 2011 +0100
+++ b/doc/loading_data.rst Wed Jul 27 20:24:46 2011 +0100
@@ -1,11 +1,99 @@
-===============
-Import Datasets
-===============
+=============
+Load Datasets
+=============
-If you have many datasets to add to CKAN, there are various ways to automate the process.
+You can upload individual datasets through the CKAN front-end, but for importing datasets on masse, you have two choices:
-* **Loader scripts**. We provide standard loading scripts for data.
-* **CKAN API**. You can use the CKAN API to upload datasets. See :doc:`api`.
-* **Harvester extension**. The harvester extension is useful for loading datasets that need to be scraped (found at https://bitbucket.org/okfn/ckanext-harvest/).
+* :ref:`load-data-api`. You can use the `CKAN API <api.html>`_ to script import. To simplify matters, we offer provide standard loading scripts for Google Spreadsheets, CSV and Excel.
-The most appropriate method will depend on the nature of your datasets. Hence, rather than attempting to provide detailed documentation, here, we recommend you `contact the ckan-dev mailing list <http://lists.okfn.org/mailman/listinfo/ckan-dev>`_ for advice.
\ No newline at end of file
+* :ref:`load-data-harvester`. The `CKAN harvester extension <https://bitbucket.org/okfn/ckanext-harvest/>`_ provides web and command-line interfaces for larger import tasks.
+
+If you need advice on data import, `contact the ckan-dev mailing list <http://lists.okfn.org/mailman/listinfo/ckan-dev>`_.
+
+.. note :: If loading your data requires scraping a web page regularly, you may find it best to write a scraper on `ScraperWiki <http://www.scraperwiki.com>`_ and combine this with either of the methods above.
+
+.. _load-data-api:
+
+Import Data with the CKAN API
+-----------------------------
+
+You can use the `CKAN API <api.html>`_ to upload datasets directly into your CKAN instance.
+
+The Simplest Approach - ckanclient
+++++++++++++++++++++++++++++++++++
+
+The most basic way to automate package loading is with a Python script using the `ckanclient library <http://pypi.python.org/pypi/ckanclient>`_. You will need to register for an API key first.
+
+You can install ckanclient with::
+
+ pip install ckanclient
+
+Here is an example script to register a new package::
+
+ import ckanclient
+ # Instantiate the CKAN client.
+ ckan = ckanclient.CkanClient(api_key=my_api_key, base_location="http://myckaninstance.com/api")
+ # Describe the package.
+ package_entity = {
+ 'name': my_package_name,
+ 'url': my_package_url,
+ 'download_url': my_package_download_url,
+ 'tags': my_package_keywords,
+ 'notes': my_package_long_description,
+ }
+ # Register the package.
+ ckan.package_register_post(package_entity)
+
+Loader Scripts
+++++++++++++++
+
+'Loader scripts' provide a simple way to take any format metadata and bulk upload it to a remote CKAN instance.
+
+Essentially each set of loader scripts converts the dataset metadata to the standard 'package' format, and then loads it into CKAN.
+
+Loader scripts are generally stored into the `ckanext` repository. To get a flavour of what loader scripts look like, take a look at `the ONS scripts <https://bitbucket.org/okfn/ckanext-dgu/src/default/ckanext/dgu/ons/>`_.
+
+Loader Scripts for CSV and Excel
+********************************
+
+For CSV and Excel formats, the `SpreadsheetPackageImporter` (found in ``ckan/lib/spreadsheet_importer.py``) loader script wraps the file in `SpreadsheetData` before extracting the records into `SpreadsheetDataRecords`.
+
+SpreadsheetPackageImporter copes with multiple title rows, data on multiple sheets, dates. The loader can reload packages based on a unique key column in the spreadsheet, choose unique names for packages if there is a clash, add/merge new resources for existing packages and manage package groups.
+
+Loader Scripts for Google Spreadsheets
+**************************************
+
+The `GoogleSpreadsheetReader` class (found in ``ckanclient.loaders``) simplifies the process of loading data from Google Spreadsheets.
+
+`This script <https://bitbucket.org/okfn/ckanext/src/default/bin/ckanload-italy-nexa>`_ has a simple example of loading data from Google Spreadsheets.
+
+Write Your Own Loader Script
+****************************
+
+## this needs work ##
+
+First, you need an importer that derives from `PackageImporter` (found in ``ckan/lib/importer.py``). This takes whatever format the metadata is in and sorts it into records of type `DataRecord`.
+
+Next, each DataRecord is converted into the correct fields for a package using the `record_2_package` method. This results in package dictionaries.
+
+The `PackageLoader` takes the package dictionaries and loads them onto a CKAN instance using the ckanclient. There are various settings to determine:
+
+ * ##how to identify the same package, previously been loaded into CKAN.## This can be simply by name or by an identifier stored in another field.
+ * how to merge in changes to an existing packages. It can simply replace it or maybe merge in resources etc.
+
+The loader should be given a command-line interface using the `Command` base class (``ckanext/command.py``).
+
+You need to add a line to the CKAN ``setup.py`` (under ``[console_scripts]``) and when you run ``python setup.py develop`` it creates a script for you in your Python environment.
+
+.. _load-data-harvester:
+
+Import Data with the Harvester Extension
+----------------------------------------
+
+The `CKAN harvester extension <https://bitbucket.org/okfn/ckanext-harvest/>`_ provides useful tools for more advanced data imports.
+
+These include a command-line interface and a web user interface for running harvesting jobs.
+
+To use the harvester extension, derive from the `base class of the harvester extension <https://bitbucket.org/okfn/ckanext-harvest/src/61844c8d2374/ckanext/harvest/harvesters/base.py>`_ and then write a custom ``_create_or_update_package`` method for your data.
+
+For more information on working with extensions, see :doc:`extensions`.
--- a/doc/model.rst Tue Jul 26 19:32:02 2011 +0100
+++ /dev/null Thu Jan 01 00:00:00 1970 +0000
@@ -1,96 +0,0 @@
-============================
-Making Changes to Model Code
-============================
-
-The structure of the CKAN data is described in the 'model'. This is in the code at::
-
- ckan/model
-
-Many of the domain objects are Revisioned and some are Stateful. These are concepts introduced by vdm.
-
-Migration
-=========
-
-When edits are made to the model code, then before the code can be used on a CKAN instance with existing data, the existing data has to be migrated. This is achieved with a migration script. CKAN currently uses `SqlAlchemy Migrate <http://code.google.com/p/sqlalchemy-migrate/>`_ to manage these scripts.
-
-When you deploy new code to a CKAN instance, as part of the process you run any required migration scripts with::
-
- paster db upgrade
-
-The scripts give their model version numbers in their filenames and are stored here::
-
- ckan/migration/versions/
-
-The current version the database is migrated to is also stored in the database. When you run the upgrade, as each migration script is run it prints to the console something like ``11->12``. If no upgrade is required because it is up to date, then nothing is printed.
-
-Creating a new migration script
-===============================
-
-A migration script should be checked into CKAN at the same time as the model changes it is related to. Before pushing the changes, ensure the tests pass when running against the migrated model, which requires the ``--ckan-migration`` setting - see `<README.html#migrationtesting>`_.
-
-To create a new migration script, create a python file in ckan/migration/versions/ and name it with a prefix numbered one higher than the previous one and some words describing the change.
-
-You need to use the special engine provided by the SqlAlchemy Migrate. Here is the standard header for your migrate script::
-
- from sqlalchemy import *
- from migrate import *
-
-
-The migration operations go in the upgrade function::
-
- def upgrade(migrate_engine):
- metadata = MetaData()
- metadata.bind = migrate_engine
-
-The following process should be followed when doing a migration. This process is here to make the process easier and to validate if any mistakes have been made.
-
-1. Get a dump of the database schema before you add your new migrate scripts.
-
- paster db clean
- paster db upgrade
- pg_dump -h host -s -f old.sql dbname
-
-2. Get a dump of the database as you have specified it in the model.
-
- paster db clean
- paster db create-test #this makes the database as defined in the model
- pg_dump -h host -s -f new.sql dbname
-
-3. Get agpdiff (apt-get it). It produces sql it thinks that you need to run on the database in order to get it to the updated schema.
-
- apgdiff old.sql new.sql > upgrade.diff
- (or if you don't want to install java use http://apgdiff.startnet.biz/diff_online.php)
-
-4. The upgrade.diff file created will have all the changes needed in sql. Delete the drop index lines as they are not created in the model.
-
-5. Put the resulting sql in your migrate script.
-
- eg migrate_engine.execute('''update table .........; update table ....''')
-
-6. Do a dump again, then a diff again to see if the the only thing left are drop index statements.
-
-7. run nosetests with --ckan-migration flag.
-
-Its that simple. Well almost..
-
-* If you are doing any table/field renaming adding that to your new migrate script first and use this as a base for your diff (i.e add a migrate script with these renaming before 1). This way the resulting sql wont try and drop and recreate the field/table!!
-* It sometimes drops the foreign key constraints in the wrong order causing an error so you may need to rearrange the order in the resulting upgrade.diff.
-* If you need to do any data transfer in the migrations then do it between the dropping of the constraints and adding of new ones.
-* May need to add some tests if you are doing data migrations.
-
-An example of a script doing it this way is 034_resource_group_table.py. This script copies the definitions of the original tables in order to do the renaming the tables/fields.
-
-In order to do some basic data migration testing extra assertions should be added to the migration script.
-
-Examples of this can also be found in 034_resource_group_table.py for example.
-
-This statement is run at the top of the migration script to get the count of rows::
-
- package_count = migrate_engine.execute('''select count(*) from package''').first()[0]
-
-And the following is run after to make sure that row count is the same::
-
- resource_group_after = migrate_engine.execute('''select count(*) from resource_group''').first()[0]
- assert resource_group_after == package_count
-
-
--- a/doc/paster.rst Tue Jul 26 19:32:02 2011 +0100
+++ b/doc/paster.rst Wed Jul 27 20:24:46 2011 +0100
@@ -101,7 +101,7 @@
Lets you initialise, upgrade, and dump database files in various formats.
-For example, to initialise the CKAN database, creating the tables that CKAN uses (you don't need to do this if you have run )::
+For example, to initialise the CKAN database, creating the tables that CKAN uses (note that you don't need to do this during setup if you have run ``create-test-data``)::
paster --plugin=ckan db init --config=/etc/ckan/std/std.ini
@@ -109,11 +109,7 @@
paster --plugin=ckan db upgrade --config=/etc/ckan/std/std.ini
-
-
-To write a simple dump to a CSV file::
-
- sudo paster --plugin=ckan db simple-dump-csv ./db.csv --config=/etc/ckan/std/std.ini
+For information on using ``db`` to create dumpfiles, see :doc:`database_dumps`.
ratings: Manage package ratings
--- a/doc/plugins.rst Tue Jul 26 19:32:02 2011 +0100
+++ b/doc/plugins.rst Wed Jul 27 20:24:46 2011 +0100
@@ -10,7 +10,7 @@
interfaces and workers. These work together to provide a simple mechanism to
extend core CKAN functionality.
-.. warning:: This is an advanced topic. At the moment, you need to have installed a developer version of CKAN to work with extensions, as described in :doc:`developer-install`. If you need help, contact the `ckan-discuss mailing list <http://lists.okfn.org/mailman/listinfo/ckan-discuss>`_.
+.. warning:: This is an advanced topic. At the moment, you need to have prepared your system to work with extensions, as described in :doc:`prepare-extensions`. We are working to make the most popular extensions more easily available as Debian packages.
.. note:: The terms **extension**, **plugin interface** and **worker** have very precise meanings: the use of the generic word **plugin** to describe any way in which CKAN might be extended is deprecated.
@@ -89,12 +89,35 @@
To build useful extensions you need to be able to "hook into" different parts
of CKAN in order to extend its functionality. You do this using CKAN's plugin
-architeture. We'll look at this in the next section.
+architecture. We'll look at this in the next section.
+Testing Extensions
+``````````````````
-If you do write a CKAN
-extension you may well want to publish it so others can use it too.
-See the `Publishing your extension`_ section below for details.
+CKAN extensions ordinarily have their own ``test.ini`` that refers to the CKAN ``test.ini``, so you can run them in exactly the same way. For example::
+
+ cd ckanext-dgu
+ nosetests ckanext/dgu/tests --ckan
+ nosetests ckanext/dgu/tests --ckan --with-pylons=test-core.ini
+
+To test your changes you'll need to use the ``paster serve`` command from the ``ckan`` directory:
+
+::
+
+ cd /home/ubuntu/pyenv/src/ckan
+ . ../../bin/activate
+ paster make-config ckan development.ini
+
+Then make any changes to the ``development.ini`` file that you need before continuing:
+
+::
+
+ paster db upgrade
+ paster serve --reload
+
+You should also make sure that your CKAN installation passes the developer tests, as described in :doc:`test`.
+
+Finally, if you write a CKAN, extension you may well want to publish it so others can use it too. See the `Publishing your extension`_ section below for details.
Plugins
-------
@@ -723,7 +746,7 @@
Working with CKANext Queue
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~
Rather than working with carrot publishers and consumers directly,
``ckanext-queue`` provides two useful Python objects to help you:
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/doc/post-installation.rst Wed Jul 27 20:24:46 2011 +0100
@@ -0,0 +1,63 @@
+========================
+Post-Installation Setup
+========================
+
+After you have completed installation (from either package or source), follow this section for instructions on setting up an initial user, loading test data, and notes on deploying CKAN.
+
+.. _create-admin-user:
+
+Create an Admin User
+====================
+
+By default, CKAN has a set of locked-down permissions. To begin
+working with it you need to set up a user and some permissions.
+
+First create an admin account from the command line (you must be root, ``sudo -s``):
+
+::
+
+ paster --plugin=ckan user add admin --config=/etc/ckan/std/std.ini
+
+When prompted, enter a password - this is the password you will use to log in to CKAN. In the resulting output, note that you will also get assigned a CKAN API key.
+
+.. note :: This command is your first introduction to some important CKAN concepts. **paster** is the script used to run CKAN commands. **std.ini** is the CKAN config file. You can change options in this file to configure CKAN.
+
+For exploratory purposes, you might was well make the ``admin`` user a
+sysadmin. You obviously wouldn't give most users these rights as they would
+then be able to do anything. You can make the ``admin`` user a sysadmin like
+this:
+
+::
+
+ paster --plugin=ckan sysadmin add admin --config=/etc/ckan/std/std.ini
+
+You can now login to the CKAN frontend with the username ``admin`` and the password you set up.
+
+.. _create-test-data:
+
+Load Test Data
+==============
+
+It can be handy to have some test data to start with. You can get test data like this:
+
+::
+
+ paster --plugin=ckan create-test-data --config=/etc/ckan/std/std.ini
+
+You now have a CKAN instance that you can log in to, with some test data to check everything
+works.
+
+.. _deployment-notes:
+
+Deployment
+==========
+
+You may want to deploy your CKAN instance at this point, to share with others.
+
+If you have installed CKAN from packages, then Apache and WSGI deployment scripts are already configured for you in standard locations.
+
+If you have installed CKAN from source, then the standard production deployment of CKAN is Apache and WSGI, which you will need to configure yourself. For more information, see http://wiki.ckan.net/Deployment
+
+CKAN has been successfully deployed by a variety of other methods including Apache reverse proxy + paster, nginx reverse proxy + paster, and nginx + uwsgi.
+
+You can now proceed to :doc:`theming`.
\ No newline at end of file
--- a/doc/preparation.rst Tue Jul 26 19:32:02 2011 +0100
+++ /dev/null Thu Jan 01 00:00:00 1970 +0000
@@ -1,426 +0,0 @@
-===================
-Prepare your System
-===================
-
-CKAN runs on Ubuntu 10.04 (either 64-bit or 32-bit). If you are already using Ubuntu 10.04, you can continue straight to :doc:`install`.
-
-However, if you're not, you should first set up Ubuntu 10.04. You have two options:
-
-* :ref:`using-virtualbox`. This is suitable if you want to host your CKAN instance on a machine running any other OS.
-* :ref:`using-amazon`. This is suitable if you want to host your CKAN instance in the cloud, on a readymade Ubuntu OS.
-
-
-.. _using-virtualbox:
-
-Option 1: Using VirtualBox
-==========================
-
-This option is suitable if you want to install CKAN on a machine running an OS other than Ubuntu 10.04. `VirtualBox <http://www.virtualbox.org>`_ lets you set up a virtual machine to run Ubuntu 10.04.
-
-Pre-requisites and Downloads
-----------------------------
-
-First, check your machine meets `the pre-requisites for VirtualBox <http://www.virtualbox.org/wiki/End-user_documentation>`_. These include a fairly recent processor and some spare memory.
-
-Then download the installation files.
-
-* `Download the VirtualBox installer <http://www.virtualbox.org/wiki/Downloads>`_.
-* `Download the Ubuntu image <http://www.ubuntu.com/download/ubuntu/download>`_ - make sure you choose Ubuntu 10.04, and 64-bit or 32-bit as appropriate for your current operating system.
-
-Install VirtualBox
-------------------
-
-.. note::
-
- This tutorial is for a Mac, but you can find instructions for installing VirtualBox on any OS `in the VirtualBox Manual <http://www.virtualbox.org/manual/ch02.html>`_.
-
-To install, double-click on the VirtualBox installer:
-
-.. image:: images/virtualbox1-package.png
- :width: 807px
- :alt: The VirtualBox installer - getting started
-
-Click Continue to begin the installation process. Enter your password when required, and wait for the installation to finish.
-
-Create Your Virtual Machine
----------------------------
-
-Go to Applications and open VirtualBox, then click New:
-
-.. image:: images/virtualbox4-newvm.png
- :width: 807px
- :alt: The VirtualBox installer - the New Virtual Machine Wizard
-
-Give your VM a name - we'll call ours ``ubuntu_ckan``. Under **OS Type**, choose **Linux** and **Ubuntu** (or **Ubuntu 64-bit** if you plan to install 64-bit Ubuntu).
-
-.. image:: images/virtualbox5-vmtype.png
- :width: 807px
- :alt: The VirtualBox installer - choosing your operating system
-
-Leave the memory size as 512MB, and choose **Create new hard disk**. This will open a new wizard:
-
-.. image:: images/virtualbox6-vmloc.png
- :width: 807px
- :alt: The VirtualBox installer - creating a new hard disk
-
-You can leave the defaults unchanged here too - click **Continue**, and then **Done**, and **Done** again, to create a new VM.
-
-Next, choose your VM from the left-hand menu, and click **Start**:
-
-.. image:: images/virtualbox7-startvm.png
- :width: 807px
- :alt: Starting your new VM
-
-This will open the First Run Wizard:
-
-.. image:: images/virtualbox8-firstrun.png
- :width: 807px
- :alt: The VirtualBox First Run Wizard
-
-After clicking **Continue**, you'll see **Select Installation Media**. This is where we need to tell our VM to boot from Ubuntu. Click on the file icon, and find your Ubuntu ``.iso`` file:
-
-.. image:: images/virtualbox9-iso.png
- :width: 807px
- :alt: When you get to Select Installation Media, choose your Ubuntu .iso file
-
-Click **Done**, wait for a few seconds, and you will see your Ubuntu VM booting.
-
-Set Up Ubuntu
---------------
-
-During boot, you will be asked if you want to try Ubuntu, or install it. Choose **Install Ubuntu**:
-
-.. image:: images/virtualbox11-ubuntu.png
- :width: 807px
- :alt: Booting Ubuntu - choose the Install Ubuntu option
-
-You can then follow the usual Ubuntu installation process.
-
-After Ubuntu is installed, from the main menu, choose **System > Administration > Update Manager**. You'll be asked if you want to install updates - say yes.
-
-When all the updates have been downloaded and installed, you'll be prompted to reboot Ubuntu.
-
-You can now proceed to :doc:`install`.
-
-.. _using-amazon:
-
-Option 2: Using Amazon EC2
-==========================
-
-
-If you prefer to run your CKAN instance in the cloud, you can use an Amazon EC2 instance, which is a fairly cheap and lightweight way to set up a server.
-
-Create an Amazon Account
-------------------------
-
-If you don't already have an Amazon AWS account you'll need to create one first. You can `create an Amazon AWS account for EC2 here <http://aws.amazon.com/ec2/>`_.
-
-Configure EC2
--------------
-
-Once you have an EC2 account, you'll need to configure settings for your CKAN instance.
-
-Start by logging into your `Amazon AWS Console <https://console.aws.amazon.com/s3/home>`_ and click on the EC2 tab.
-
-Select the region you want to run your CKAN instance in - the security group you set up is region-specific. In this tutorial, we use EU West, so it will be easier to follow if you do too.
-
-.. image :: images/1.png
-
-Set up a Security Group
-^^^^^^^^^^^^^^^^^^^^^^^
-
-Click the **Security Groups** link in the **My Resources** section in the right-hand side of the dashboard.
-
-.. image :: images/2.png
- :width: 807px
-
-Create a security group called ``web_test`` that gives access to ports 22, 80 and 5000 as shown below. This is needed so that you'll actually be able to access your server once it is created. You can't change these settings once the instance is running, so you need to do so now.
-
-.. image :: images/3a.png
- :width: 807px
-
-.. image :: images/3b.png
- :width: 807px
-
-Create a Keypair
-^^^^^^^^^^^^^^^^
-
-Now create a new keypair ``ckan_test`` to access your instance:
-
-.. image :: images/4.png
- :width: 807px
-
-When you click **Create**, your browser will prompt you to save a keypair called ``ckan_test.pem``:
-
-.. image :: images/5.png
- :width: 807px
-
-In this tutorial, we save the keypair in ``~/Downloads/ckan_test.pem``, but you should save it
-somewhere safe.
-
-.. note :: If you plan to boot your EC2 instance from the command line, you need to remember where you've put this file.
-
-
-Boot the EC2 Image
-------------------
-
-CKAN requires Ubuntu 10.04 to run (either the i386 or amd64
-architectures). Luckily Canonical provide a `range of suitable images <http://uec-images.ubuntu.com/releases/10.04/release/>`_.
-
-The cheapest EC2 instance is the micro one, but that isn't very powerful, so in this tutorial,
-we'll use the 32-bit small version.
-
-We're in ``eu-west-1`` and we'll use an instance-only image (i.e. all the data will be lost when you shut it down) so we need the `ami-3693a542 <https://console.aws.amazon.com/ec2/home?region=eu-west-1#launchAmi=ami-3693a542>`_ AMI.
-
-.. note ::
-
- There are more recent Ubuntu images at http://cloud.ubuntu.com/ami/ but we need the older 10.04 LTS release.
-
-At this point, you can either boot this image from the AWS
-console or launch it from the command line.
-
-
-Option 1: Boot the EC2 Image AMI via the AWS Console
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-From the EC2 dashboard, choose **Launch instance >**:
-
-.. image :: images/2.png
- :width: 807px
- :alt: Choose launch instance from the EC2 dashboard
-
-Now work through the wizard as shown in the following screenshots.
-
-In the first step search for ``ami-3693a542`` and select it from the results (it may take a few seconds for Amazon to find it).
-
-.. warning ::
-
- No image other than ``ami-3693a542`` will work with CKAN.
-
-.. image :: images/i1.png
- :width: 807px
- :alt: Search for image ami-3693a542
-
-You can keep the defaults for all of the following screens:
-
-.. image :: images/i2.png
- :width: 807px
- :alt: Keep the defaults while setting up your instance
-.. image :: images/i3.png
- :width: 807px
- :alt: Keep the defaults while setting up your instance
-.. image :: images/i4.png
- :width: 807px
- :alt: Keep the defaults while setting up your instance
-.. image :: images/i5.png
- :width: 807px
- :alt: Keep the defaults while setting up your instance
-
-Choose the ``web_test`` security group you created earlier:
-
-.. image :: images/i6.png
- :width: 807px
- :alt: Choose the web_test security group you created earlier
-
-Then finish the wizard:
-
-.. image :: images/i7.png
- :width: 807px
- :alt: Finish the wizard
-
-Finally click the **View your instances on the Instances page** link:
-
-.. image :: images/i8.png
- :width: 807px
- :alt: View your instance
-
-After a few seconds you'll see your instance has booted. Now skip to :ref:`log-in-to-instance`.
-
-Option 2: Boot the EC2 Image AMI from the Command Line
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-[You can skip this section if you've just booted from the AWS console and go straight to :ref:`log-in-to-instance`]
-
-To boot from the command line you still need the same information but you enter it in one command. I'll show you now.
-
-Install The EC2 Tools Locally
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-If you are on Linux, you can just install the tools like this:
-
-::
-
- sudo apt-get install ec2-ami-tools
- sudo apt-get install ec2-api-tools
-
-If you are on Windows or Mac you'll need to `download them from the Amazon website <http://aws.amazon.com/developertools/351>`_.
-
-Once the software is installed you can use the files you've just downloaded to do create your instance.
-
-Get Security Certificates
-~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Next click on the **Account** link, right at the top of the screen, and you'll see this screen:
-
-.. image :: images/6.png
- :width: 807px
- :alt: The Account screen
-
-From this screen choose **Security Credentials** from the left hand side. Once
-the page has loaded scroll down and you'll see the **Access Credentials**
-section. Click on the **X.509 Certificate** tab:
-
-.. image :: images/7.png
- :width: 807px
- :alt: The Access Credentials screen
-
-Here you'll be able to create an X.509 certificate and private key.
-
-.. tip ::
-
- You can only have two X.509 certificates at any given time, so you might need
- to inactivate an old one first and then delete it before you are allowed to
- create a new one, as in the screenshot above.
-
-Once you click the **Create New Certificate** link you get a popup which allows
-you to download the certificate and private key - do this. Once again, ours are in
-``~/Downloads``, but you should save it somewhere safe.
-
-.. image :: images/8.png
- :width: 807px
- :alt: Download your certificate
-
-.. tip ::
-
- Amazon will only give you a private key file once when you create it so
- although you can always go back to get a copy of the certificate, you can only
- get the private key once. Make sure you save it in a safe place.
-
-You now have:
-
-* Your private key (``pk-[ID].pem``)
-* Your certificate file (``cert-[ID].pem``)
-* Your new keypair (``ckan-test.pem``)
-
-The private key and the certificate files have the same name in the ``ID`` part.
-
-Create an Ubuntu Instance
-~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Once the tools are installed, run this command:
-
-::
-
- ec2-run-instances ami-3693a542 --instance-type m1.small --region eu-west-1 --group web_test \
- --key ckan_test \
- --private-key ~/Downloads/pk-[ID].pem \
- --cert ~/Downloads/cert-[ID].pem
-
-
-.. note ::
-
- The ``--key`` argument is the name of the keypair (``ckan_test``), not the certificate
- itself (``ckan_test.pem``).
-
-.. warning ::
-
- Amazon charge you for a minimum of one hour usage, so you shouldn't create and
- destroy lots of EC2 instances unless you want to be charged a lot.
-
-.. _log-in-to-instance:
-
-Log in to the Instance
-----------------------
-
-Once your instance has booted, you will need to find out its public DNS. Give it
-a second or two for the instance to load then browse to the running instance in
-the AWS console. If you tick your instance you'll be able to find the public
-DNS by scrolling down to the bottom of the **Description** tag.
-
-.. image :: images/8a.png
- :width: 807px
- :alt: Find the public DNS
-
-Here you can see that our public DNS is
-``ec2-79-125-86-107.eu-west-1.compute.amazonaws.com``. The private DNS only works
-from other EC2 instances so isn't any use to us.
-
-Once you've found your instance's public DNS, ensure the key has the correct permissions:
-
-::
-
- chmod 0600 "ckan_test.pem"
-
-You can then log in like this:
-
-::
-
- ssh -i ~/Downloads/ckan_test.pem ubuntu at ec2-46-51-149-132.eu-west-1.compute.amazonaws.com
-
-The first time you connect you'll see this, choose ``yes``:
-
-::
-
- RSA key fingerprint is 6c:7e:8d:a6:a5:49:75:4d:9e:05:2e:50:26:c9:4a:71.
- Are you sure you want to continue connecting (yes/no)? yes
- Warning: Permanently added 'ec2-79-125-86-107.eu-west-1.compute.amazonaws.com,79.125.86.107' (RSA) to the list of known hosts.
-
-When you log in you'll see a welcome message:
-
-::
-
- Linux ip-10-226-201-129 2.6.32-316-ec2 #31-Ubuntu SMP Wed May 18 14:09:06 UTC 2011 i686 GNU/Linux
- Ubuntu 10.04.2 LTS
-
- Welcome to Ubuntu!
- * Documentation: https://help.ubuntu.com/
-
- System information as of Wed Jun 22 20:13:11 UTC 2011
-
- System load: 0.01 Processes: 78
- Usage of /: 5.8% of 9.84GB Users logged in: 0
- Memory usage: 1% IP address for eth0: 10.226.201.129
- Swap usage: 0%
-
- Graph this data and manage this system at https://landscape.canonical.com/
- ---------------------------------------------------------------------
- At the moment, only the core of the system is installed. To tune the
- system to your needs, you can choose to install one or more
- predefined collections of software by running the following
- command:
-
- sudo tasksel --section server
- ---------------------------------------------------------------------
-
- 0 packages can be updated.
- 0 updates are security updates.
-
-
- The programs included with the Ubuntu system are free software;
- the exact distribution terms for each program are described in the
- individual files in /usr/share/doc/*/copyright.
-
- Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
- applicable law.
-
- To run a command as administrator (user "root"), use "sudo <command>".
- See "man sudo_root" for details.
-
- ubuntu at ip-10-226-201-129:~$
-
-You can now proceed to :doc:`install`.
-
-
-Note: Shutting Down Your Instance
----------------------------------
-
-If this is a test install of CKAN, when you have finished using CKAN, you can shut down your EC2 instance through the AWS console.
-
-.. warning ::
-
- Shutting down your EC2 instance will lose all your data.
-
-.. warning ::
-
- Amazon charge you for a minimum usage of one hour, so don't create and
- destroy lots of EC2 instances unless you want to be charged a lot!
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/doc/prepare-extensions.rst Wed Jul 27 20:24:46 2011 +0100
@@ -0,0 +1,31 @@
+===========================
+Prepare to Write Extensions
+===========================
+
+If you are running a package installation of CKAN, before you start using and testing extensions (described in :doc:`extensions`) you need to prepare your system.
+
+Firstly, you'll need to set up and enter a virtual Python environment, as follows:
+
+::
+
+ sudo apt-get install virtualenv pip mercurial
+ virtualenv /home/ubuntu/pyenv
+ . /home/ubuntu/pyenv/bin/activate
+
+Then, you need to install the CKAN source into your virtual environment. You can install CKAN like this:
+
+::
+
+ pip install -e hg+http://bitbucket.org/okfn/ckan#egg=ckan
+
+Your new CKAN developer install will be running on http://localhost:5000/
+
+When you start using extensions, you should install any of the developer versions of the CKAN extensions you want to work on like this (using the appropriate URL):
+
+::
+
+ pip install -e hg+http://bitbucket.org/okfn/<dependency-name>@<version>#egg=<egg-name>
+
+The dependency you've installed will appear in ``/home/ubuntu/pyenv/src/`` where you can work on it.
+
+After working on extensions, you should make sure that your deployment passes the tests, as described in :doc:`test`.
--- a/doc/test.rst Tue Jul 26 19:32:02 2011 +0100
+++ b/doc/test.rst Wed Jul 27 20:24:46 2011 +0100
@@ -2,50 +2,14 @@
Testing for Developers
======================
-After completing your developer installation of CKAN, you should check that tests run. Tests should be re-run (and amended if necessary) before checking in changes.
+Basic CKAN tests are run as described in :ref:`run-tests`.
-Preparing to Test
------------------
+This section describes advanced testing topics for developers, including migration testing and testing against PostgreSQL.
-Make sure you've created a config file at ``pyenv/ckan/development.ini``. Then activate the Python environment::
+Testing against PostgreSQL
+--------------------------
- . pyenv/bin/activate
-
-Install nose into your virtual environment if you haven't already::
-
- pip install --ignore-installed nose
-
-At this point you will need to deactivate and then re-activate your
-virtual environment to ensure that all the scripts point to the correct
-locations:
-
-::
-
- deactivate
- . pyenv/bin/activate
-
-
-Running Developer Tests
------------------------
-
-Here's how you start the quick development tests::
-
- cd pyenv/src/ckan
- nosetests ckan/tests --ckan
-
-You *must* run the tests from the CKAN directory as shown above, otherwise the
-``--ckan`` plugin won't work correctly.
-
-.. caution ::
-
- By default, the test run is 'quick and dirty' - only good enough as a check
- before committing code. See the next section for improved ways of running tests.
-
-
-Test Configurations
--------------------
-
-The default way to run tests is defined in ``test.ini`` (which is the default config file for nose - change it with option ``--with-pylons``). This specifies to use SQLite and sets ``faster_db_test_hacks``, which are compromises.
+The default way to run tests is defined in ``test.ini`` (which is the default config file for nose - change it with option ``--with-pylons``). This specifies using SQLite and sets ``faster_db_test_hacks``, which are compromises.
::
@@ -76,6 +40,9 @@
.. _migrationtesting:
+Migration Testing
+-----------------
+
If your changes require a model change, you'll need to write a migration script. To ensure this is tested as well, you should instead run the tests this way::
nosetests ckan/tests --ckan --ckan-migrate --with-pylons=test-core.ini
@@ -93,13 +60,4 @@
.. warning ::
- A common error when wanting to run tests against a particular database is to change ``sqlalchemy.url`` in ``test.ini`` or ``test-core.ini``. The problem is that these are versioned files and people have checked in these by mistake, creating problems for other developers and the CKAN buildbot. This is easily avoided by only changing ``sqlalchemy.url`` in your local ``development.ini`` and testing ``--with-pylons=test-core.ini``.
-
-Testing Extensions
-------------------
-
-CKAN extensions ordinarily have their own ``test.ini`` that refers to the CKAN ``test.ini``, so you can run them in exactly the same way. For example::
-
- cd ckanext-dgu
- nosetests ckanext/dgu/tests --ckan
- nosetests ckanext/dgu/tests --ckan --with-pylons=test-core.ini
\ No newline at end of file
+ A common error when wanting to run tests against a particular database is to change ``sqlalchemy.url`` in ``test.ini`` or ``test-core.ini``. The problem is that these are versioned files and people have checked in these by mistake, creating problems for other developers and the CKAN buildbot. This is easily avoided by only changing ``sqlalchemy.url`` in your local ``development.ini`` and testing ``--with-pylons=test-core.ini``.
\ No newline at end of file
http://bitbucket.org/okfn/ckan/changeset/c8db3937a349/
changeset: c8db3937a349
user: Anna PS
date: 2011-07-28 14:03:14
summary: Add details of using virtualenv for extensions
affected #: 1 file (211 bytes)
--- a/doc/extensions.rst Wed Jul 27 20:24:46 2011 +0100
+++ b/doc/extensions.rst Thu Jul 28 13:03:14 2011 +0100
@@ -11,6 +11,8 @@
Finding Extensions
------------------
+Many CKAN extensions are listed on the CKAN wiki's `List of Extensions <http://wiki.ckan.net/List_of_Extensions>`_. All CKAN extensions can be found at `OKFN's bitbucket page <https://bitbucket.org/okfn/>`_, prefaced with ``ckanext-``.
+
Some popular extensions include:
* `ckanext-admin <https://bitbucket.org/okfn/ckanext-admin>`_: Admin web interface for CKAN.
@@ -24,14 +26,16 @@
* `ckanext-stats <https://bitbucket.org/okfn/ckanext-stats>`_: Statistics (and visuals) about the datasets in a CKAN instance.
* `ckanext-wordpresser <https://bitbucket.org/okfn/ckanext-wordpresser>`_: CKAN plugin / WSGI middleware for combining CKAN with a Wordpress site.
-Many CKAN extensions are listed on the `CKAN wiki <http://wiki.ckan.net/Main_Page>`_. All CKAN extensions can be found at `OKFN's bitbucket page <https://bitbucket.org/okfn/>`_, prefaced with ``ckanext-``.
-
Installing an Extension
-----------------------
-To install an extension on a CKAN instance:
+You can install an extension on a CKAN instance as follows.
-1. Install the extension package code using ``pip``.
+1. First, ensure you are working within your virtualenv (see :doc:`prepare-extensions` if you are not sure what this means)::
+
+ . /home/ubuntu/pyenv/bin/activate
+
+2. Install the extension package code using ``pip``.
For example, to install the Disqus extension, which allows users to comment on datasets::
@@ -43,7 +47,7 @@
The dependency you've installed will appear in the ``src/`` directory under your Python environment.
-2. Add the names of any plugin implementations the extension uses to the CKAN
+3. Add the names of any plugin implementations the extension uses to the CKAN
config file. You can find these in the plugin's ``setup.py`` file under ``[ckan.plugins]``.
The config plugins variable is in the '[app:main]' section under 'ckan.plugins'. e.g.::
@@ -55,7 +59,7 @@
ckan.plugins = disqus amqp myplugin
-3. If necessary, restart WSGI, which usually means restarting Apache::
+4. If necessary, restart WSGI, which usually means restarting Apache::
sudo /etc/init.d/apache2 restart
http://bitbucket.org/okfn/ckan/changeset/27def0bebcdd/
changeset: 27def0bebcdd
user: Anna PS
date: 2011-07-28 14:03:36
summary: Move details of standard tests back to main test chapter
affected #: 1 file (1001 bytes)
--- a/doc/test.rst Thu Jul 28 13:03:14 2011 +0100
+++ b/doc/test.rst Thu Jul 28 13:03:36 2011 +0100
@@ -2,9 +2,46 @@
Testing for Developers
======================
-Basic CKAN tests are run as described in :ref:`run-tests`.
+If you are installing CKAN from source, or developing extensions, then you need to know how to run CKAN tests.
-This section describes advanced testing topics for developers, including migration testing and testing against PostgreSQL.
+This section describes testing topics for developers, including basic tests, migration testing and testing against PostgreSQL.
+
+.. _basic-tests:
+
+Basic Tests
+-----------
+
+After completing your source installation of CKAN, you should check that tests pass. You should also check this before checking in changes to CKAN code.
+
+Make sure you've created a config file at ``pyenv/ckan/development.ini``. Then activate the Python environment::
+
+ . pyenv/bin/activate
+
+Install nose into your virtual environment::
+
+ pip install --ignore-installed nose
+
+At this point you will need to deactivate and then re-activate your
+virtual environment to ensure that all the scripts point to the correct
+locations:
+
+::
+
+ deactivate
+ . pyenv/bin/activate
+
+Then run the quick development tests::
+
+ cd pyenv/src/ckan
+ nosetests ckan/tests --ckan
+
+You *must* run the tests from the CKAN directory as shown above, otherwise the
+``--ckan`` plugin won't work correctly.
+
+.. warning ::
+
+ By default, the test run is 'quick and dirty' - only good enough as an initial check.
+
Testing against PostgreSQL
--------------------------
http://bitbucket.org/okfn/ckan/changeset/8d9d8b7bdfbd/
changeset: 8d9d8b7bdfbd
user: Anna PS
date: 2011-07-28 14:04:51
summary: Update package install instructions.
Correct to 64-bit only, correct missing link, clarify re Ubuntu requirements.
affected #: 1 file (59 bytes)
--- a/doc/install-from-package.rst Thu Jul 28 13:03:36 2011 +0100
+++ b/doc/install-from-package.rst Thu Jul 28 13:04:51 2011 +0100
@@ -4,21 +4,22 @@
This section describes how to install CKAN from packages. This is the recommended and by far the easiest way to install CKAN.
-It requires you to use Ubuntu 10.04. If you don't have access to Ubuntu 10.04, we provide support for two options:
+Package install requires you to use 64-bit Ubuntu 10.04: either locally, through a virtual machine or Amazon EC2. Your options are as follows:
+* Using 64-bit Ubuntu 10.04 directly.
* :ref:`using-virtualbox`. This is suitable if you want to host your CKAN instance on a machine running any other OS.
-* :ref:`using-amazon`. This is suitable if you want to host your CKAN instance in the cloud, on a readymade Ubuntu OS.
+* :ref:`using-amazon`. This is suitable if you want to host your CKAN instance in the cloud, on a ready-made Ubuntu OS.
-.. note:: We recommend you use this method unless you are an experimental user, a core CKAN developer, or you have no access to Ubuntu 10.04 through either of the methods above, in which
+.. note:: We recommend you use package installation unless you are a core CKAN developer or have no access to Ubuntu 10.04 through any of the methods above, in which case, you should use :doc:`install-from-source`.
For support during installation, please contact `the ckan-dev mailing list <http://lists.okfn.org/mailman/listinfo/ckan-dev>`_.
Prepare your System
--------------------
-CKAN runs on Ubuntu 10.04 (either 64-bit or 32-bit). If you are already using Ubuntu 10.04, you can continue straight to :doc:`install`.
+CKAN runs on 64-bit Ubuntu 10.04. If you are already using Ubuntu 10.04, you can continue straight to :ref:`run-package-installer`.
-However, if you're not, you can either use VirtualBox to set up an Ubuntu VM, or an Amazon EC2 instance.
+However, if you're not, you can either use VirtualBox to set up an Ubuntu VM on Windows, Linux, Macintosh and Solaris. Alternatively, you can use an Amazon EC2 instance.
.. _using-virtualbox:
@@ -35,7 +36,7 @@
Then download the installation files.
* `Download the VirtualBox installer <http://www.virtualbox.org/wiki/Downloads>`_.
-* `Download the Ubuntu image <http://www.ubuntu.com/download/ubuntu/download>`_ - make sure you choose Ubuntu 10.04, and 64-bit or 32-bit as appropriate for your current operating system.
+* `Download the Ubuntu image <http://www.ubuntu.com/download/ubuntu/download>`_ - make sure you choose Ubuntu 10.04 and 64-bit.
Install VirtualBox
******************
@@ -61,7 +62,7 @@
:width: 807px
:alt: The VirtualBox installer - the New Virtual Machine Wizard
-Give your VM a name - we'll call ours ``ubuntu_ckan``. Under **OS Type**, choose **Linux** and **Ubuntu** (or **Ubuntu 64-bit** if you plan to install 64-bit Ubuntu).
+Give your VM a name - we'll call ours ``ubuntu_ckan``. Under **OS Type**, choose **Linux** and **Ubuntu 64-bit**.
.. image:: images/virtualbox5-vmtype.png
:width: 807px
@@ -110,7 +111,7 @@
When all the updates have been downloaded and installed, you'll be prompted to reboot Ubuntu.
-At this point, you can begin installing CKAN.
+At this point, you can proceed to :ref:`run-package-installer`.
.. _using-amazon:
http://bitbucket.org/okfn/ckan/changeset/34e4eb62c6a2/
changeset: 34e4eb62c6a2
user: Anna PS
date: 2011-07-28 14:05:29
summary: Update source install instructions.
Add links to standard packages, and remove test information - link to test chapter instead.
affected #: 1 file (352 bytes)
--- a/doc/install-from-source.rst Thu Jul 28 13:04:51 2011 +0100
+++ b/doc/install-from-source.rst Thu Jul 28 13:05:29 2011 +0100
@@ -4,7 +4,7 @@
This section describes how to install CKAN from source. This removes the requirement for Ubuntu 10.04 that exists with :doc:`install-from-package`.
-.. warning:: Installing from source is complex and recommended only for users with no access to Ubuntu 10.04 and core CKAN developers.
+.. warning:: Installing from source is complex and recommended only for CKAN developers or for distributions other than Ubuntu 10.04, for which these instructions provide a basis.
For support during installation, please contact `the ckan-dev mailing list <http://lists.okfn.org/mailman/listinfo/ckan-dev>`_.
@@ -19,32 +19,36 @@
1. Ensure the required packages are installed.
- ===================== ===============================================
- Package Description
- ===================== ===============================================
- mercurial Source control
- python-dev Python interpreter v2.5 - v2.7 and dev headers
- postgresql PostgreSQL database
- libpq-dev PostgreSQL library
- python-psycopg2 PostgreSQL python module
- libxml2-dev XML library development files
- libxslt-dev XSLT library development files
- python-virtualenv Python virtual environments
- wget Command line tool for downloading from the web
- build-essential Tools for building source code
- git-core Git source control (for getting MarkupSafe src)
- subversion Subversion source control (for pyutilib)
- ===================== ===============================================
+ If you have access to ``apt-get``, you can install these packages as follows:
- If you have access to ``apt-get``, you can install these packages as follows:
-
::
-
+
sudo apt-get install build-essential libxml2-dev libxslt-dev
sudo apt-get install wget mercurial postgresql libpq-dev git-core
sudo apt-get install python-dev python-psycopg2 python-virtualenv
sudo apt-get install subversion
+
+ Otherwise, you should install these packages from source.
+
+ ===================== ===============================================
+ Package Description
+ ===================== ===============================================
+ mercurial `Source control <http://mercurial.selenic.com/>`_
+ python `Python interpreter v2.5 - v2.7 and dev headers <http://www.python.org/getit/>`_
+ postgresql `PostgreSQL database <http://www.postgresql.org/download/>`_
+ libpq `PostgreSQL library <http://www.postgresql.org/docs/8.1/static/libpq.html>`_
+ psycopg2 `PostgreSQL python module <http://initd.org/psycopg/install/>`_
+ libxml2 `XML library development files <http://xmlsoft.org/>`_
+ libxslt `XSLT library development files <http://www.linuxfromscratch.org/blfs/view/6.3/general/libxslt.html>`_
+ virtualenv `Python virtual environments <http://pypi.python.org/pypi/virtualenv>`_
+ wget `Command line tool for downloading from the web <http://www.gnu.org/s/wget/>`_
+ build-essential Tools for building source code
+ git `Git source control (for getting MarkupSafe src) <http://book.git-scm.com/2_installing_git.html>`_
+ subversion `Subversion source control (for pyutilib) <http://subversion.apache.org/packages.html>`_
+ ===================== ===============================================
+
+
2. Create a Python virtual environment.
In your home directory run the command below. It is currently important to
@@ -261,41 +265,6 @@
The CKAN homepage should load.
-.. _run-tests:
-
-Run Tests
----------
-
-After completing your source installation of CKAN, you should check that tests run. Tests should be re-run (and amended if necessary) before checking in changes.
-
-Make sure you've created a config file at ``pyenv/ckan/development.ini``. Then activate the Python environment::
-
- . pyenv/bin/activate
-
-Install nose into your virtual environment::
-
- pip install --ignore-installed nose
-
-At this point you will need to deactivate and then re-activate your
-virtual environment to ensure that all the scripts point to the correct
-locations:
-
-::
-
- deactivate
- . pyenv/bin/activate
-
-Then run the quick development tests::
-
- cd pyenv/src/ckan
- nosetests ckan/tests --ckan
-
-You *must* run the tests from the CKAN directory as shown above, otherwise the
-``--ckan`` plugin won't work correctly.
-
-.. caution ::
-
- By default, the test run is 'quick and dirty' - only good enough as an initial check.
- See :doc:`test` for information on full developer tests.
+Finally, make sure that tests pass, as described in :ref:`basic-tests`.
You can now proceed to :doc:`post-installation`.
http://bitbucket.org/okfn/ckan/changeset/e5039135cf4f/
changeset: e5039135cf4f
user: Anna PS
date: 2011-07-28 15:38:40
summary: Amend title of prepare-for-extensions section
affected #: 1 file (6 bytes)
--- a/doc/prepare-extensions.rst Thu Jul 28 13:05:29 2011 +0100
+++ b/doc/prepare-extensions.rst Thu Jul 28 14:38:40 2011 +0100
@@ -1,6 +1,6 @@
-===========================
-Prepare to Write Extensions
-===========================
+=========================
+Prepare to Use Extensions
+=========================
If you are running a package installation of CKAN, before you start using and testing extensions (described in :doc:`extensions`) you need to prepare your system.
http://bitbucket.org/okfn/ckan/changeset/0c7051133744/
changeset: 0c7051133744
user: Anna PS
date: 2011-07-28 15:39:11
summary: Add Xcode note to install-from-source section
affected #: 1 file (2 bytes)
--- a/doc/install-from-source.rst Thu Jul 28 14:38:40 2011 +0100
+++ b/doc/install-from-source.rst Thu Jul 28 14:39:11 2011 +0100
@@ -34,7 +34,7 @@
Package Description
===================== ===============================================
mercurial `Source control <http://mercurial.selenic.com/>`_
- python `Python interpreter v2.5 - v2.7 and dev headers <http://www.python.org/getit/>`_
+ python `Python v2.5-2.7 <http://www.python.org/getit/>`_
postgresql `PostgreSQL database <http://www.postgresql.org/download/>`_
libpq `PostgreSQL library <http://www.postgresql.org/docs/8.1/static/libpq.html>`_
psycopg2 `PostgreSQL python module <http://initd.org/psycopg/install/>`_
@@ -42,7 +42,7 @@
libxslt `XSLT library development files <http://www.linuxfromscratch.org/blfs/view/6.3/general/libxslt.html>`_
virtualenv `Python virtual environments <http://pypi.python.org/pypi/virtualenv>`_
wget `Command line tool for downloading from the web <http://www.gnu.org/s/wget/>`_
- build-essential Tools for building source code
+ build-essential Tools for building source code (or up-to-date Xcode on Mac)
git `Git source control (for getting MarkupSafe src) <http://book.git-scm.com/2_installing_git.html>`_
subversion `Subversion source control (for pyutilib) <http://subversion.apache.org/packages.html>`_
===================== ===============================================
http://bitbucket.org/okfn/ckan/changeset/4d3a9ef497b2/
changeset: 4d3a9ef497b2
user: Anna PS
date: 2011-07-28 15:40:49
summary: Make install-from-source warning less scary
affected #: 1 file (35 bytes)
--- a/doc/install-from-source.rst Thu Jul 28 14:39:11 2011 +0100
+++ b/doc/install-from-source.rst Thu Jul 28 14:40:49 2011 +0100
@@ -4,7 +4,7 @@
This section describes how to install CKAN from source. This removes the requirement for Ubuntu 10.04 that exists with :doc:`install-from-package`.
-.. warning:: Installing from source is complex and recommended only for CKAN developers or for distributions other than Ubuntu 10.04, for which these instructions provide a basis.
+.. warning:: This option is more complex than :doc:`install-from-package`, so we suggest only doing it this way if you plan to work on CKAN core, or have no access to Ubuntu 10.04 via any of the suggested methods.
For support during installation, please contact `the ckan-dev mailing list <http://lists.okfn.org/mailman/listinfo/ckan-dev>`_.
http://bitbucket.org/okfn/ckan/changeset/42bf62a5539b/
changeset: 42bf62a5539b
branch: release-v1.4.3
user: dread
date: 2011-07-28 18:18:48
summary: [merge] docs revamp from annaps.
affected #: 71 files (90.2 KB)
--- a/README.txt Thu Jul 28 16:45:02 2011 +0100
+++ b/README.txt Thu Jul 28 17:18:48 2011 +0100
@@ -1,443 +1,10 @@
README
++++++
-Introduction
-============
+Welcome to CKAN.
-Comprehensive Knowledge Archive Network (CKAN) Software.
+For installation instructions, see http://docs.ckan.org/install-from-source.html
-See :mod:`ckan.__long_description__` for more information.
+For background information and developer notes, see http://wiki.ckan.net
-
-Developer Installation
-======================
-
-These are instructions to get developing with CKAN. Instructions for deploying
-CKAN to a server are at: :doc:`deployment` (doc/deployment.rst).
-
-Before you start it may be worth checking CKAN has passed the auto build and
-tests. See: http://buildbot.okfn.org/waterfall
-
-
-1. Ensure these packages are installed:
-
- ===================== ===============================================
- Package Description
- ===================== ===============================================
- mercurial Source control
- python-dev Python interpreter v2.5 - v2.7 and dev headers
- postgresql PostgreSQL database
- libpq-dev PostgreSQL library
- python-psycopg2 PostgreSQL python module
- libxml2-dev XML library development files
- libxslt-dev XSLT library development files
- python-virtualenv Python virtual environments
- wget Command line tool for downloading from the web
- build-essential Tools for building source code
- git-core Git source control (for getting MarkupSafe src)
- subversion Subversion source control (for pyutilib)
- ===================== ===============================================
-
- For Ubuntu you can install these like so:
-
- ::
-
- sudo apt-get install build-essential libxml2-dev libxslt-dev
- sudo apt-get install wget mercurial postgresql libpq-dev git-core
- sudo apt-get install python-dev python-psycopg2 python-virtualenv
- sudo apt-get install subversion
-
-2. Create a python virtual environment
-
- In your home directory run the command below. It is currently important to
- call your virtual environment ``pyenv`` so that the automated deployment tools
- work correctly.
-
- ::
-
- cd ~
- virtualenv pyenv
-
- .. tip ::
-
- If you don't have a ``python-virtualenv`` package in your distribution
- you can get a ``virtualenv.py`` script from within the
- `virtualenv source distribution <http://pypi.python.org/pypi/virtualenv/>`_
- and then run ``python virtualenv.py pyenv`` instead.
-
- To help with automatically installing CKAN dependencies we use a tool
- called ``pip``. Make sure you have activated your environment (see step 3)
- and then install it from an activated shell like this:
-
- ::
-
- easy_install pip
-
-3. Activate your virtual environment
-
- To work with CKAN it is best to adjust your shell settings so that your
- shell uses the virtual environment you just created. You can do this like
- so:
-
- ::
-
- . pyenv/bin/activate
-
- When your shell is activated you will see the prompt change to something
- like this:
-
- ::
-
- (pyenv)[ckan at host ~/]$
-
- An activated shell looks in your virtual environment first when choosing
- which commands to run. If you enter ``python`` now it will actually
- run ``~/pyenv/bin/python`` which is what you want.
-
-4. Install CKAN code and required Python packages into the new environment
-
- First you'll need to install CKAN. For the latest version run:
-
- ::
-
- pip install --ignore-installed -e hg+http://bitbucket.org/okfn/ckan#egg=ckan
-
- CKAN has dependent packages it requires you install. They are listed in the three pyenv/src/ckan/requires files.
-
- If you are running Ubuntu Lucid then it is recommended you take advantage of the Lucid packages to cover the dependencies listed in lucid_present.txt. In this case you should install the Lucid present packages::
-
- sudo apt-get install python-psycopg2 python-lxml python-sphinx
- sudo apt-get install python-pylons python-formalchemy python-repoze.who
- sudo apt-get install python-repoze.who-plugins python-tempita python-zope.interface
-
- And the remaining (non-Lucid) packages are installed like this::
-
- pip install --ignore-installed -r pyenv/src/ckan/requires/lucid_missing.txt -r pyenv/src/ckan/requires/lucid_conflict.txt
-
- If you are not using Ubuntu Lucid then should install all three sets of dependencies with this single command::
-
- pip install --ignore-installed -r pyenv/src/ckan/requires/lucid_present.txt -r pyenv/src/ckan/requires/lucid_missing.txt -r pyenv/src/ckan/requires/lucid_conflict.txt
-
- This will take a **long** time. Particularly the install of the ``lxml``
- package.
-
- The ``--ignore-installed`` option ensures ``pip`` installs software into
- this virtual environment even if it is already present on the system. Because of this, to ensure you get a mutually agreeable version of WebOb installed, you shouldn't install each requirements file on its own with this option.
-
- At this point you will need to deactivate and then re-activate your
- virtual environment to ensure that all the scripts point to the correct
- locations:
-
- ::
-
- deactivate
- . pyenv/bin/activate
-
-5. Setup a PostgreSQL database
-
- List existing databases:
-
- ::
-
- psql -l
-
- It is advisable to ensure that the encoding of databases is 'UTF8', or
- internationalisation may be a problem. Since changing the encoding of PostgreSQL
- may mean deleting existing databases, it is suggested that this is fixed before
- continuing with the CKAN install.
-
- Next you'll need to create a database user if one doesn't already exist.
-
- .. tip ::
-
- If you choose a database name, user or password which are different from those
- suggested below then you'll need to update the configuration file you'll create in
- the next step.
-
- Here we choose ``ckantest`` as the database and ``ckanuser`` as the user:
-
- ::
-
- sudo -u postgres createuser -S -D -R -P ckantest
-
- It should prompt you for a new password for the CKAN data in the database.
- It is suggested you enter ``pass`` for the password.
-
- Now create the database, which we'll call ``ckantest`` (the last argument):
-
- ::
-
- sudo -u postgres createdb -O ckantest ckantest
-
-6. Create a CKAN config file
-
- Make sure you are in an activated environment (see step 3) so that Python
- Paste and other modules are put on the python path (your command prompt will
- start with ``(pyenv)`` if you have) then change into the ``ckan`` directory
- which will have been created when you installed CKAN in step 4 and create the
- config file ``development.ini`` using Paste:
-
- ::
-
- cd pyenv/src/ckan
- paster make-config ckan development.ini
-
- You can give your config file a different name but the tests will expect you
- to have used ``development.ini`` so it is strongly recommended you use this
- name, at least to start with.
-
- If you used a different database name or password when creating the database
- in step 5 you'll need to now edit ``development.ini`` and change the
- ``sqlalchemy.url`` line, filling in the database name, user and password you used.
-
- ::
-
- sqlalchemy.url = postgresql://ckantest:pass@localhost/ckantest
-
- Other configuration, such as setting the language of the site or editing the
- visual theme are described in :doc:`configuration` (doc/configuration.rst)
-
- .. note ::
-
- If the paster command gives you ``ImportError: cannot import name UnicodeMultiDict`` then you have gotten a version of WebOb that is too new for our Pylons. This usually occurs if you install Remove WebOb and reinstall version 1.0.8.
-
- .. caution ::
-
- Advanced users: If you are using CKAN's fab file capability you currently need to create
- your config file as ``pyenv/ckan.net.ini`` so you will probably have
- ignored the advice about creating a ``development.ini`` file in the
- ``pyenv/src/ckan`` directory. This is fine but CKAN probably won't be
- able to find your ``who.ini`` file. To fix this edit ``pyenv/ckan.net.ini``,
- search for the line ``who.config_file = %(here)s/who.ini`` and change it
- to ``who.config_file = who.ini``.
-
- We are moving to a new deployment system where this incompatibility
- will be fixed.
-
-7. Create database tables
-
- Now that you have a configuration file that has the correct settings for
- your database, you'll need to create the tables. Make sure you are still in an
- activated environment with ``(pyenv)`` at the front of the command prompt and
- then from the ``pyenv/src/ckan`` directory run this command:
-
- ::
-
- paster db init
-
- You should see ``Initialising DB: SUCCESS``. If you are not in the
- ``pyenv/src/ckan`` directory or you don't have an activated shell, the command
- will not work.
-
- If the command prompts for a password it is likely you haven't set up the
- database configuration correctly in step 6.
-
-8. Create the cache directory
-
- You need to create the Pylon's cache directory specified by 'cache_dir'
- in the config file.
-
- (from the ``pyenv/src/ckan`` directory):
-
- ::
-
- mkdir data
-
-
-9. Run the CKAN webserver
-
- NB If you've started a new shell, you'll have to activate the environment
- again first - see step 3.
-
- (from the pyenv/src/ckan directory):
-
- ::
-
- paster serve development.ini
-
-10. Point your web browser at: http://127.0.0.1:5000/
-
- The CKAN homepage should load without problem.
-
-If you ever want to upgrade to a more recent version of CKAN, read the
-``UPGRADE.txt`` file in ``pyenv/src/ckan/``.
-
-Test
-====
-
-Setting up to test
-------------------
-
-Make sure you've created a config file: ``pyenv/ckan/development.ini``
-
-Ensure you have activated the environment::
-
- . pyenv/bin/activate
-
-
-Install nose into your virtual environment if you haven't already::
-
- pip install --ignore-installed nose
-
-At this point you will need to deactivate and then re-activate your
-virtual environment to ensure that all the scripts point to the correct
-locations:
-
-::
-
- deactivate
- . pyenv/bin/activate
-
-
-Running developer tests
------------------------
-
-Here's how you start the quick development tests::
-
- cd pyenv/src/ckan
- nosetests ckan/tests --ckan
-
-You *must* run the tests from the CKAN directory as shown above, otherwise the
-``--ckan`` plugin won't work correctly.
-
-.. caution ::
-
- By default, the test run is 'quick and dirty' - only good enough as a check
- before committing code. See the next section for improved ways of running tests.
-
-
-Test configurations
--------------------
-
-The default way to run tests is defined in test.ini (which is the default config file for nose - change it with option "--with-pylons"). This specifies to use Sqlite and sets faster_db_test_hacks, which are compromises.
-
-::
-
- cd pyenv/src/ckan
- nosetests ckan/tests --ckan
-
-Although Sqlite is useful for testing a large proportion of CKAN, actually in deployment, CKAN must run with PostgreSQL. Running the tests against PosgreSQL is slower but more thorough for two reasons:
- 1. You test subtleties of PostgreSQL
- 2. CKAN's default search relies on PostgreSQL's custom Full Text Search, so these (100 or so) tests are skipped when running against Sqlite.
-
-So when making changes to anything involved with search or closely related to the database, it is wise to test against PostgreSQL.
-
-To test against PosgreSQL:
- 1. Edit your local development.ini to specify a PostgreSQL database with the `sqlalchemy.url` parameter.
- 2. Tell nose to use test-core.ini (which imports settings from development.ini)
-
-::
-
- nosetests ckan/tests --ckan --with-pylons=test-core.ini
-
-The test suite takes a long time to run against standard PostgreSQL (approx. 15 minutes, or close to an hour on Ubuntu/10.04 Lucid).
-
-This can be improved to between 5 and 15 minutes by running PostgreSQL in memory and turning off durability, as described at <http://www.postgresql.org/docs/9.0/static/non-durability.html>.
-
-.. _migrationtesting:
-
-If your changes require a model change, you'll need to write a migration script. To ensure this is tested as well, you should instead run the tests this way::
-
- nosetests ckan/tests --ckan --ckan-migrate --with-pylons=test-core.ini
-
-By default, tests are run using the model defined in ckan/model, but by using the ``--ckan-migrate`` option the tests will run using a database that has been created using the migration scripts, which is the way the database is created and upgraded in production. These tests are the most thorough and will take around 20 minutes.
-
-.. caution ::
-
- Ordinarily, you should set ``development.ini`` to specify a PostgreSQL database
- so these also get used when running ``test-core.ini``, since ``test-core.ini``
- inherits from ``development.ini``. If you were to change the ``sqlalchemy.url``
- option in your ``development.ini`` file to use SQLite, the command above would
- actually test SQLite rather than PostgreSQL so always check the setting in
- ``development.ini`` to ensure you are running the full tests.
-
-.. note ::
-
- A common error when wanting to run tests against a particular database is to change the sqlalchemy.url in test.ini or test-core.ini. The problem is that these are versioned files and people have checked in these by mistake, creating problems for all other developers and the buildbot. This is easily avoided by only changing the sqlalchemy.url in your local development.ini and testing --with-pylons=test-core.ini.
-
-Common problems running tests
------------------------------
-
-* `nose.config.ConfigError: Error reading config file 'setup.cfg': no such option 'with-pylons'`
-
- This error can result when you run nosetests for two reasons:
-
- 1. Pylons nose plugin failed to run. If this is the case, then within a couple of lines of running `nosetests` you'll see this warning: `Unable to load plugin pylons` followed by an error message. Fix the error here first.
-
- 2. The Python module 'Pylons' is not installed into you Python environment. Confirm this with::
-
- python -c "import pylons"
-
-* `OperationalError: (OperationalError) no such function: plainto_tsquery ...`
-
- This error usually results from running a test which involves search functionality, which requires using a PostgreSQL database, but another (such as SQLite) is configured. The particular test is either missing a `@search_related` decorator or there is a mixup with the test configuration files leading to the wrong database being used.
-
-
-Testing extensions
-------------------
-
-CKAN extensions ordinarily have their own test.ini that refers to the ckan test.ini, so you can run them in exactly the same way. For example::
-
- cd ckanext-dgu
- nosetests ckanext/dgu/tests --ckan
- nosetests ckanext/dgu/tests --ckan --with-pylons=test-core.ini
-
-
-Development
-===========
-
-CKAN is an open source project and contributions are welcome!
-
-There are a number of stakeholders in the direction of the project, so we discuss large changes and new features on the ckan-discuss list: http://lists.okfn.org/mailman/listinfo/ckan-discuss
-
-New developers should aquaint themselves with the documentation (see below). Proposed changes should be made on a personal CKAN fork (on BitBucket for example). Request merging with the mainline via the ckan-discuss list.
-
-We have policies for check-ins that ensure the build doesn't break etc. on http://ckan.org/#ProjectProcessesandPolicies which should be followed unless someone builds concensus to change it.
-
-
-Documentation
-=============
-
-The home page for the CKAN project is: http://ckan.org
-
-This README file is part of the Developer Documentation, viewable at:
-http://packages.python.org/ckan/ and stored in the CKAN
-repo at ``ckan/doc``.
-
-The Developer Docs are built using `Sphinx <http://sphinx.pocoo.org/>`_:
-
- python setup.py build_sphinx
-
-(An admin might upload the resulting html to packages.python.org/ckan/ by doing: `easy_install sphinx-pypi-upload` and `python setup.py upload_sphinx`)
-
-(The docs are also uploaded via dav to http://knowledgeforge.net/ckan/doc/ckan/for backwards compatability).
-
-
-Contributors
-============
-
- * Rufus Pollock <rufus [at] rufuspollock [dot] org>
- * David Read
- * John Bywater
- * Nick Stenning (css and js)
-
-Also especial thanks to the following projects without whom this would not have
-been possible:
-
- * CKAN logo: "angry hamster" http://www.maedelmaedel.com/ and
- http://www.villainous.biz/
- * famfamfam.com for silk icons <http://www.famfamfam.com/lab/icons/silk/>
- * Pylons: <http://pylonshq.com/>
- * Python: <http://www.python.org>
-
-
-Copying and License
-===================
-
-This material is copyright (c) 2006-2011 Open Knowledge Foundation.
-
-It is open and licensed under the GNU Affero General Public License (AGPL) v3.0
-whose full text may be found at:
-
-<http://www.fsf.org/licensing/licenses/agpl-3.0.html>
-
+For help during installation, contact ckan-dev at lists.okfn.org
\ No newline at end of file
--- a/doc/README.rst Thu Jul 28 16:45:02 2011 +0100
+++ /dev/null Thu Jan 01 00:00:00 1970 +0000
@@ -1,1 +0,0 @@
-../README.txt
\ No newline at end of file
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/doc/_templates/layout.html Thu Jul 28 17:18:48 2011 +0100
@@ -0,0 +1,20 @@
+{% extends '!layout.html' %}
+
+{% block footer %}
+ <div class="footer">
+ {% if hasdoc('copyright') %}
+ {% trans path=pathto('copyright'), copyright=copyright|e %}© <a href="{{ path }}">Copyright</a> {{ copyright }}.{% endtrans %}
+ {% else %}
+ {% trans copyright=copyright|e %}© Copyright {{ copyright }}.{% endtrans %}
+ {% endif %}
+ {% if last_updated %}
+ {% trans last_updated=last_updated|e %}Last updated on {{ last_updated }}.{% endtrans %}
+ {% endif %}
+ {% if show_sphinx %}
+ {% trans sphinx_version=sphinx_version|e %}Created using <a href="http://sphinx.pocoo.org/">Sphinx</a> {{ sphinx_version }}.{% endtrans %}
+ {% endif %}
+ <br/>
+ Please contact the <a href="http://lists.okfn.org/mailman/listinfo/ckan-dev">ckan-dev mailing list</a>
+ with any questions.
+ </div>
+{% endblock %}
\ No newline at end of file
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/doc/about.rst Thu Jul 28 17:18:48 2011 +0100
@@ -0,0 +1,33 @@
+About CKAN
+===========
+
+Development
+-----------
+
+CKAN is an open source project and contributions are welcome!
+
+We discuss large changes and new features on the `ckan-discuss <http://lists.okfn.org/mailman/listinfo/ckan-discuss>`_ mailing list, and technical issues on the `ckan-dev <http://lists.okfn.org/mailman/listinfo/ckan-dev>`_ mailing list. Please join us there.
+
+You can find developer resources and links to our ticketing system on the `CKAN wiki <http://wiki.ckan.net/Main_Page>`_.
+
+Acknowledgements
+----------------
+
+Thanks to the following projects, without which CKAN would not have
+been possible:
+
+ * `Python <http://www.python.org>`_
+ * `Pylons <http://pylonshq.com/>`_
+ * CKAN logo: "angry hamster" by http://www.maedelmaedel.com/ and
+ http://www.villainous.biz/
+ * `FamFamFam silk icons <http://www.famfamfam.com/lab/icons/silk/>`_
+
+Copying and Licence
+-------------------
+
+This material is copyright (c) 2006-2011 Open Knowledge Foundation.
+
+It is open and licensed under the GNU Affero General Public License (AGPL) v3.0
+whose full text may be found at:
+
+<http://www.fsf.org/licensing/licenses/agpl-3.0.html>
\ No newline at end of file
--- a/doc/api.rst Thu Jul 28 16:45:02 2011 +0100
+++ b/doc/api.rst Thu Jul 28 17:18:48 2011 +0100
@@ -16,8 +16,11 @@
.. #|version_3_doc| replace:: :doc:`api/version3`
-|site| |api|
-======================
+.. index:: API
+
+===========================
+Reference: The |site| |api|
+===========================
.. toctree::
:hidden:
@@ -25,6 +28,8 @@
The |site| |api| presents the catalog of metadata behind |site|.
+.. note:: This section contains general information about the |site| API. If you are looking for API documentation, please see |version_2_doc|.
+
.. include:: api/overview.rst.inc
.. include:: api/versions.rst.inc
--- a/doc/api/clients.rst.inc Thu Jul 28 16:45:02 2011 +0100
+++ b/doc/api/clients.rst.inc Thu Jul 28 17:18:48 2011 +0100
@@ -3,5 +3,5 @@
There are also some code modules (Python, PHP, Drupal, Perl etc.) that provide
convenient wrappers around much of the CKAN API. For full details of these,
-please consult: http://trac.ckan.org/wiki/API
+please consult http://wiki.ckan.net/API
--- a/doc/api/purpose.rst.inc Thu Jul 28 16:45:02 2011 +0100
+++ b/doc/api/purpose.rst.inc Thu Jul 28 17:18:48 2011 +0100
@@ -1,2 +1,1 @@
-This document describes |version| of the |main_doc|.
-
+This document describes |version| of the |main_doc|.
\ No newline at end of file
--- a/doc/api/version1.rst Thu Jul 28 16:45:02 2011 +0100
+++ b/doc/api/version1.rst Thu Jul 28 17:18:48 2011 +0100
@@ -3,6 +3,7 @@
.. include:: title.rst.inc
.. include:: purpose.rst.inc
+.. include:: warning.rst.inc
.. include:: overview.rst.inc
.. include:: interfaces1.rst.inc
.. include:: location.rst.inc
--- a/doc/api/version2.rst Thu Jul 28 16:45:02 2011 +0100
+++ b/doc/api/version2.rst Thu Jul 28 17:18:48 2011 +0100
@@ -97,6 +97,7 @@
.. |site| replace:: CKAN
.. |api| replace:: API
.. |version| replace:: Version 2
+.. |warning| replace:: This is the latest version of the API.
.. |base_location| replace:: ``http://ckan.net/api/2``
.. |main_doc| replace:: :doc:`../api`
.. |usage| replace:: to view and change
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/doc/api/warning.rst.inc Thu Jul 28 17:18:48 2011 +0100
@@ -0,0 +1,1 @@
+.. warning:: This version is now deprecated in favour of :doc:`version2`.
\ No newline at end of file
--- a/doc/authentication.rst Thu Jul 28 16:45:02 2011 +0100
+++ /dev/null Thu Jan 01 00:00:00 1970 +0000
@@ -1,7 +0,0 @@
-CKAN Authentication and Identification
-======================================
-
-CKAN operates a delegated authentication model based on openid. Integration of
-authentication into CKAN is provided by repoze.who and the repoze.who.openid
-plugin.
-
--- a/doc/authorization.rst Thu Jul 28 16:45:02 2011 +0100
+++ b/doc/authorization.rst Thu Jul 28 17:18:48 2011 +0100
@@ -1,325 +1,206 @@
-=====================================
-CKAN Authorization and Access Control
-=====================================
-
-This document gives an overview of CKAN's authorization capabilities and model
-in relation to access control. The authentication/identification aspects of
-access control are dealt with separately in :doc:`authentication`.
-
-
-Overview
-========
+==========================
+Set and Manage Permissions
+==========================
CKAN implements a fine-grained role-based access control system.
- *In a nutshell*: For a particular **package** (or other protected object) a **user** can be assigned a **role** which specifies permitted **actions** (edit, delete, change permissions, etc.).
+This section describes:
-Protected Objects and Authorization
------------------------------------
+* :ref:`permissions-overview`. An overview of the concepts underlying CKAN authorization: objects, actions and roles.
+* :ref:`permissions-default`. The default permissions in CKAN.
+* :ref:`permissions-manage`. Managing and setting permissions.
+* :ref:`permissions-publisher-mode`. Suitable for systems where you want to limit write access to CKAN.
-There are variety of **protected objects** to which access can be controlled,
-for example the ``System``, ``Packages``, Package ``Groups`` and
-``Authorization Groups``. Access control is fine-grained in that it can be
-set for each individual package, group or authorization group instance.
+.. _permissions-overview:
-For each protected object there are a set of relevant **actions** such as
-create', 'admin', 'edit' etc. To facilitate mapping Users and Objects with Actions, Actions are aggregated into a set of **roles** (e.g. an 'editor' role would have 'edit' and 'read' action).
+Overview
+--------
-A special role is taken by the ``System`` object, which serves as an
-authorization object for any assignments which do not relate to a specific
-object. For example, the creation of a package cannot be linked to a
-specific package instance and is therefore a system operation.
+In a nutshell: for a particular **object** (e.g. a package) a CKAN **user** can be assigned a **role** (e.g. editor) which allows permitted **actions** (e.g. read, edit).
-To gain further flexibility, users can be assigned to **authorization
-groups**. Authz groups are both the object of authorization (i.e. one
-can have several roles with regards to an authz group) and the subject
-of authorization (i.e. they can be assigned roles on other objects which
-will apply to their members.
+In more detail, these concepts are as follows:
+
+* There are **objects** to which access can be controlled, such as packages and groups.
+* For each object there are a set of relevant **actions**, such as create and edit, which users can perform on the object.
+* To simplify mapping users to actions and objects, actions are aggregated into a set of **roles**. For example, an editor role would automatically have edit and read actions.
+* Finally, CKAN has registered **users**.
-The assignment of users and authorization groups to roles on a given
-protected object (such as a package) can be done by 'admins' via the
-'authorization' tab of the web interface (or by system admins via that
-interface or the system admin interface). There is also a command-line
-based authorization manager, detailed below.
+Objects
++++++++
-Command-line authorization management
--------------------------------------
+Permissions are controlled per object: access can be controlled for an individual
+package, group or authorization group instance. Current objects include
+**packages**, package **groups**, **authorization groups** and the **system**.
-Although the Admin Extension provides a Web interface for managing authorization, there is a set of more powerful :doc:`paster` commands for fine-grained control.
+* A package is the basic CKAN concept of metadata about a dataset.
+* A group of packages can be set up to specify which users have permission to add or remove packages from the group.
+* Users can be assigned to authorization groups, to increase flexibility. Instead of specifying the privileges of specific users on a package or group, you can also specify a set of users that share the same rights. To do that, an authorization group can be set up and users can be added to it. Authorization groups are both the object of authorization (i.e. one can have several roles with regards to an authorization group, such as being allowed to read or edit it) and the subject of authorization (i.e. they can be assigned roles on other objects which will apply to their members, such as the group having edit rights on a particular group).
+* Finally, the system object is special, serving as an object for assignments that do not relate to a specific object. For example, creating a package cannot be linked to a specific package instance, and is therefore a operation.
-The ``roles`` command will list and modify the assignment of actions to
-roles::
- $ paster --plugin=ckan roles -c my.ini list
- $ paster --plugin=ckan roles -c my.ini deny editor create-package
- $ paster --plugin=ckan roles -c my.ini allow editor create-package
+Actions
++++++++
-This would first list all role action assignments, then remove the
-'create-package' action from the 'editor' role and finally re-assign
-that role.
+**Actions** are defined in the Action enumeration in ``ckan/model/authz.py`` and currently include: **edit**, **change-state**, **read**, **purge**, **edit-permissions**, **create-package**, **create-group**, **create-authorization-group**, **read-site**, **read-user**, **create-user**.
-Similarly, the ``rights`` command will set the authorization roles of
-a specific user on a given object within the system::
+As noted above, some of these (e.g. **read**) have meaning for any type of object, while some (e.g. **create-package**) can not be associated with any particular object, and are therefore only associated with the system object.
- $ paster --plugin=ckan rights -c my.ini list
+The **read-site** action (associated with the system object) allows or denies access to pages not associated with specific objects. These currently include:
+
+ * Package search
+ * Group index
+ * Tags index
+ * Authorization Group index
+ * All requests to the API (on top of any other authorization requirements)
-Will list all assigned rights within the system. It is recommended to then
-grep over the resulting output to search for specific object roles.
+There are also some shortcuts that are provided directly by the authorization
+system (rather than being expressed as subject-object-role tuples):
-Rights assignment follows a similar pattern::
-
- $ paster --plugin=ckan rights -c my.ini make bar admin package:foo
-
-This would assign the user named ``bar`` the ``admin`` role on the package
-foo. Instead of user names and package names, a variety of different
-entities can be the subject or object of a role assignment. Some of those
-include authorization groups, package groups and the system as a whole
-(``system:```)::
-
- # make 'chef' a system-wide admin:
- $ paster --plugin=ckan rights -c my.ini make chef admin system
- # allow all members of authz group 'foo' to edit group 'bar'
- $ paster --plugin=ckan rights -c my.ini make agroup:foo edit \
- group:bar
-
-To revoke one of the roles assigned using ``make``, the ``remove`` command
-is available::
-
- $ paster --plugin=ckan rights -c my.ini remove bar admin package:foo
-
-For more help on either of these commands, also refer to the paster help.
+ * A user given the **admin** right for the **system** object is a 'sysadmin' and can perform any action on any object. (A shortcut for creating a sysadmin is by using the ``paster sysadmin`` command.)
+ * A user given the **admin** right for a particular object can perform any action on that object.
Roles
------
++++++
-Each role has a list of permitted *actions* appropriate for a protected object.
+Each **role** has a list of permitted actions appropriate for a protected object.
-Currently there are three basic roles (although you can add others if these defaults do not suit):
+Currently there are four basic roles:
* **reader**: can read the object
- * **anon_editor**: (anonymous i.e. not logged in) can edit and read the object
+ * **anon_editor**: anonymous users (i.e. not logged in) can edit and read the object
* **editor**: can edit, read and create new objects
* **admin**: admin can do anything including: edit, read, delete,
update-permissions (change authorizations for that object)
-When you install a new CKAN extension or upgrade your version of CKAN then new actions may be created, and permissions may given to these basic roles, according to the broad intention of the name of the roles.
+You can add other roles if these defaults are not sufficient for your system.
-It is suggested that if the broad idea of these basic roles and their actions are not suitable for your CKAN instance then you create new roles and assign them actions of your own choosing, rather than edit the roles. If the definition of the roles drift from their name then it can be confusing for admins and cause problems for CKAN upgrades and new extensions.
+.. warning:: If the broad idea of these basic roles and their actions is not suitable for your CKAN system, we suggest you create new roles, rather than edit the basic roles. If the definition of a role changes but its name does not, it is likely to confuse administrators and cause problems for CKAN upgrades and extensions.
-Actions
--------
+.. note:: When you install a new CKAN extension, or upgrade your version of CKAN, new actions may be created, and permissions given to these basic roles, in line with the broad intention of the roles.
-Actions are defined in the Action enumeration in ckan/model/authz.py and currently include: `edit`, `change-state`, `read`, `purge`, `edit-permissions`, `create-package`, `create-group`, `create-authorization-group`, `read-site`, `read-user`, `create-user`.
+Users
++++++
-Obviously, some of these (e.g. `read`) have meaning for any type of Domain Object, and some (e.g. `create-package`) can not be associated with any particular Domain Object, so the Context for Roles with these Actions is `system`.
+You can manage CKAN users via the command line with the ``paster user`` command - for more information, see :ref:`paster-user`.
-The `read-site` action (with System context) is designed to provide/deny access to pages not associated with Domain Objects. This currently includes:
-
- * Package Search
- * Group index
- * Tags index
- * Authorization Group index
- * all requests to the API (on top of any other authorization requirements)
+There are two special *pseudo-users* in CKAN, **visitor** and **logged-in**. These are used to refer to special sets of users, respectively those who are a) not logged-in ("visitor") and b) logged-in ("logged-in").
-There are also some shortcuts that are provided directly by the authorization
-system (rather than being expressed as subject-object-role tuples):
+The ``default_roles`` config option in the CKAN config file lets you set the default authorization roles (i.e. permissions) for these two types of users. For more information, see :doc:`configuration`.
- * A user given the admin right for the System object is a 'System Admin' and can do any action on any object. (A shortcut for creating a System Admin is by using the ``paster sysadmin`` command.)
- * A user given the admin right for a particular object can do any action to that object.
-Publisher mode setup
+.. _permissions-default:
+
+Default Permissions
+-------------------
+
+CKAN ships with the following default permissions:
+
+* When a new package is created, its creator automatically becomes **admin** for it. This user can then change permissions for other users.
+* By default, any other user (including both visitors and logged-ins) can read and write to this package.
+
+These defaults can be changed in the CKAN config - see ``default_roles`` in :doc:`configuration`.
+
+
+.. _permissions-manage:
+
+Managing Permissions
--------------------
-Although ckan.net is forging ahead with the Wikipedia model of allowing anyone to add and improve metadata, some CKAN instances prefer to operate in 'Publisher mode' which allows edits only from authorized users.
+The assignment of users and authorization groups to roles on a given
+protected object (such as a package) can be done by 'admins' via the
+'authorization' tab of the web interface (or by sysadmins via that
+interface or the system admin interface).
+
+There is also a command-line authorization manager, detailed below.
+
+Command-line authorization management
++++++++++++++++++++++++++++++++++++++
+
+Although the admin extension provides a Web interface for managing authorization,
+there is a set of more powerful ``paster`` commands for fine-grained control
+(see :doc:`paster`).
+
+The ``rights`` command is used to configure the authorization roles of
+a specific user on a given object within the system.
+
+For example, to list all assigned rights in the system (which you can then grep if needed)::
+
+ paster --plugin=ckan rights -c my.ini list
+
+The ``rights make`` command lets you assign specific permissions. For example, to give the user named **bar** the **admin** role on the package foo::
+
+ paster --plugin=ckan rights -c my.ini make bar admin package:foo
+
+As well as users and packages, you can assign rights to other objects. These
+include authorization groups, package groups and the system as a whole.
+
+For example, to make the user 'chef' a system-wide admin::
+
+ paster --plugin=ckan rights -c my.ini make chef admin system
+
+Or to allow all members of authorization group 'foo' to edit group 'bar'::
+
+ paster --plugin=ckan rights -c my.ini make agroup:foo edit \
+ group:bar
+
+To revoke one of the roles assigned using ``rights make``, the ``rights remove`` command
+is available. For example, to remove **bar**'s **admin** role on the foo package::
+
+ paster --plugin=ckan rights -c my.ini remove bar admin package:foo
+
+The ``roles`` command lists and modifies the assignment of actions to
+roles.
+
+To list all role assignments::
+
+ paster --plugin=ckan roles -c my.ini list
+
+To remove the 'create-package' action from the 'editor' role::
+
+ paster --plugin=ckan roles -c my.ini deny editor create-package
+
+And to re-assign 'create-package' to the 'editor' role::
+
+ paster --plugin=ckan roles -c my.ini allow editor create-package
+
+For more help on either of these commands, you can use ``--help`` (as described in :ref:`paster-help`)::
+
+ paster --plugin=ckan roles --help
+ paster --plugin=ckan rights --help
+
+
+.. _permissions-publisher-mode:
+
+Publisher Mode
+--------------
+
+Although the `Data Hub <http://datahub.org>`_ prefers the Wikipedia-style model of allowing anyone to add and improve metadata, some CKAN instances prefer to operate in 'publisher mode'.
+
+This allows edits only from authorized users. It is designed for installations where you wish to limit write access to CKAN and orient the system around specific publishing groups (e.g. government departments or specific institutions).
+
+The key features are:
+
+* Packages are assigned to a specific publishing group.
+* Only users associated to that group are able to create or update packages associated to that group.
To operate in this mode:
- 1. Remove the rights for general public to edit existing packages and create new ones.::
+1. First, remove the general public's rights to create and edit packages::
- paster rights remove visitor anon_editor package:all
- paster rights remove logged_in editor package:all
- paster rights remove visitor anon_editor system
- paster rights remove logged_in editor system
+ paster rights remove visitor anon_editor package:all
+ paster rights remove logged_in editor package:all
+ paster rights remove visitor anon_editor system
+ paster rights remove logged_in editor system
- 2. If logged-in users have already created packages in your system then you may also wish to remove admin rights. e.g.::
+2. If logged-in users have already created packages in your system, you may also wish to remove their admin rights. For example::
- paster rights remove bob admin package:all
+ paster rights remove bob admin package:all
- 3. Change the default rights for newly created packages. Do this by using these values in your config (.ini file)::
+3. Change the default rights for newly created packages. Do this by using these values in your config file (see :doc:`configuration`)::
- ckan.default_roles.Package = {"visitor": ["reader"], "logged_in": ["reader"]}
- ckan.default_roles.Group = {"visitor": ["reader"], "logged_in": ["reader"]}
- ckan.default_roles.System = {"visitor": ["reader"], "logged_in": ["reader"]}
- ckan.default_roles.AuthorizationGroup = {"visitor": ["reader"], "logged_in": ["reader"]}
+ ckan.default_roles.Package = {"visitor": ["reader"], "logged_in": ["reader"]}
+ ckan.default_roles.Group = {"visitor": ["reader"], "logged_in": ["reader"]}
+ ckan.default_roles.System = {"visitor": ["reader"], "logged_in": ["reader"]}
+ ckan.default_roles.AuthorizationGroup = {"visitor": ["reader"], "logged_in": ["reader"]}
- Note there is also the possibility to restrict package edits by a user's authorization group. See http://wiki.ckan.net/Publisher_Mode
-
-
-Examples
---------
-
-Example 1: Package 'paper-industry-stats':
-
- * David Brent is an 'admin'
- * Gareth Keenan is an 'editor'
- * Logged-in is a 'reader' (This is a special user, meaning 'anyone who is
- logged in')
- * Visitor is a 'reader' (Another special user, meaning 'anyone')
-
-That is, Gareth and David can edit this package, but only Gareth can assign
-roles (privileges) to new team members. Anyone can see (read) the package.
-
-
-Example 2: The current default for new packages is:
-
- * the user who creates it is an 'admin'
- * Visitor and Logged-in are both an 'editor' and 'reader'
-
-NB: "Visitor" and "Logged-in" are special "pseudo-users" used as a way of
-concretely referring to the special sets of users, namely those that are a) not
-logged-in ("visitor") and b) logged-in ("Logged-in")
-
-User Notes
-----------
-
-When a new package is created, its creator automatically become admin for
-it. This user can then change permissions for other users.
-
-NB: by default any user (including someone who is not logged-in) will be able
-to read and write. This default can be changed in the CKAN configuration - see ``default_roles`` in :doc:`configuration`.
-
-
-Developer Notes
-===============
-
-We record tuples of the form:
-
-======== ================= ======= ====================
-user authorized_group role object
-======== ================= ======= ====================
-levin editor package::warandpeace
-======== ================= ======= ====================
-
-
-
-
-Requirements and Use Cases
---------------------------
-
- * A user means someone who is logged in.
- * A visitor means someone who is not logged in.
- * An protected object is the subject of a permission (either a user or a
- pseudo-user)
- * There are roles named: Admin, Reader, Writer
-
- 1. A visitor visits a package page and reads the content
- 2. A visitor visits a package page and edits the package
- 3. Ditto 1 for a user
- 4. Ditto 2 for a user
- 5. On package creation if done by a user and not a visitor then user is made
- the 'admin'
- 6. An admin of a package adds a user as an admin
- 7. An admin of a package removes a user as an admin
- 8. Ditto for admin re. editor
- 9. Ditto for admin re. reader
- 10. We wish to be able assign roles to 2 specific entire groups in addition
- to specific users: 'visitor', 'users'. These will be termed pseudo-users
- as we do not have AC 'groups' as such.
- 11. The sysadmin alters the assignment of entities to roles for any package
- 12. A visitor goes to a package where the editor role does not include
- 'visitor' pseudo-user. They are unable to edit the package.
- 13. Ditto for user where users pseudo-user does not have editor role and user
- is not an editor for the package
- 14. Ditto 12 re reader role.
- 15. Ditto 13 re reader role.
- 16. Try to edit over REST interface a package for which 'visitor' has Editor
- role, but no API is supplied. Not allowed.
-
-
-Not Yet Implemented
-+++++++++++++++++++
-
- * Support for access-related groups
- * Support for blacklisting
-
-
-Conceptual Overview
--------------------
-
-**Warning: not all of what is described in this conceptual overview is yet
-fully implemented.**
-
- * There are Users and (User) Authorization Groups
- * There are actions which may be performed on "protected objects" such as
- Package, Group, System
- * Roles aggregate actions
- * UserObjectRole which assign users (or Authorization groups) a role on an
- object (user, role, object). We will often refer to these informally as
- "permissions".
-
-NB: there is no object explicitly named "Permission". This is to avoid
-confusion: a 'normal' "Permission" (as in e.g. repoze.what) would correspond to
-an action-object tuple. This works for the case where protected objects are
-limited e.g. a few core subsystems like email, admin panel etc). However, we
-have many protected objects (e.g. one for each package) and we use roles so
-this 'normal' model does not work well.
-
-Question: do we require for *both* Users and UserAuthorizationGroup to be
-subject of Role or not?
-
-Ans: Yes. Why? Consider, situation where I just want to give an individual user
-permission on a given object (e.g. assigning authz permission for a package)?
-If I just have UserAuthorizationGroups one would need to create a group just
-for that individual. This isn't impossible but consider next how to assign
-permissions to edit the Authorization Groups? One would need create another
-group for this but then we have recursion ad infinitum (unless this were
-especially encompassed in some system level permission or one has some group
-which is uneditable ...)
-
-Thus, one requires both Users and UserAuthorizationGroups to be subject of
-"permissions". To summarize the approximate structure we have is::
-
- class SubjectOfAuthorization
- class User
- class UserAuthorizationGroup
-
- class ObjectOfAuthorization
- class Package
- class Group
- class UserAuthorizationGroup
- ...
-
- class SubjectRoleObject
- subject_of_authorization
- object_of_authorization
- role
-
-
-Determining permissions
------------------------
-
-See ckan.authz.Authorizer.is_authorized
-
-.. automethod:: ckan.authz.Authorizer.is_authorized
-
-
-Comparison with other frameworks and approaches
-===============================================
-
-repoze.what
------------
-
-Demo example model::
-
- User
- Group
- Permission
-
- * Users are assigned to groups
- * Groups are assigned permissions
-
-Capabilities
-------------
-
-Each possible action-object tuple receive an identifier which we term the
-"capability". We would then list tuples (capability_subject, capability).
+Note you can also restrict package edits by a user's authorization group.
--- a/doc/buildbot.rst Thu Jul 28 16:45:02 2011 +0100
+++ b/doc/buildbot.rst Thu Jul 28 17:18:48 2011 +0100
@@ -1,13 +1,15 @@
-Setting up buildbot
-+++++++++++++++++++
+================
+Install Buildbot
+================
-These directions provide some rough information for setting up a build bot on a Lucid machine.
+This section provides information for CKAN core developers setting up buildbot on an Ubuntu Lucid machine.
+If you simply want to check the status of the latest CKAN builds, visit http://buildbot.okfn.org/.
Apt Installs
============
-Install CKAN core dependencies from Lucid distribution (as per :doc:`README`)::
+Install CKAN core dependencies from Lucid distribution::
sudo apt-get install build-essential libxml2-dev libxslt-dev
sudo apt-get install wget mercurial postgresql libpq-dev git-core
@@ -22,7 +24,7 @@
sudo apt-get install buildbot
-Deb building software (as per :doc:`deb`)::
+Deb building software::
sudo apt-get install -y dh-make devscripts fakeroot cdbs
@@ -36,7 +38,7 @@
sudo dpkg-reconfigure locales
-Postgres setup
+Postgres Setup
==============
If installation before failed to create a cluster, do this after fixing errors::
@@ -51,7 +53,7 @@
sudo -u postgres createdb -O buildslave ckanext
-Buildslave setup
+Buildslave Setup
================
Rough commands::
@@ -74,7 +76,7 @@
pip -E pyenv-tools install buildkit
-Buildmaster setup
+Buildmaster Setup
=================
Rough commands::
@@ -113,23 +115,23 @@
sudo /etc/init.d/buildbot start
-Now check you can view buildbot at: http://localhost:8010/
+Now check you can view buildbot at http://localhost:8010/
-Connect ports
+Connect Ports
=============
It's preferable to view the buildbot site at port 80 rather than 8010.
-If there is no other web service on this machine, you might connect up the addresses using iptables::
+If there is no other web service on this machine, you might connect up the addresses using ``iptables``::
sudo iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 8010
-Otherwise it is best to do a reverse proxy. Using apache, edit this file::
+Otherwise it is best to set up a reverse proxy. Using Apache, edit this file::
sudo vim /etc/apache2/sites-available/buildbot.okfn.org
-to be like this::
+to look like this::
<VirtualHost *:80>
ServerName buildbot.okfn.org
--- a/doc/ckan-features.svg Thu Jul 28 16:45:02 2011 +0100
+++ /dev/null Thu Jan 01 00:00:00 1970 +0000
@@ -1,323 +0,0 @@
-<?xml version="1.0" encoding="UTF-8" standalone="no"?>
-<!-- Created with Inkscape (http://www.inkscape.org/) -->
-<svg
- xmlns:dc="http://purl.org/dc/elements/1.1/"
- xmlns:cc="http://creativecommons.org/ns#"
- xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
- xmlns:svg="http://www.w3.org/2000/svg"
- xmlns="http://www.w3.org/2000/svg"
- xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
- xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
- width="744.09448819"
- height="1052.3622047"
- id="svg2"
- sodipodi:version="0.32"
- inkscape:version="0.46"
- sodipodi:docname="ckan-features.svg"
- inkscape:output_extension="org.inkscape.output.svg.inkscape">
- <defs
- id="defs4">
- <linearGradient
- id="linearGradient3557">
- <stop
- style="stop-color:#f52971;stop-opacity:0.71794873;"
- offset="1"
- id="stop3559" />
- </linearGradient>
- <linearGradient
- id="linearGradient3497">
- <stop
- style="stop-color:#f50b00;stop-opacity:0.54700857;"
- offset="1"
- id="stop3499" />
- </linearGradient>
- <linearGradient
- id="linearGradient3435">
- <stop
- style="stop-color:#f56c00;stop-opacity:0.64957267;"
- offset="0"
- id="stop3543" />
- <stop
- id="stop3549"
- offset="1"
- style="stop-color:#f56c00;stop-opacity:0.58039216;" />
- </linearGradient>
- <linearGradient
- id="linearGradient3155">
- <stop
- id="stop3457"
- offset="0"
- style="stop-color:#eff529;stop-opacity:0.67521369;" />
- <stop
- style="stop-color:#eff529;stop-opacity:0.65811968;"
- offset="1"
- id="stop3515" />
- <stop
- id="stop3455"
- offset="1"
- style="stop-color:#eff529;stop-opacity:1;" />
- </linearGradient>
- <inkscape:perspective
- sodipodi:type="inkscape:persp3d"
- inkscape:vp_x="0 : 526.18109 : 1"
- inkscape:vp_y="0 : 1000 : 0"
- inkscape:vp_z="744.09448 : 526.18109 : 1"
- inkscape:persp3d-origin="372.04724 : 350.78739 : 1"
- id="perspective10" />
- </defs>
- <sodipodi:namedview
- id="base"
- pagecolor="#ffffff"
- bordercolor="#666666"
- borderopacity="1.0"
- gridtolerance="10000"
- guidetolerance="10"
- objecttolerance="10"
- inkscape:pageopacity="0.0"
- inkscape:pageshadow="2"
- inkscape:zoom="1"
- inkscape:cx="380.55961"
- inkscape:cy="767.44485"
- inkscape:document-units="px"
- inkscape:current-layer="layer1"
- showgrid="true"
- inkscape:snap-global="false"
- inkscape:window-width="1014"
- inkscape:window-height="726"
- inkscape:window-x="-5"
- inkscape:window-y="25">
- <inkscape:grid
- type="xygrid"
- id="grid3377" />
- </sodipodi:namedview>
- <metadata
- id="metadata7">
- <rdf:RDF>
- <cc:Work
- rdf:about="">
- <dc:format>image/svg+xml</dc:format>
- <dc:type
- rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
- </cc:Work>
- </rdf:RDF>
- </metadata>
- <g
- inkscape:label="Layer 1"
- inkscape:groupmode="layer"
- id="layer1">
- <path
- sodipodi:type="arc"
- style="opacity:0.6;fill:#a3bcce;fill-opacity:1;stroke:none;stroke-width:4.65620232;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
- id="path2383"
- sodipodi:cx="252.85715"
- sodipodi:cy="389.50504"
- sodipodi:rx="144.28572"
- sodipodi:ry="137.14285"
- d="M 397.14287,389.50504 A 144.28572,137.14285 0 1 1 108.57143,389.50504 A 144.28572,137.14285 0 1 1 397.14287,389.50504 z"
- transform="matrix(0.5964695,0,0,0.5964695,105.76357,68.787717)"
- inkscape:export-filename="/home/rgrp/hgroot/ckan/doc/ckan-features.png"
- inkscape:export-xdpi="90"
- inkscape:export-ydpi="90" />
- <text
- xml:space="preserve"
- style="font-size:14.81217575px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:center;line-height:125%;writing-mode:lr-tb;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;stroke-width:3;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;font-family:Sans;-inkscape-font-specification:Sans"
- x="234.04362"
- y="312.77817"
- id="text3367"
- sodipodi:linespacing="125%"
- inkscape:export-filename="/home/rgrp/hgroot/ckan/doc/ckan-features.png"
- inkscape:export-xdpi="90"
- inkscape:export-ydpi="90"><tspan
- sodipodi:role="line"
- x="236.40141"
- y="312.77817"
- id="tspan3371">Wiki / </tspan><tspan
- sodipodi:role="line"
- x="234.04362"
- y="331.29337"
- id="tspan3375">Versioned DB</tspan></text>
- <path
- sodipodi:type="arc"
- style="opacity:0.6;fill:#a0cfa1;fill-opacity:1;stroke:none;stroke-width:4.65620232;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
- id="path3405"
- sodipodi:cx="252.85715"
- sodipodi:cy="389.50504"
- sodipodi:rx="144.28572"
- sodipodi:ry="137.14285"
- d="M 397.14287,389.50504 A 144.28572,137.14285 0 1 1 108.57143,389.50504 A 144.28572,137.14285 0 1 1 397.14287,389.50504 z"
- transform="matrix(0.5964695,0,0,0.5964695,170.8961,-0.1971008)"
- inkscape:export-filename="/home/rgrp/hgroot/ckan/doc/ckan-features.png"
- inkscape:export-xdpi="90"
- inkscape:export-ydpi="90" />
- <text
- xml:space="preserve"
- style="font-size:14.81217575px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:center;line-height:125%;writing-mode:lr-tb;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;font-family:Sans;-inkscape-font-specification:Sans"
- x="323.72488"
- y="187.99576"
- id="text3407"
- sodipodi:linespacing="125%"
- inkscape:export-filename="/home/rgrp/hgroot/ckan/doc/ckan-features.png"
- inkscape:export-xdpi="90"
- inkscape:export-ydpi="90"><tspan
- sodipodi:role="line"
- x="323.72488"
- y="187.99576"
- id="tspan3461">Package Index</tspan><tspan
- sodipodi:role="line"
- x="323.72488"
- y="206.51097"
- id="tspan3511">(APT/PyPI/CPAN)</tspan></text>
- <path
- transform="matrix(0.5964695,0,0,0.5964695,240.3075,69.096304)"
- d="M 397.14287,389.50504 A 144.28572,137.14285 0 1 1 108.57143,389.50504 A 144.28572,137.14285 0 1 1 397.14287,389.50504 z"
- sodipodi:ry="137.14285"
- sodipodi:rx="144.28572"
- sodipodi:cy="389.50504"
- sodipodi:cx="252.85715"
- id="path3415"
- style="opacity:0.6;fill:#c7d9a0;fill-opacity:1;stroke:none;stroke-width:4.65620232;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
- sodipodi:type="arc"
- inkscape:export-filename="/home/rgrp/hgroot/ckan/doc/ckan-features.png"
- inkscape:export-xdpi="90"
- inkscape:export-ydpi="90" />
- <text
- sodipodi:linespacing="125%"
- id="text3417"
- y="311.95526"
- x="423.10461"
- style="font-size:14.81217575px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:center;line-height:125%;writing-mode:lr-tb;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;stroke-width:3;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;font-family:Sans;-inkscape-font-specification:Sans"
- xml:space="preserve"
- inkscape:export-filename="/home/rgrp/hgroot/ckan/doc/ckan-features.png"
- inkscape:export-xdpi="90"
- inkscape:export-ydpi="90"><tspan
- y="311.95526"
- x="423.10461"
- sodipodi:role="line"
- id="tspan3447">Listing</tspan><tspan
- y="330.47049"
- x="423.10461"
- sodipodi:role="line"
- id="tspan3179">(Freshmeat)</tspan></text>
- <text
- xml:space="preserve"
- style="font-size:18.10377121px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:center;line-height:125%;writing-mode:lr-tb;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;stroke-width:3;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;font-family:Sans;-inkscape-font-specification:Sans"
- x="323.94531"
- y="289.737"
- id="text3183"
- sodipodi:linespacing="125%"
- inkscape:export-filename="/home/rgrp/hgroot/ckan/doc/ckan-features.png"
- inkscape:export-xdpi="90"
- inkscape:export-ydpi="90"><tspan
- sodipodi:role="line"
- x="323.94531"
- y="289.737"
- id="tspan3185">CKAN</tspan></text>
- <path
- transform="matrix(0.7248396,0,0,0.7248396,91.826254,471.23864)"
- d="M 397.14287,389.50504 A 144.28572,137.14285 0 1 1 108.57143,389.50504 A 144.28572,137.14285 0 1 1 397.14287,389.50504 z"
- sodipodi:ry="137.14285"
- sodipodi:rx="144.28572"
- sodipodi:cy="389.50504"
- sodipodi:cx="252.85715"
- id="path3178"
- style="opacity:0.6;fill:#a3bcce;fill-opacity:1;stroke:none;stroke-width:4.65620232;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
- sodipodi:type="arc" />
- <text
- sodipodi:linespacing="125%"
- id="text3180"
- y="732.73987"
- x="224.71428"
- style="font-size:15px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:center;line-height:125%;writing-mode:lr-tb;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;stroke-width:3;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;font-family:Sans;-inkscape-font-specification:Sans"
- xml:space="preserve"><tspan
- id="tspan3182"
- y="732.73987"
- x="227.10197"
- sodipodi:role="line">Wiki / </tspan><tspan
- id="tspan3184"
- y="751.48987"
- x="224.71428"
- sodipodi:role="line">Versioned DB</tspan></text>
- <path
- transform="matrix(0.7248396,0,0,0.7248396,170.97637,387.40715)"
- d="M 397.14287,389.50504 A 144.28572,137.14285 0 1 1 108.57143,389.50504 A 144.28572,137.14285 0 1 1 397.14287,389.50504 z"
- sodipodi:ry="137.14285"
- sodipodi:rx="144.28572"
- sodipodi:cy="389.50504"
- sodipodi:cx="252.85715"
- id="path3186"
- style="opacity:0.6;fill:#a0cfa1;fill-opacity:1;stroke:none;stroke-width:4.65620232;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
- sodipodi:type="arc" />
- <text
- sodipodi:linespacing="125%"
- id="text3188"
- y="608.10217"
- x="356.69644"
- style="font-size:15px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:center;line-height:125%;writing-mode:lr-tb;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;font-family:Sans;-inkscape-font-specification:Sans"
- xml:space="preserve"><tspan
- id="tspan3190"
- y="608.10217"
- x="356.69644"
- sodipodi:role="line">Package Index</tspan><tspan
- id="tspan3192"
- y="626.85217"
- x="356.69644"
- sodipodi:role="line">(APT/PyPI/CPAN)</tspan></text>
- <path
- sodipodi:type="arc"
- style="opacity:0.6;fill:#c7d9a0;fill-opacity:1;stroke:none;stroke-width:4.65620232;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
- id="path3194"
- sodipodi:cx="252.85715"
- sodipodi:cy="389.50504"
- sodipodi:rx="144.28572"
- sodipodi:ry="137.14285"
- d="M 397.14287,389.50504 A 144.28572,137.14285 0 1 1 108.57143,389.50504 A 144.28572,137.14285 0 1 1 397.14287,389.50504 z"
- transform="matrix(0.7248396,0,0,0.7248396,255.32625,471.61364)" />
- <text
- xml:space="preserve"
- style="font-size:15px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:center;line-height:125%;writing-mode:lr-tb;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;stroke-width:3;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;font-family:Sans;-inkscape-font-specification:Sans"
- x="494.46429"
- y="732.73987"
- id="text3196"
- sodipodi:linespacing="125%"><tspan
- id="tspan3198"
- sodipodi:role="line"
- x="494.46429"
- y="732.73987">Listing</tspan><tspan
- id="tspan3200"
- sodipodi:role="line"
- x="494.46429"
- y="751.48987">(Freshmeat)</tspan></text>
- <path
- transform="matrix(0.7248396,0,0,0.7248396,170.97637,545.68123)"
- d="M 397.14287,389.50504 A 144.28572,137.14285 0 1 1 108.57143,389.50504 A 144.28572,137.14285 0 1 1 397.14287,389.50504 z"
- sodipodi:ry="137.14285"
- sodipodi:rx="144.28572"
- sodipodi:cy="389.50504"
- sodipodi:cx="252.85715"
- id="path3202"
- style="opacity:0.6;fill:#b2a3c4;fill-opacity:1;stroke:none;stroke-width:4.65620232;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
- sodipodi:type="arc" />
- <text
- xml:space="preserve"
- style="font-size:15px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:center;line-height:125%;writing-mode:lr-tb;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;stroke-width:3;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;font-family:Sans;-inkscape-font-specification:Sans"
- x="354.71429"
- y="886.48987"
- id="text3204"
- sodipodi:linespacing="125%"><tspan
- sodipodi:role="line"
- x="354.71429"
- y="886.48987"
- id="tspan3206">Forum</tspan></text>
- <text
- sodipodi:linespacing="125%"
- id="text3208"
- y="753.73987"
- x="356.96429"
- style="font-size:18px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:center;line-height:125%;writing-mode:lr-tb;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;stroke-width:3;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;font-family:Sans;-inkscape-font-specification:Sans"
- xml:space="preserve"><tspan
- id="tspan3210"
- y="753.73987"
- x="356.96429"
- sodipodi:role="line">CKAN</tspan></text>
- </g>
-</svg>
--- a/doc/ckan-vision.dia Thu Jul 28 16:45:02 2011 +0100
+++ /dev/null Thu Jan 01 00:00:00 1970 +0000
@@ -1,825 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<dia:diagram xmlns:dia="http://www.lysator.liu.se/~alla/dia/">
- <dia:diagramdata>
- <dia:attribute name="background">
- <dia:color val="#ffffff"/>
- </dia:attribute>
- <dia:attribute name="pagebreak">
- <dia:color val="#000099"/>
- </dia:attribute>
- <dia:attribute name="paper">
- <dia:composite type="paper">
- <dia:attribute name="name">
- <dia:string>#A4#</dia:string>
- </dia:attribute>
- <dia:attribute name="tmargin">
- <dia:real val="2.8222000598907471"/>
- </dia:attribute>
- <dia:attribute name="bmargin">
- <dia:real val="2.8222000598907471"/>
- </dia:attribute>
- <dia:attribute name="lmargin">
- <dia:real val="2.8222000598907471"/>
- </dia:attribute>
- <dia:attribute name="rmargin">
- <dia:real val="2.8222000598907471"/>
- </dia:attribute>
- <dia:attribute name="is_portrait">
- <dia:boolean val="true"/>
- </dia:attribute>
- <dia:attribute name="scaling">
- <dia:real val="1"/>
- </dia:attribute>
- <dia:attribute name="fitto">
- <dia:boolean val="false"/>
- </dia:attribute>
- </dia:composite>
- </dia:attribute>
- <dia:attribute name="grid">
- <dia:composite type="grid">
- <dia:attribute name="width_x">
- <dia:real val="1"/>
- </dia:attribute>
- <dia:attribute name="width_y">
- <dia:real val="1"/>
- </dia:attribute>
- <dia:attribute name="visible_x">
- <dia:int val="1"/>
- </dia:attribute>
- <dia:attribute name="visible_y">
- <dia:int val="1"/>
- </dia:attribute>
- <dia:composite type="color"/>
- </dia:composite>
- </dia:attribute>
- <dia:attribute name="color">
- <dia:color val="#d8e5e5"/>
- </dia:attribute>
- <dia:attribute name="guides">
- <dia:composite type="guides">
- <dia:attribute name="hguides"/>
- <dia:attribute name="vguides"/>
- </dia:composite>
- </dia:attribute>
- </dia:diagramdata>
- <dia:layer name="Background" visible="true">
- <dia:object type="Standard - Box" version="0" id="O0">
- <dia:attribute name="obj_pos">
- <dia:point val="7.5,18.5"/>
- </dia:attribute>
- <dia:attribute name="obj_bb">
- <dia:rectangle val="7.45,18.45;14.55,25.55"/>
- </dia:attribute>
- <dia:attribute name="elem_corner">
- <dia:point val="7.5,18.5"/>
- </dia:attribute>
- <dia:attribute name="elem_width">
- <dia:real val="7"/>
- </dia:attribute>
- <dia:attribute name="elem_height">
- <dia:real val="7"/>
- </dia:attribute>
- <dia:attribute name="show_background">
- <dia:boolean val="false"/>
- </dia:attribute>
- <dia:attribute name="line_style">
- <dia:enum val="1"/>
- </dia:attribute>
- <dia:attribute name="corner_radius">
- <dia:real val="1.1754943508222875e-38"/>
- </dia:attribute>
- </dia:object>
- <dia:object type="Standard - Box" version="0" id="O1">
- <dia:attribute name="obj_pos">
- <dia:point val="17.5,13"/>
- </dia:attribute>
- <dia:attribute name="obj_bb">
- <dia:rectangle val="17.45,12.95;30.55,32.05"/>
- </dia:attribute>
- <dia:attribute name="elem_corner">
- <dia:point val="17.5,13"/>
- </dia:attribute>
- <dia:attribute name="elem_width">
- <dia:real val="13"/>
- </dia:attribute>
- <dia:attribute name="elem_height">
- <dia:real val="19"/>
- </dia:attribute>
- <dia:attribute name="show_background">
- <dia:boolean val="false"/>
- </dia:attribute>
- <dia:attribute name="line_style">
- <dia:enum val="1"/>
- </dia:attribute>
- <dia:attribute name="corner_radius">
- <dia:real val="1.1754943508222875e-38"/>
- </dia:attribute>
- </dia:object>
- <dia:object type="Flowchart - Box" version="0" id="O2">
- <dia:attribute name="obj_pos">
- <dia:point val="19,16"/>
- </dia:attribute>
- <dia:attribute name="obj_bb">
- <dia:rectangle val="18.95,15.95;29.045,21.05"/>
- </dia:attribute>
- <dia:attribute name="elem_corner">
- <dia:point val="19,16"/>
- </dia:attribute>
- <dia:attribute name="elem_width">
- <dia:real val="9.9949999999999992"/>
- </dia:attribute>
- <dia:attribute name="elem_height">
- <dia:real val="5"/>
- </dia:attribute>
- <dia:attribute name="show_background">
- <dia:boolean val="true"/>
- </dia:attribute>
- <dia:attribute name="padding">
- <dia:real val="0.5"/>
- </dia:attribute>
- <dia:attribute name="text">
- <dia:composite type="text">
- <dia:attribute name="string">
- <dia:string>#CKAN
-(The Registry)
-+Web User Interface (WUI)
-+Machine Web API (WAPI)#</dia:string>
- </dia:attribute>
- <dia:attribute name="font">
- <dia:font family="sans" style="0" name="Helvetica"/>
- </dia:attribute>
- <dia:attribute name="height">
- <dia:real val="0.80000000000000004"/>
- </dia:attribute>
- <dia:attribute name="pos">
- <dia:point val="23.9975,17.4425"/>
- </dia:attribute>
- <dia:attribute name="color">
- <dia:color val="#000000"/>
- </dia:attribute>
- <dia:attribute name="alignment">
- <dia:enum val="1"/>
- </dia:attribute>
- </dia:composite>
- </dia:attribute>
- </dia:object>
- <dia:object type="Flowchart - Box" version="0" id="O3">
- <dia:attribute name="obj_pos">
- <dia:point val="8.065,21"/>
- </dia:attribute>
- <dia:attribute name="obj_bb">
- <dia:rectangle val="8.015,20.95;13.985,24.05"/>
- </dia:attribute>
- <dia:attribute name="elem_corner">
- <dia:point val="8.065,21"/>
- </dia:attribute>
- <dia:attribute name="elem_width">
- <dia:real val="5.8699999999999992"/>
- </dia:attribute>
- <dia:attribute name="elem_height">
- <dia:real val="3"/>
- </dia:attribute>
- <dia:attribute name="show_background">
- <dia:boolean val="true"/>
- </dia:attribute>
- <dia:attribute name="padding">
- <dia:real val="0.5"/>
- </dia:attribute>
- <dia:attribute name="text">
- <dia:composite type="text">
- <dia:attribute name="string">
- <dia:string>#Is It Open
-isitopendata.org#</dia:string>
- </dia:attribute>
- <dia:attribute name="font">
- <dia:font family="sans" style="0" name="Helvetica"/>
- </dia:attribute>
- <dia:attribute name="height">
- <dia:real val="0.80000000000000004"/>
- </dia:attribute>
- <dia:attribute name="pos">
- <dia:point val="11,22.2425"/>
- </dia:attribute>
- <dia:attribute name="color">
- <dia:color val="#000000"/>
- </dia:attribute>
- <dia:attribute name="alignment">
- <dia:enum val="1"/>
- </dia:attribute>
- </dia:composite>
- </dia:attribute>
- </dia:object>
- <dia:object type="Flowchart - Box" version="0" id="O4">
- <dia:attribute name="obj_pos">
- <dia:point val="21,28"/>
- </dia:attribute>
- <dia:attribute name="obj_bb">
- <dia:rectangle val="20.95,27.95;27.05,31.05"/>
- </dia:attribute>
- <dia:attribute name="elem_corner">
- <dia:point val="21,28"/>
- </dia:attribute>
- <dia:attribute name="elem_width">
- <dia:real val="6"/>
- </dia:attribute>
- <dia:attribute name="elem_height">
- <dia:real val="3"/>
- </dia:attribute>
- <dia:attribute name="show_background">
- <dia:boolean val="true"/>
- </dia:attribute>
- <dia:attribute name="padding">
- <dia:real val="0.5"/>
- </dia:attribute>
- <dia:attribute name="text">
- <dia:composite type="text">
- <dia:attribute name="string">
- <dia:string>#CKAN CLIENT#</dia:string>
- </dia:attribute>
- <dia:attribute name="font">
- <dia:font family="sans" style="0" name="Helvetica"/>
- </dia:attribute>
- <dia:attribute name="height">
- <dia:real val="0.80000000000000004"/>
- </dia:attribute>
- <dia:attribute name="pos">
- <dia:point val="24,29.6425"/>
- </dia:attribute>
- <dia:attribute name="color">
- <dia:color val="#000000"/>
- </dia:attribute>
- <dia:attribute name="alignment">
- <dia:enum val="1"/>
- </dia:attribute>
- </dia:composite>
- </dia:attribute>
- </dia:object>
- <dia:object type="Flowchart - Box" version="0" id="O5">
- <dia:attribute name="obj_pos">
- <dia:point val="33,28"/>
- </dia:attribute>
- <dia:attribute name="obj_bb">
- <dia:rectangle val="32.95,27.95;42.05,31.05"/>
- </dia:attribute>
- <dia:attribute name="elem_corner">
- <dia:point val="33,28"/>
- </dia:attribute>
- <dia:attribute name="elem_width">
- <dia:real val="9"/>
- </dia:attribute>
- <dia:attribute name="elem_height">
- <dia:real val="3"/>
- </dia:attribute>
- <dia:attribute name="show_background">
- <dia:boolean val="true"/>
- </dia:attribute>
- <dia:attribute name="padding">
- <dia:real val="0.5"/>
- </dia:attribute>
- <dia:attribute name="text">
- <dia:composite type="text">
- <dia:attribute name="string">
- <dia:string>#DATAPKG
-(like apt-get,gem,pip)#</dia:string>
- </dia:attribute>
- <dia:attribute name="font">
- <dia:font family="sans" style="0" name="Helvetica"/>
- </dia:attribute>
- <dia:attribute name="height">
- <dia:real val="0.80000000000000004"/>
- </dia:attribute>
- <dia:attribute name="pos">
- <dia:point val="37.5,29.2425"/>
- </dia:attribute>
- <dia:attribute name="color">
- <dia:color val="#000000"/>
- </dia:attribute>
- <dia:attribute name="alignment">
- <dia:enum val="1"/>
- </dia:attribute>
- </dia:composite>
- </dia:attribute>
- </dia:object>
- <dia:object type="Flowchart - Box" version="0" id="O6">
- <dia:attribute name="obj_pos">
- <dia:point val="32,16"/>
- </dia:attribute>
- <dia:attribute name="obj_bb">
- <dia:rectangle val="31.95,15.95;43.05,21.05"/>
- </dia:attribute>
- <dia:attribute name="elem_corner">
- <dia:point val="32,16"/>
- </dia:attribute>
- <dia:attribute name="elem_width">
- <dia:real val="11"/>
- </dia:attribute>
- <dia:attribute name="elem_height">
- <dia:real val="5"/>
- </dia:attribute>
- <dia:attribute name="show_background">
- <dia:boolean val="true"/>
- </dia:attribute>
- <dia:attribute name="padding">
- <dia:real val="0.5"/>
- </dia:attribute>
- <dia:attribute name="text">
- <dia:composite type="text">
- <dia:attribute name="string">
- <dia:string>#STORAGE
-(The Payload)
-archive.org / knowledgeforge.net
-repositories (svn/hg/git/...)#</dia:string>
- </dia:attribute>
- <dia:attribute name="font">
- <dia:font family="sans" style="0" name="Helvetica"/>
- </dia:attribute>
- <dia:attribute name="height">
- <dia:real val="0.80000000000000004"/>
- </dia:attribute>
- <dia:attribute name="pos">
- <dia:point val="37.5,17.4425"/>
- </dia:attribute>
- <dia:attribute name="color">
- <dia:color val="#000000"/>
- </dia:attribute>
- <dia:attribute name="alignment">
- <dia:enum val="1"/>
- </dia:attribute>
- </dia:composite>
- </dia:attribute>
- </dia:object>
- <dia:object type="Flowchart - Ellipse" version="0" id="O7">
- <dia:attribute name="obj_pos">
- <dia:point val="22,22.5"/>
- </dia:attribute>
- <dia:attribute name="obj_bb">
- <dia:rectangle val="21.95,22.45;26.05,26.2338"/>
- </dia:attribute>
- <dia:attribute name="elem_corner">
- <dia:point val="22,22.5"/>
- </dia:attribute>
- <dia:attribute name="elem_width">
- <dia:real val="4.0000000000000009"/>
- </dia:attribute>
- <dia:attribute name="elem_height">
- <dia:real val="3.6838104413780557"/>
- </dia:attribute>
- <dia:attribute name="show_background">
- <dia:boolean val="true"/>
- </dia:attribute>
- <dia:attribute name="padding">
- <dia:real val="0.35355339059327379"/>
- </dia:attribute>
- <dia:attribute name="text">
- <dia:composite type="text">
- <dia:attribute name="string">
- <dia:string>#API
-(JSON)#</dia:string>
- </dia:attribute>
- <dia:attribute name="font">
- <dia:font family="sans" style="0" name="Helvetica"/>
- </dia:attribute>
- <dia:attribute name="height">
- <dia:real val="0.80000000000000004"/>
- </dia:attribute>
- <dia:attribute name="pos">
- <dia:point val="24,24.0844"/>
- </dia:attribute>
- <dia:attribute name="color">
- <dia:color val="#000000"/>
- </dia:attribute>
- <dia:attribute name="alignment">
- <dia:enum val="1"/>
- </dia:attribute>
- </dia:composite>
- </dia:attribute>
- </dia:object>
- <dia:object type="Standard - ZigZagLine" version="1" id="O8">
- <dia:attribute name="obj_pos">
- <dia:point val="24,27.9496"/>
- </dia:attribute>
- <dia:attribute name="obj_bb">
- <dia:rectangle val="23.95,26.1636;24.05,27.9496"/>
- </dia:attribute>
- <dia:attribute name="orth_points">
- <dia:point val="24,27.9496"/>
- <dia:point val="24,27.9496"/>
- <dia:point val="24,26.2343"/>
- <dia:point val="24,26.2343"/>
- </dia:attribute>
- <dia:attribute name="orth_orient">
- <dia:enum val="0"/>
- <dia:enum val="1"/>
- <dia:enum val="0"/>
- </dia:attribute>
- <dia:attribute name="autorouting">
- <dia:boolean val="true"/>
- </dia:attribute>
- <dia:attribute name="end_arrow">
- <dia:enum val="22"/>
- </dia:attribute>
- <dia:attribute name="end_arrow_length">
- <dia:real val="0.5"/>
- </dia:attribute>
- <dia:attribute name="end_arrow_width">
- <dia:real val="0.5"/>
- </dia:attribute>
- <dia:connections>
- <dia:connection handle="0" to="O4" connection="16"/>
- <dia:connection handle="1" to="O7" connection="16"/>
- </dia:connections>
- </dia:object>
- <dia:object type="Standard - ZigZagLine" version="1" id="O9">
- <dia:attribute name="obj_pos">
- <dia:point val="32.9497,29.5"/>
- </dia:attribute>
- <dia:attribute name="obj_bb">
- <dia:rectangle val="26.9797,29.45;32.9497,29.55"/>
- </dia:attribute>
- <dia:attribute name="orth_points">
- <dia:point val="32.9497,29.5"/>
- <dia:point val="32.9497,29.5"/>
- <dia:point val="27.0504,29.5"/>
- <dia:point val="27.0504,29.5"/>
- </dia:attribute>
- <dia:attribute name="orth_orient">
- <dia:enum val="0"/>
- <dia:enum val="1"/>
- <dia:enum val="0"/>
- </dia:attribute>
- <dia:attribute name="autorouting">
- <dia:boolean val="true"/>
- </dia:attribute>
- <dia:attribute name="end_arrow">
- <dia:enum val="22"/>
- </dia:attribute>
- <dia:attribute name="end_arrow_length">
- <dia:real val="0.5"/>
- </dia:attribute>
- <dia:attribute name="end_arrow_width">
- <dia:real val="0.5"/>
- </dia:attribute>
- <dia:connections>
- <dia:connection handle="0" to="O5" connection="16"/>
- <dia:connection handle="1" to="O4" connection="16"/>
- </dia:connections>
- </dia:object>
- <dia:object type="Standard - ZigZagLine" version="1" id="O10">
- <dia:attribute name="obj_pos">
- <dia:point val="29.0453,18.5"/>
- </dia:attribute>
- <dia:attribute name="obj_bb">
- <dia:rectangle val="29.0453,18.45;32.0204,18.55"/>
- </dia:attribute>
- <dia:attribute name="orth_points">
- <dia:point val="29.0453,18.5"/>
- <dia:point val="29.0453,18.5"/>
- <dia:point val="31.9497,18.5"/>
- <dia:point val="31.9497,18.5"/>
- </dia:attribute>
- <dia:attribute name="orth_orient">
- <dia:enum val="0"/>
- <dia:enum val="1"/>
- <dia:enum val="0"/>
- </dia:attribute>
- <dia:attribute name="autorouting">
- <dia:boolean val="true"/>
- </dia:attribute>
- <dia:attribute name="start_arrow">
- <dia:enum val="22"/>
- </dia:attribute>
- <dia:attribute name="start_arrow_length">
- <dia:real val="0.5"/>
- </dia:attribute>
- <dia:attribute name="start_arrow_width">
- <dia:real val="0.5"/>
- </dia:attribute>
- <dia:attribute name="end_arrow">
- <dia:enum val="22"/>
- </dia:attribute>
- <dia:attribute name="end_arrow_length">
- <dia:real val="0.5"/>
- </dia:attribute>
- <dia:attribute name="end_arrow_width">
- <dia:real val="0.5"/>
- </dia:attribute>
- <dia:connections>
- <dia:connection handle="0" to="O2" connection="16"/>
- <dia:connection handle="1" to="O6" connection="16"/>
- </dia:connections>
- </dia:object>
- <dia:object type="Standard - ZigZagLine" version="1" id="O11">
- <dia:attribute name="obj_pos">
- <dia:point val="37.5,27.9504"/>
- </dia:attribute>
- <dia:attribute name="obj_bb">
- <dia:rectangle val="37,20.9994;38,28.0004"/>
- </dia:attribute>
- <dia:attribute name="orth_points">
- <dia:point val="37.5,27.9504"/>
- <dia:point val="37.5,25"/>
- <dia:point val="37.5,25"/>
- <dia:point val="37.5,21.0494"/>
- </dia:attribute>
- <dia:attribute name="orth_orient">
- <dia:enum val="1"/>
- <dia:enum val="0"/>
- <dia:enum val="1"/>
- </dia:attribute>
- <dia:attribute name="autorouting">
- <dia:boolean val="false"/>
- </dia:attribute>
- <dia:attribute name="end_arrow">
- <dia:enum val="22"/>
- </dia:attribute>
- <dia:attribute name="end_arrow_length">
- <dia:real val="0.5"/>
- </dia:attribute>
- <dia:attribute name="end_arrow_width">
- <dia:real val="0.5"/>
- </dia:attribute>
- <dia:connections>
- <dia:connection handle="0" to="O5" connection="16"/>
- <dia:connection handle="1" to="O6" connection="16"/>
- </dia:connections>
- </dia:object>
- <dia:object type="Standard - Text" version="1" id="O12">
- <dia:attribute name="obj_pos">
- <dia:point val="25,11"/>
- </dia:attribute>
- <dia:attribute name="obj_bb">
- <dia:rectangle val="15.0375,9.21016;34.9966,12.7898"/>
- </dia:attribute>
- <dia:attribute name="text">
- <dia:composite type="text">
- <dia:attribute name="string">
- <dia:string>#The "Debian" of Data
-Automating use and reuse of data#</dia:string>
- </dia:attribute>
- <dia:attribute name="font">
- <dia:font family="sans" style="0" name="Helvetica"/>
- </dia:attribute>
- <dia:attribute name="height">
- <dia:real val="1.6000000000000001"/>
- </dia:attribute>
- <dia:attribute name="pos">
- <dia:point val="25,10.3292"/>
- </dia:attribute>
- <dia:attribute name="color">
- <dia:color val="#000000"/>
- </dia:attribute>
- <dia:attribute name="alignment">
- <dia:enum val="1"/>
- </dia:attribute>
- </dia:composite>
- </dia:attribute>
- <dia:attribute name="valign">
- <dia:enum val="2"/>
- </dia:attribute>
- </dia:object>
- <dia:object type="Standard - ZigZagLine" version="1" id="O13">
- <dia:attribute name="obj_pos">
- <dia:point val="24,22.4495"/>
- </dia:attribute>
- <dia:attribute name="obj_bb">
- <dia:rectangle val="23.4975,21.0003;24.4975,22.4995"/>
- </dia:attribute>
- <dia:attribute name="orth_points">
- <dia:point val="24,22.4495"/>
- <dia:point val="24,21.7499"/>
- <dia:point val="23.9975,21.7499"/>
- <dia:point val="23.9975,21.0503"/>
- </dia:attribute>
- <dia:attribute name="orth_orient">
- <dia:enum val="1"/>
- <dia:enum val="0"/>
- <dia:enum val="1"/>
- </dia:attribute>
- <dia:attribute name="autorouting">
- <dia:boolean val="true"/>
- </dia:attribute>
- <dia:attribute name="end_arrow">
- <dia:enum val="22"/>
- </dia:attribute>
- <dia:attribute name="end_arrow_length">
- <dia:real val="0.5"/>
- </dia:attribute>
- <dia:attribute name="end_arrow_width">
- <dia:real val="0.5"/>
- </dia:attribute>
- <dia:connections>
- <dia:connection handle="0" to="O7" connection="16"/>
- <dia:connection handle="1" to="O2" connection="16"/>
- </dia:connections>
- </dia:object>
- <dia:object type="Standard - Text" version="1" id="O14">
- <dia:attribute name="obj_pos">
- <dia:point val="24,22.5"/>
- </dia:attribute>
- <dia:attribute name="obj_bb">
- <dia:rectangle val="24,22.1;24,23.3"/>
- </dia:attribute>
- <dia:attribute name="text">
- <dia:composite type="text">
- <dia:attribute name="string">
- <dia:string>##</dia:string>
- </dia:attribute>
- <dia:attribute name="font">
- <dia:font family="sans" style="0" name="Helvetica"/>
- </dia:attribute>
- <dia:attribute name="height">
- <dia:real val="0.80000000000000004"/>
- </dia:attribute>
- <dia:attribute name="pos">
- <dia:point val="24,22.5"/>
- </dia:attribute>
- <dia:attribute name="color">
- <dia:color val="#000000"/>
- </dia:attribute>
- <dia:attribute name="alignment">
- <dia:enum val="0"/>
- </dia:attribute>
- </dia:composite>
- </dia:attribute>
- <dia:attribute name="valign">
- <dia:enum val="3"/>
- </dia:attribute>
- <dia:connections>
- <dia:connection handle="0" to="O1" connection="8"/>
- </dia:connections>
- </dia:object>
- <dia:object type="Standard - Text" version="1" id="O15">
- <dia:attribute name="obj_pos">
- <dia:point val="24,22.5"/>
- </dia:attribute>
- <dia:attribute name="obj_bb">
- <dia:rectangle val="24,22.1;24,23.3"/>
- </dia:attribute>
- <dia:attribute name="text">
- <dia:composite type="text">
- <dia:attribute name="string">
- <dia:string>##</dia:string>
- </dia:attribute>
- <dia:attribute name="font">
- <dia:font family="sans" style="0" name="Helvetica"/>
- </dia:attribute>
- <dia:attribute name="height">
- <dia:real val="0.80000000000000004"/>
- </dia:attribute>
- <dia:attribute name="pos">
- <dia:point val="24,22.5"/>
- </dia:attribute>
- <dia:attribute name="color">
- <dia:color val="#000000"/>
- </dia:attribute>
- <dia:attribute name="alignment">
- <dia:enum val="0"/>
- </dia:attribute>
- </dia:composite>
- </dia:attribute>
- <dia:attribute name="valign">
- <dia:enum val="3"/>
- </dia:attribute>
- <dia:connections>
- <dia:connection handle="0" to="O1" connection="8"/>
- </dia:connections>
- </dia:object>
- <dia:object type="Standard - Text" version="1" id="O16">
- <dia:attribute name="obj_pos">
- <dia:point val="20,15"/>
- </dia:attribute>
- <dia:attribute name="obj_bb">
- <dia:rectangle val="20,14.1875;27.865,15.595"/>
- </dia:attribute>
- <dia:attribute name="text">
- <dia:composite type="text">
- <dia:attribute name="string">
- <dia:string>#The CKAN System#</dia:string>
- </dia:attribute>
- <dia:attribute name="font">
- <dia:font family="sans" style="0" name="Helvetica"/>
- </dia:attribute>
- <dia:attribute name="height">
- <dia:real val="1.2"/>
- </dia:attribute>
- <dia:attribute name="pos">
- <dia:point val="20,15"/>
- </dia:attribute>
- <dia:attribute name="color">
- <dia:color val="#000000"/>
- </dia:attribute>
- <dia:attribute name="alignment">
- <dia:enum val="0"/>
- </dia:attribute>
- </dia:composite>
- </dia:attribute>
- <dia:attribute name="valign">
- <dia:enum val="3"/>
- </dia:attribute>
- </dia:object>
- <dia:object type="Standard - Text" version="1" id="O17">
- <dia:attribute name="obj_pos">
- <dia:point val="11,22"/>
- </dia:attribute>
- <dia:attribute name="obj_bb">
- <dia:rectangle val="11,21.6;11,22.8"/>
- </dia:attribute>
- <dia:attribute name="text">
- <dia:composite type="text">
- <dia:attribute name="string">
- <dia:string>##</dia:string>
- </dia:attribute>
- <dia:attribute name="font">
- <dia:font family="sans" style="0" name="Helvetica"/>
- </dia:attribute>
- <dia:attribute name="height">
- <dia:real val="0.80000000000000004"/>
- </dia:attribute>
- <dia:attribute name="pos">
- <dia:point val="11,22"/>
- </dia:attribute>
- <dia:attribute name="color">
- <dia:color val="#000000"/>
- </dia:attribute>
- <dia:attribute name="alignment">
- <dia:enum val="0"/>
- </dia:attribute>
- </dia:composite>
- </dia:attribute>
- <dia:attribute name="valign">
- <dia:enum val="3"/>
- </dia:attribute>
- <dia:connections>
- <dia:connection handle="0" to="O0" connection="8"/>
- </dia:connections>
- </dia:object>
- <dia:object type="Standard - Text" version="1" id="O18">
- <dia:attribute name="obj_pos">
- <dia:point val="8,20"/>
- </dia:attribute>
- <dia:attribute name="obj_bb">
- <dia:rectangle val="8,19.3013;14.0938,20.5375"/>
- </dia:attribute>
- <dia:attribute name="text">
- <dia:composite type="text">
- <dia:attribute name="string">
- <dia:string>#Related Services#</dia:string>
- </dia:attribute>
- <dia:attribute name="font">
- <dia:font family="sans" style="0" name="Helvetica"/>
- </dia:attribute>
- <dia:attribute name="height">
- <dia:real val="1"/>
- </dia:attribute>
- <dia:attribute name="pos">
- <dia:point val="8,20"/>
- </dia:attribute>
- <dia:attribute name="color">
- <dia:color val="#000000"/>
- </dia:attribute>
- <dia:attribute name="alignment">
- <dia:enum val="0"/>
- </dia:attribute>
- </dia:composite>
- </dia:attribute>
- <dia:attribute name="valign">
- <dia:enum val="3"/>
- </dia:attribute>
- </dia:object>
- <dia:object type="Standard - ZigZagLine" version="1" id="O19">
- <dia:attribute name="obj_pos">
- <dia:point val="13.9854,22.5"/>
- </dia:attribute>
- <dia:attribute name="obj_bb">
- <dia:rectangle val="13.9354,18;19.05,22.55"/>
- </dia:attribute>
- <dia:attribute name="orth_points">
- <dia:point val="13.9854,22.5"/>
- <dia:point val="16.4927,22.5"/>
- <dia:point val="16.4927,18.5"/>
- <dia:point val="19,18.5"/>
- </dia:attribute>
- <dia:attribute name="orth_orient">
- <dia:enum val="0"/>
- <dia:enum val="1"/>
- <dia:enum val="0"/>
- </dia:attribute>
- <dia:attribute name="autorouting">
- <dia:boolean val="true"/>
- </dia:attribute>
- <dia:attribute name="end_arrow">
- <dia:enum val="22"/>
- </dia:attribute>
- <dia:attribute name="end_arrow_length">
- <dia:real val="0.5"/>
- </dia:attribute>
- <dia:attribute name="end_arrow_width">
- <dia:real val="0.5"/>
- </dia:attribute>
- <dia:connections>
- <dia:connection handle="0" to="O3" connection="16"/>
- <dia:connection handle="1" to="O2" connection="7"/>
- </dia:connections>
- </dia:object>
- </dia:layer>
-</dia:diagram>
--- a/doc/conf.py Thu Jul 28 16:45:02 2011 +0100
+++ b/doc/conf.py Thu Jul 28 17:18:48 2011 +0100
@@ -29,7 +29,7 @@
extensions = ['sphinx.ext.autodoc']
# Add any paths that contain templates here, relative to this directory.
-templates_path = ['.templates']
+templates_path = ['_templates']
# The suffix of source filenames.
source_suffix = '.rst'
@@ -41,8 +41,10 @@
master_doc = 'index'
# General information about the project.
-project = u'Comprehensive Knowledge Archive Network (CKAN)'
+project = u'CKAN (Comprehensive Knowledge Archive Network)'
+project_short_name = u'CKAN'
copyright = u'2009, Open Knowledge Foundation'
+html_show_sphinx = False
# The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the
@@ -92,31 +94,40 @@
# Options for HTML output
# -----------------------
+html_theme = 'default'
+html_theme_options = {
+"relbarbgcolor": "#777",
+'sidebarbgcolor': '#F2F2F2',
+'sidebartextcolor': 'black',
+'sidebarlinkcolor': '#355F7C',
+'headfont': 'Trebuchet MS'
+}
+
# The style sheet to use for HTML and HTML Help pages. A file of that name
# must exist either in Sphinx' static/ path, or in one of the custom paths
# given in html_static_path.
-html_style = 'default.css'
+#html_style = 'default.css'
# The name for this set of Sphinx documents. If None, it defaults to
# "<project> v<release> documentation".
-#html_title = None
+html_title = "%s v%s Administration Guide" % (project, release)
# A shorter title for the navigation bar. Default is the same as html_title.
-#html_short_title = None
+html_short_title = "%s Admin Guide" % (project_short_name)
# The name of an image file (relative to this directory) to place at the top
# of the sidebar.
-#html_logo = None
+html_logo = 'images/ckan_logo_box.png'
# The name of an image file (within the static path) to use as favicon of the
# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32
# pixels large.
-#html_favicon = None
+html_favicon = 'images/favicon.ico'
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
-html_static_path = ['.static']
+#html_static_path = ['.static']
# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
# using the given strftime format.
--- a/doc/configuration.rst Thu Jul 28 16:45:02 2011 +0100
+++ b/doc/configuration.rst Thu Jul 28 17:18:48 2011 +0100
@@ -1,23 +1,26 @@
-CKAN Configuration
-==================
+.. index::
+ single: config file
-A CKAN instance is configured with the .ini file in the root ckan directory. The most important parameter to set is the `sqlalchemy.url` giving the database connection. But there are also several additional options to change the way the CKAN site operates.
+=====================================
+Reference: CKAN Configuration Options
+=====================================
-On a production machine the file will be probably be named after the site name. e.g. `ca.ckan.net.ini` On a development machine it is probably `development.ini`
+You can change many important CKAN settings in the CKAN config file. This is the file called ``std.ini`` that you first encountered in :ref:`create-admin-user`. It is usually located at ``/etc/ckan/std/std.ini``.
-On a new installation of ckan, you have to create a new config file from the template, something like this::
+The file is well-documented, but we recommend reading this section in full to learn about the CKAN config options available to you.
- paster --plugin ckan make-config ckan ca.ckan.net.ini
+.. note:: After editing this file, you will need to restart Apache for the changes to take effect.
-This creates ca.ckan.net.ini based on the template for this file in ckan/config/deployment.ini_tmpl
+.. note:: The CKAN config file also includes general Pylons options. All CKAN-specific settings are in the `[app:main]` section.
-There are several general Pylons options, and all the CKAN-specific ones are in the `[app:main]` section.
+Database Settings
+-----------------
-Once the config file is changed, Apache needs to be restarted to read in the new changes.
-
+.. index::
+ single: sqlalchemy.url
sqlalchemy.url
---------------
+^^^^^^^^^^^^^^
Example::
@@ -28,20 +31,47 @@
sqlalchemy.url = postgres://USERNAME:PASSWORD@HOST/DBNAME
-package_form
-------------
+Front-End Settings
+------------------
+
+
+.. index::
+ single: site_description
+
+site_description
+^^^^^^^^^^^^^^^^
Example::
- package_form = ca
+ ckan.site_description=
-Default value: ``standard``
+Default value: (none)
-This sets the name of the form to use when editing a package. This can be a form defined in the core CKAN code or in another setuputils-managed python module. The only requirement is that the setup.py has an entrypoint for the form defined in the `ckan.forms` section. See :doc:`forms`
+This is for a description, or tag line for the site, as displayed in the header of the CKAN web interface.
+.. index::
+ single: site_logo
+
+site_logo
+^^^^^^^^^
+
+Example::
+
+ ckan.site_logo=/images/ckan_logo_fullname_long.png
+
+Default value: (none)
+
+This sets the logo used in the title bar.
+
+.. index::
+ single: site_url
+
+
+.. index::
+ single: package_hide_extras
package_hide_extras
--------------------
+^^^^^^^^^^^^^^^^^^^
Example::
@@ -49,12 +79,15 @@
Default value: (empty)
-This sets a space-seperated list of extra field key values which will not be shown on the package read page. While this is useful to create internal notes etc., it is not a security measure in any way. The keys will
-still be available via the API and in revision diffs.
+This sets a space-separated list of extra field key values which will not be shown on the package read page.
+.. warning:: While this is useful to e.g. create internal notes, it is not a security measure. The keys will still be available via the API and in revision diffs.
+
+.. index::
+ single: rdf_packages
rdf_packages
-------------
+^^^^^^^^^^^^
Example::
@@ -68,9 +101,30 @@
3. A visible RDF link on the page. e.g. `<a href="http://semantic.ckan.net/record/b410e678-8a96-40cf-8e46-e8bd4bf02684.rdf">`
+.. index::
+ single: dumps_url, dumps_format
+
+dumps_url & dumps_format
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+Example::
+
+ ckan.dumps_url = http://ckan.net/dump/
+ ckan.dumps_format = CSV/JSON
+
+If there is a page which allows you to download a dump of the entire catalogue then specify the URL and the format here, so that it can be advertised in the web interface. ``dumps_format`` is just a string for display.
+
+For more information on using dumpfiles, see :doc:`database_dumps`.
+
+
+Cache Settings
+--------------
+
+.. index::
+ single: cache_validation_enabled
cache_validation_enabled
-------------------------
+^^^^^^^^^^^^^^^^^^^^^^^^
Example::
@@ -78,13 +132,15 @@
Default value: ``True``
-This option determines whether browsers (or other caching services running between the browser and CKAN) are helped to cache particular CKAN pages, by validating when the page content hasn't changed. This is achieved using ETags headers provided by CKAN, which is a hash that changes when the content has changed.
+This option determines whether browsers (or other caching services running between the browser and CKAN) are helped to cache particular CKAN pages, by validating when the page content hasn't changed. This is achieved using ETag headers provided by CKAN, which is a hash that changes when the content has changed.
-Developers editing the templates should set this to False, since Etags hashes don't look for template changes.
+Developers editing the templates should set this to False, since ETag hashes don't look for template changes.
+.. index::
+ single: cache_enabled
cache_enabled
--------------
+^^^^^^^^^^^^^
Example::
@@ -92,13 +148,13 @@
Default value: ``False``
-Setting this option to True turns on several server-side caches. When the caching is on, caching can be further configured as follows. (The key has been renamed from ``cache_enabled``, which is deprecated, but still works ``ckan.cache_enabled`` for now.)
+Setting this option to True turns on several server-side caches. When caching is on, caching can be further configured as follows.
To set the type of Beaker storage::
beaker.cache.type = file
-To set the expiry times (in seconds) for specific controllers (which use the proxy_cache) specifiy the methods like this::
+To set the expiry times (in seconds) for specific controllers (which use the proxy_cache) specify the methods like this::
ckan.controllers.package.list.expires = 600
ckan.controllers.tag.read.expires = 600
@@ -107,13 +163,19 @@
ckan.controllers.apiv2.package.list.expires = 600
ckan.controllers.apiv2.package.show.expires = 600
-There is also en option to set the max-age value of static files delivered by
-paster::
+There is also an option to set the max-age value of static files delivered by paster::
ckan.static_max_age = 3600
+
+Authentication Settings
+-----------------------
+
+.. index::
+ single: openid_enabled
+
openid_enabled
---------------
+^^^^^^^^^^^^^^
Example::
@@ -121,33 +183,21 @@
Default value: ``True``
-Setting this option to Fase turns off openid for login.
+CKAN operates a delegated authentication model based on `OpenID <http://openid.net/>`_.
+Setting this option to False turns off OpenID for login.
-licenses_group_url
-------------------
-A url pointing to a JSON file containing a list of license objects. This list
-determines the licenses offered by the system to users, for example when
-creating or editing a package.
+.. _config-i18n:
-This is entirely optional -- by default the system will use the ckan list of
-licenses available in the Licenses package.
+Internationalisation Settings
+-----------------------------
-.. _licenses python package: http://pypi.python.org/pypi/licenses
-
-More details about the license objects including the license format and some
-example license lists can be found on the open license service at
-http://licenses.opendefinition.org/.
-
-Examples::
-
- licenses_group_url = file:///path/to/my/local/json-list-of-licenses.js
- licenses_group_url = http://licenses.opendefinition.org/2.0/ckan_original
-
+.. index::
+ single: lang
lang
-----
+^^^^
Example::
@@ -155,58 +205,108 @@
Default value: ``en`` (English)
-Use this to specify the language of the text displayed in the CKAN web UI. This requires a suitable `mo` file installed for the language. For more information on internationalization, see: http://wiki.okfn.org/ckan/i18n#DeployingaTranslation
+Use this to specify the language of the text displayed in the CKAN web UI. This requires a suitable `mo` file installed for the language. For more information on internationalization, see :doc:`i18n`.
+Theming Settings
+----------------
+
+.. index::
+ single: extra_template_paths
extra_template_paths
---------------------
+^^^^^^^^^^^^^^^^^^^^
Example::
extra_template_paths=/home/okfn/brazil_ckan_config/templates
-To customise the display of CKAN you can supply replacements for the Genshi template files. Use this option to specify where CKAN should look for them, before reverting to the 'ckan/templates' folder. You can supply more than one folder, separating the paths with a comma (,).
+To customise the display of CKAN you can supply replacements for the Genshi template files. Use this option to specify where CKAN should look for additional templates, before reverting to the ``ckan/templates`` folder. You can supply more than one folder, separating the paths with a comma (,).
-The example value for the extra_template_paths option could, for example, be used to override CKAN templates with these ones:
+For more information on theming, see :doc:`theming`.
- * /home/okfn/brazil_ckan_config/templates/layout.html
- * /home/okfn/brazil_ckan_config/templates/package/edit.html
-
-More details about this feature are found at: http://wiki.okfn.org/ckan/doc/theme
-
+.. index::
+ single: extra_public_paths
extra_public_paths
-------------------
+^^^^^^^^^^^^^^^^^^
Example::
extra_public_paths = /home/okfn/brazil_ckan_config/public
-To customise the display of CKAN you can supply replacements for staticly served files such as HTML, CSS, script and PNG files. Use this option to specify where CKAN should look for them, before reverting to the 'ckan/public' folder. You can supply more than one folder, separating the paths with a comma (,).
+To customise the display of CKAN you can supply replacements for static files such as HTML, CSS, script and PNG files. Use this option to specify where CKAN should look for additional files, before reverting to the ``ckan/public`` folder. You can supply more than one folder, separating the paths with a comma (,).
-The example value for the extra_public_paths option could, for example, be used to provide an image and stylesheet:
+For more information on theming, see :doc:`theming`.
- * /home/okfn/brazil_ckan_config/public/images/brazil.png
- * /home/okfn/brazil_ckan_config/public/css/extra.css
-More details about this feature are found at: http://wiki.okfn.org/ckan/doc/theme
+Form Settings
+-------------
+.. index::
+ single: package_form
+
+package_form
+^^^^^^^^^^^^
+
+Example::
+
+ package_form = ca
+
+Default value: ``standard``
+
+This sets the name of the form to use when editing a package. This can be a form defined in the core CKAN code or in another setuputils-managed python module. The only requirement is that the ``setup.py`` file has an entry point for the form defined in the ``ckan.forms`` section.
+
+For more information on forms, see :doc:`forms`.
+
+.. _config-package-urls:
+
+.. index::
+ single: package_new_return_url, package_edit_return_url
package_new_return_url & package_edit_return_url
-------------------------------------------------
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Example::
package_new_return_url = http://datadotgc.ca/new_dataset_complete?name=<NAME>
package_edit_return_url = http://datadotgc.ca/dataset/<NAME>
-To allow the Edit Package and New Package forms to be integrated into a third party interface, setting these options allows you to set a the return address. So when the user has completed the form and presses 'commit', the user is redirected to the URL specified.
+If integrating the Edit Package and New Package forms into a third-party interface, setting these options allows you to set the return address. When the user has completed the form and presses 'commit', the user is redirected to the URL specified.
-The '<NAME>' string is replaced with the name of the package edited. Full details of this process are given in :doc:`form-integration`.
+The ``<NAME>`` string is replaced with the name of the package edited. Full details of this process are given in :doc:`form-integration`.
+.. index::
+ single: licenses_group_url
+
+licenses_group_url
+^^^^^^^^^^^^^^^^^^
+
+A url pointing to a JSON file containing a list of licence objects. This list
+determines the licences offered by the system to users, for example when
+creating or editing a package.
+
+This is entirely optional - by default, the system will use the CKAN list of
+licences available in the `Python licenses package <http://pypi.python.org/pypi/licenses>`_.
+
+More details about the CKAN license objects - including the licence format and some
+example licence lists - can be found at the `Open Licenses Service
+<http://licenses.opendefinition.org/>`_.
+
+Examples::
+
+ licenses_group_url = file:///path/to/my/local/json-list-of-licenses.js
+ licenses_group_url = http://licenses.opendefinition.org/2.0/ckan_original
+
+
+Messaging Settings
+------------------
+
+.. index::
+ single: carrot_messaging_library
+
carrot_messaging_library
-------------------------
+^^^^^^^^^^^^^^^^^^^^^^^^
Example::
@@ -222,11 +322,13 @@
* ``queue`` - native Python Queue (default) - NB this doesn't work inter-process
-See `carrot documentation <http://packages.python.org/carrot/index.html>`_ for details.
+See the `Carrot documentation <http://packages.python.org/carrot/index.html>`_ for details.
+.. index::
+ single: amqp_hostname, amqp_port, amqp_user_id, amqp_password
amqp_hostname, amqp_port, amqp_user_id, amqp_password
------------------------------------------------------
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Example::
@@ -235,11 +337,16 @@
amqp_user_id=guest
amqp_password=guest
-These are the setup parameters for AMQP messaging. These only apply if the messageing library has been set to use AMQP (see `carrot_messaging_library`_). The values given in the example are the default values.
+These are the setup parameters for AMQP messaging. These only apply if the messaging library has been set to use AMQP (see `carrot_messaging_library`_). The values given above are the default values.
+Search Settings
+---------------
+
+.. index::
+ single: build_search_index_synchronously
build_search_index_synchronously
---------------------------------
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Example::
@@ -248,11 +355,15 @@
Default (if you don't define it)::
indexing is on
-This controls the operation of the CKAN Postgres full text search indexing. If you don't define this option then indexing is on. You will want to turn this off if you want to use a different search engine for CKAN (e.g. SOLR). In this case you need to define the option equal to blank (as in the given example).
+This controls the operation of the CKAN Postgres full text search indexing. If you don't define this option then indexing is on. You will want to turn this off if you want to use a different search engine for CKAN (e.g. Solr). In this case you need to define the option equal to blank (as in the example).
+.. _config-search-backend:
search_backend
---------------
+^^^^^^^^^^^^^^
+
+.. index::
+ single: search_backend
Example::
@@ -260,24 +371,31 @@
Default value: ``sql``
-This controls the type of search backend. Currently valid values are ``sql`` (meaning Postgres full text search) and ``solr``. If you specify ``sql`` then ensure indexing is on (`build_search_index_synchronously`_ is not defined). If you specify ``solr`` then ensure you specify a `solr_url`_.
+This controls the type of search backend. Currently valid values are ``sql`` (meaning Postgres full text search) and ``solr`` (meaning Solr). If you specify ``sql`` then ensure indexing is on (`build_search_index_synchronously`_ is not defined). If you specify ``solr`` then ensure you specify a `solr_url`_.
+.. index::
+ single: solr_url
solr_url
---------
+^^^^^^^^
Example::
solr_url = http://solr.okfn.org/solr/test.ckan.net
-This configures SOLR search, (if selected with 'search_backend'_). Running solr will require a schema.xml file, such as the one
-in `the ckan-solr-index repository <http://bitbucket.org/pudo/ckan-solr-index>`_.
+This configures Solr search (if selected with `search_backend`_). Running Solr will require a schema.xml file, such as the one in `the ckanext-solr repository <https://bitbucket.org/okfn/ckanext-solr/src>`_.
-Optionally, ``solr_user`` and ``solr_password`` can also be passed along to specify HTTP Basic authentication details for all solr requests.
+Optionally, ``solr_user`` and ``solr_password`` can also be passed along to specify HTTP Basic authentication details for all Solr requests.
+Site Settings
+-------------
+
+.. index::
+ single: site_title
+
site_title
-----------
+^^^^^^^^^^
Example::
@@ -287,33 +405,11 @@
This sets the name of the site, as displayed in the CKAN web interface.
-
-site_description
-----------------
-
-Example::
-
- ckan.site_description=
-
-Default value: (none)
-
-This is for a description, or tag line for the site, as displayed in the header of the CKAN web interface.
-
-
-site_logo
----------
-
-Example::
-
- ckan.site_logo=/images/ckan_logo_fullname_long.png
-
-Default value: (none)
-
-This sets the logo used in the title bar.
-
+.. index::
+ single: site_url
site_url
---------
+^^^^^^^^
Example::
@@ -321,13 +417,13 @@
Default value: (none)
-The primary URL used by this site. Uses::
+The primary URL used by this site. Used in the API to provide packages with links to themselves in the web UI.
- * in the API to provide packages with links to themselves in the web UI.
-
+.. index::
+ single: api_url
api_url
---------
+^^^^^^^
Example::
@@ -335,71 +431,91 @@
Default value: ``/api``
-The URL which resolves to the CKAN API part of the site. This is useful if the
-API is hosted on a different domain, for example when a third party site uses
+The URL that resolves to the CKAN API part of the site. This is useful if the
+API is hosted on a different domain, for example when a third-party site uses
the forms API.
+Authorization Settings
+----------------------
+
+.. index::
+ single: default_roles
+
default_roles
--------------
+^^^^^^^^^^^^^
-This allows you to set the default authorization roles (i.e. permissions) for new objects. Currently this extends to new packages, groups, authorization groups and the 'system' object. For full details of these, see :doc:`authorization`.
+This allows you to set the default authorization roles (i.e. permissions) for new objects. Currently this extends to new packages, groups, authorization groups and the ``system`` object. For full details of these, see :doc:`authorization`.
-The value is a strict JSON dictionary of user names "visitor" and "logged_in" with lists of their roles.
+The value is a strict JSON dictionary of user names ``visitor`` (any user who is not logged in) and ``logged_in`` (any user who is logged in) with lists of their roles.
Example::
ckan.default_roles.Package = {"visitor": ["editor"], "logged_in": ["editor"]}
ckan.default_roles.Group = {"visitor": ["reader"], "logged_in": ["reader"]}
-With this example setting, visitors (any user who is not logged in) and logged in users can only read packages that get created (only sysadmins can edit).
+With this example setting, visitors and logged-in users can only read packages that get created.
-Defaults: see in ckan/model/authz.py for: ``default_default_user_roles``
+Defaults: see in ``ckan/model/authz.py`` for: ``default_default_user_roles``
+Plugin Settings
+---------------
+
+.. index::
+ single: plugins
+
plugins
--------
+^^^^^^^
Example::
ckan.plugins = disqus synchronous_search datapreview googleanalytics stats storage admin follower
-Specify which CKAN extensions are to be enabled. If you specify an extension but have not installed the code then CKAN will not start. Format in a space separated list of the extension names. The extension name is the key in the [ckan.plugins] section of the extension's setup.py.
+Specify which CKAN extensions are to be enabled.
+.. warning:: If you specify an extension but have not installed the code, CKAN will not start.
-dumps_url & dumps_format
-------------------------
+Format as a space-separated list of the extension names. The extension name is the key in the [ckan.plugins] section of the extension's ``setup.py``. For more information on extensions, see :doc:`extensions`.
-Example::
- ckan.dumps_url = http://ckan.net/dump/
- ckan.dumps_format = CSV/JSON
-If there is a page which allows you to download a dump of the entire catalogue then specify the URL and the format here, so that it can be advertised in the web interface. The dumps_format is just a string for display.
+Directory Settings
+------------------
+.. index::
+ single: log_dir
log_dir
--------
+^^^^^^^
Example::
ckan.log_dir = /var/log/ckan/
-This is a directory where CKAN cron scripts (if there are any installed) should write log files to. Note: this setting is nothing to do with the main CKAN log file, whose filepath is set in the [handler_file] args.
+This is the directory to which CKAN cron scripts (if there are any installed) should write log files.
+.. note:: This setting is nothing to do with the main CKAN log file, whose filepath is set in the ``[handler_file]`` args.
+
+.. index::
+ single: dump_dir
dump_dir
---------
+^^^^^^^^
Example::
ckan.dump_dir = /var/lib/ckan/dump/
-This is a directory where JSON or CSV dumps of the database are to be written, assuming a script has been installed to do this. Note it is usual to setup the apache config to serve this directory.
+This is the directory to which JSON or CSV dumps of the database are to be written, assuming a script has been installed to do this.
+.. note:: It is usual to set up the Apache config to serve this directory.
+
+.. index::
+ single: backup_dir
backup_dir
-----------
+^^^^^^^^^^
Example::
--- a/doc/database_dumps.rst Thu Jul 28 16:45:02 2011 +0100
+++ b/doc/database_dumps.rst Thu Jul 28 17:18:48 2011 +0100
@@ -1,24 +1,47 @@
-Database dumps
+Database Dumps
==============
-It's often useful to allow users to download the complete CKAN database in a dumpfile. For example, you can download ckan.net daily dump at: http://ckan.net/dump/ in JSON format: ckan.net-daily.json.gz
+It's often useful to allow users to download a complete CKAN database in a dumpfile.
+For example, you can download ckan.net's daily dump at: http://ckan.net/dump/ in JSON format. The file is called ``ckan.net-daily.json.gz``.
-Creating a dump
+Creating a Dump
-----------------
-Your dump script needs to enable the python environment and run the paster command. For example you could create `/home/okfn/var/srvc/ckan.net/dump.sh`::
+We provide two ``paster`` methods to create dumpfiles.
+
+* ``db simple-dump-json`` - A simple dumpfile, useful to create a public listing of the packages with no user information. All packages are dumped, including deleted packages and ones with strict authorization.
+* ``db dump`` - A more complicated dumpfile, useful for backups. Replicates the database completely, including users, their personal info and API keys, and hence should be kept private.
+
+For more information on paster, see :doc:`paster`.
+
+Using db simple-dump-json
++++++++++++++++++++++++++
+
+If you are using a Python environment, as part of a development installation, first enable the environment::
. /home/okfn/var/srvc/ckan.net/pyenv/bin/activate || exit 1
- paster --plugin=ckan db simple-dump-json /home/okfn/var/srvc/ckan.net/dumps/ckan.net-daily.json --config=/home/okfn/var/srvc/ckan.net/ckan.net.ini
- gzip /home/okfn/var/srvc/ckan.net/dumps/ckan.net-daily.json
-You could change simple-dump-json to simple-dump-csv if you want CSV format instead of JSON. Or you could run both!
+Then create and zip the dumpfile::
-These dump functions dump the entire database as it is stored in CKAN, omitting user account details.
+ paster --plugin=ckan db simple-dump-json /var/srvc/ckan/dumps/ckan.net-daily.json --config=/etc/ckan/std/std.ini
+ gzip /var/srvc/ckan/dumps/ckan.net-daily.json
+Change ``simple-dump-json`` to ``simple-dump-csv`` if you want CSV format instead of JSON.
-Daily dumps
+Using db dump
++++++++++++++
+
+If you are using a Python environment, as part of a development installation, first enable the environment::
+
+ . /var/srvc/ckan/pyenv/bin/activate || exit 1
+
+Then create and zip the dumpfile::
+
+ paster --plugin=ckan db dump /var/srvc/ckan/dumps/ckan.net-daily --config=/etc/ckan/std/std.ini
+ gzip /var/srvc/ckan/dumps/ckan.net-daily
+
+Daily Dumps
-----------
You can set the dump to be created daily with a cron job.
@@ -31,13 +54,12 @@
0 21 * * * /home/okfn/var/srvc/ckan.net/dump.sh
-
-Serving the files
+Serving the Files
-----------------
Some simple additions to the Apache config can serve the files to users in a directory listing.
-To do this, add these lines to the virtual host config (e.g. `/etc/apache2/sites-enabled/ckan.net`)::
+To do this, add these lines to your virtual host config (e.g. ``/etc/apache2/sites-enabled/ckan.net``)::
Alias /dump/ /home/okfn/var/srvc/ckan.net/dumps/
@@ -46,4 +68,3 @@
SetHandler None
Options +Indexes
</Location>
-
--- a/doc/deb.rst Thu Jul 28 16:45:02 2011 +0100
+++ /dev/null Thu Jan 01 00:00:00 1970 +0000
@@ -1,902 +0,0 @@
-CKAN's Approach to Dependencies
-+++++++++++++++++++++++++++++++
-
-.. WARNING::
- This document is still under development, use only if you are a member
- of the CKAN team who wishes to be an early adopter and are interested in
- experimenting with virtual machines.
-
-.. contents ::
-
-Abstract
-========
-
-A typical CKAN install can have many dependencies, in the form of required
-system software such as PostgreSQL, Python libraries such as SQLAlchemy and
-other CKAN extensions such as ``ckanext-qa`` or ``ckanext-harvest``.
-
-As such we have to deal with lots of interdependencies which are often
-different depending on the combinations of features a particular CKAN install
-requires.
-
-There are three audiences we are primarily targetting that our dependency
-approach is designed to support:
-
-* Interested parties who just want to get a CKAN instance installed quickly
- and easily to test it or to begin contributing
-* CKAN developers who want to have editable versions of all of the dependant
- libraries so that they can improve them
-* System administrators who want to be able to deploy and upgrade CKAN
- instances quickly, easily and reliably
-* Deployment and test managers who want to be confident that the live system
- is running off exactly the libraries they have tested and configured in
- exactly the same way
-
-In order to support these three groups, we allow installation of CKAN in two ways:
-
-* Development install via ``pip`` as described in the `README.rst <../README.html>`_ file
-* Production install on Ubuntu Lucid 10.04 LTS via ``apt-get install``
-
-The old instructions for a `production deployment <../deployment.html>`_ can
-also be followed but these will be deprecated over time as the ``.deb``
-packaging approach becomes better documented and understood.
-
-Virutally all CKAN instances currently run on Ubuntu Lucid 10.04 LTS so we have
-decided that this is the only officially supported production deployment
-platform. Of course CKAN will also run on any modern Mac or Linux distribution
-but if you go down this route you'll need to do a development install.
-
-In this document I'll explain in detail how and why we package CKAN the way we
-do.
-
-
-Overview of CKAN's Structure from a Packaging Point of View
-===========================================================
-
-There are conceptually two main parts to any CKAN installation:
-
-* Python libraries such as CKAN itself, SQLAlchemy, owslib and all the CKAN
- extensions libraries that a particular CKAN or customized CKAN rely on
-* The configuration and deployment scripts that lead to a running CKAN
- application (which may be a vanilla CKAN or a client-specific version (eg
- CKAN, CKAN-DGU, CKAN-DATANL etc)
-* Third party servers that CKAN relies on (eg Apache, PostgreSQL etc)
-
-Luckily all third party servers that CKAN relies on are all ready packaged in
-Ubuntu Lucid for us so we don't need to worry about packaging them. We do need
-to worry about configuring them though. Let's look more at the other two.
-
-Python Libraries
- The Python libraries CKAN relies on all have their own interdependencies.
- If the libraries have already been packaged as ``.deb`` files we don't need to worry about
- them because their dependencies will already be specified in the package.
- Other libraries usually have their dependencies specified as the ``install_requires`` line in their
- ``setup.py`` files. For the sorts of libraries that are already available
-
-
-Configuration and Deployment
-
-
-Understanding The Dependency Difficulty
----------------------------------------
-
-In the past
-
-
-
-The Three Different Requires Files for CKAN
-===========================================
-
-In order to support both a development install and a package-based install it
-is important that we package the same versions of libraries that we develop
-against. There are three categories of dependant Python libraries:
-
-``present``
- Those that already exist as packages in Ubuntu Lucid
-
-``missing``
- Those that don't exist as packages in Ubuntu Lucid
-
-``conflict``
- Those that have a version which is different from the version in Ubuntu
- Lucid
-
-For each of these categories we have a file in the ``ckan`` source tree's
-``requires`` directory which you can view `here
-<https://bitbucket.org/okfn/ckan/src/default/requires/>`_.
-
-
-Understanding the ``lucid_present.txt`` File
---------------------------------------------
-
-The Python dependencies listed in the ``lucid_present.txt`` file are ``pip``
-installable links to the source tree holding the exact versions of the Python
-dependencies that Ubuntu uses. By running the command below you get development
-copies of the same software that Ubuntu has packaged:
-
-::
-
- pip install --ignore-installed -r lucid_present.txt
-
-We never need to package software in the ``lucid_present.txt`` file because it
-already exists so most of the time you would just install it directly rather
-than running the command above to get source versions. You can see the packages
-you would need to install by looking at the comment at the top of the file. At
-the time of writing it reads:
-
-::
-
- # The CKAN dependencies are already in Lucid and should be installed via
- # apt-get if you are on that platform. If you are using a different platform
- # you can install these dependencies via pip instead.
- #
- # sudo apt-get install python-psycopg2 python-lxml python-sphinx
- # sudo apt-get install python-pylons python-formalchemy python-repoze.who
- # sudo apt-get install python-repoze.who-plugins python-tempita python-zope.interface
-
-Packaging Dependencies Listed in ``lucid_missing.txt``
-------------------------------------------------------
-
-.. note ::
-
- These are already packaged, so you don't need to package them yourself, this
- section just describes how you *could* do if you wanted to.
-
-Python dependencies listed in the ``lucid_missing.txt`` file are ``pip``
-installable links to the source tree holding the exact versions of the Python
-dependencies that CKAN requries. We have an automatic build process which can
-take these entries and automatically generate Ubuntu packages for them. The
-resulting packages are then published to our CKAN apt repository so that they
-can be automatically installed in production environments.
-
-To follow the automatic build process to build the missing packages you can do this:
-
-
-::
-
- sudo apt-get install -y python wget dh-make devscripts build-essential fakeroot cdbs mercurial git-core subversion python-virtualenv
- virtualenv missing
- bin/pip install --ignore-installed -r lucid_missing.txt
- bin/pip install Buildkit
-
-BuildKit script will build and place Debian packages in your ``missing``
-directory. Make sure there is nothing in there that shouldn't be overwritten by
-this script.
-
-Now run the BuildKit command like this:
-
-::
-
- cd missing
- bin/python -m buildkit.update_all .
-
-For each package you'll be loaded into ``vim`` to edit the changelog. Save and
-quit when you are done. Names, version numbers and dependencies are
-automatically generated.
-
-.. caution ::
-
- Most of the time you will never use the automatic process above for lazy
- batch packaging. You'll more likely generate a single package with explicit
- version numbers using the ``buildkit.deb`` command or build your package
- manually. Both approaches are described later.
-
-Packaging Conflicting Python dependencies from ``lucid_conflicts.txt``
-----------------------------------------------------------------------
-
-.. note ::
-
- These are already packaged, so you don't need to package them yourself, this
- section just describes how you *could* do if you wanted to.
-
-Python packages where CKAN depends on a version that is different from the one
-in the Ubuntu Lucid repositories are handled slightly differently. If we were
-to simply package them up and make them available the same way we do with
-missing packages there is a slim chance that any existing software which used
-the other version of the library would stop working. To avoid the risk of
-interfering with other software on the system we take the following approach:
-
-* Create a ``python-ckan-deps`` package with copies of all the libraries we need
-* Change the ``python-ckan`` library to automatically try to import
- ``ckan_deps`` if it can and then adjust the Python's ``sys.path`` just for
- this instance to use the versions of the libraries in ``python-ckan-deps`` in
- preference to any other versions installed.
-
-In this way we can use any arbitrary versions, without introducing conflicts.
-
-.. caution ::
-
- The ``repoze.who`` sets of libraries are nigh-on impossible to package in
- this way so we don't actually package ``repoze.who.openid`` at all, even
- though we need a slightly more recent version. This is such an edge case
- though that you should just install it manually into the system Python
- and not worry too much for the time being.
-
-To actually build the ``python-ckan-deps`` package we follow the semi-manual
-Python packaging approach described next. (The example in the next section is
-actually for a CKAN Python extension called ``python-ckanext-qa`` but the same
-process applies).
-
-hg clone ckan-deps
-
-::
-
- python -m buildkit.deb /path/to/ckan-deps/.. python-ckan-deps 1.3~01+lucid http://ckan.org
-
-
-Semi-Manual Python Packaging
-============================
-
-The easiest way to package a Python library is with a tool called BuildKit I
-wrote specfically for the purpose. This section describes how to use it, but
-even if you don't want to use BuildKit and prefer to understand the
-complexities yourself by reading the `Understanding .deb files`_ section,
-please read this section too so you at least understand the naming conventions
-we are using.
-
-::
-
- pip install buildkit
-
-For each Python package you wish to build a ``.deb`` file for you run the
-``buildkit.deb`` command. Here's an example:
-
-::
-
- python -m buildkit.deb /path/to/virtualenv ckanext-qa 1.3~01+lucid http://ckan.org python-owslib python-ckanext-csw
-
-Let's break this down.
-
-``python -m buildkit.deb``
- This is just how you invoke the command from the command line
-
-``/path/to/virtualenv``
- I think this can just be the path to the directory containing the
- installed source code directory you wish to package, it doesn't
- have to be a virtualenv does it?
-
-``ckanext-qa``
- The lowercase Python package name of the ``.deb`` file to be created.
-
-
-``1.3~01+lucid``
- The version number of the package. There are three parts to this:
-x
- ``1.3``
- This should always match exactly the version number specified in the
- ``setup.py`` file for the library being packaged.
-
- ``~01``
- This is an incrementing number (starting at 01 each time the version
- number above changes) which you change every time you re-package the
- same version of the code to force apt to recognise your new package
- as being more recent than the old one, even if the underlying code
- hasn't changed.
-
- ``+lucid``
- This is a string representing the Debian/Ubuntu distribution that the
- package targets. The apt repository doesn't assign any meaning to it,
- it is just that in order to eventually support more than one flavour
- of Debian or Ubuntu, the packages for each must have different
- filenames *in addition* to being in a separate part of the apt repo
- so we begin this convention now.
-
-``http://ckan.org``
- The homepage for the package, usually ckan.org for ckan extensions.
-
-``python-owslib python-ckanext-csw ... etc``
-
- Any extra arguments are treated as the Debian names of dependencies. These
- always begin ``python-`` for Python libraries and would usually follow
- ``ckanext-`` for all CKAN extensions.
-
- .. tip ::
-
- You can also specify any other Debian
- packages here that are dependcies of the software you are packaging but as
- you'll see later it is usually best to add such dependencies to the
- *packaged application*. See "Packaging CKAN Extensions" for more information.
-
-When you run the command you will get your ``.deb`` file created.
-
-To release an upgrade of a package it must have a higher version number. There
-is a chance you may want to release a more recent version of a package despite
-the fact the underlying version number hasn't changed. For this reason, we
-always add a ``~`` character followed by a two digit number to the end of the
-actual version number as specified in ``setup.py`` for the package.
-
-For example, if the version number for the ``ckanext-qa`` package in the
-example above is ``1.3~01``, a package named
-``python-ckanext-qa_1.3~01_amd64.deb`` would be produced by the command we've
-looked at.
-
-.. note ::
-
- All packages that CKAN itself depends on are already packaged according to
- the settings in the three ``requires`` files that from part of the ``ckan``
- source distribution so you shouldn't need to use the approach above to
- package any of them, you should only need to do this for your own extensions
- or libraries they rely on which aren't already CKAN dependencies. See
- "The Three Different Requires Files" for more information on how packaging
- of the core CKAN dependencies is managed.
-
-Understanding ``.deb`` files
-============================
-
-Broad Structure
----------------
-
-Naming Conventions
-------------------
-
-The base naming conventions we use for packages are as follows:
-
-``ckan``
- Uninstalls CKAN, PostgreSQL, Apache etc. It adds the ``ckan-instance-create`` command which is then the only thing you need to create a new instance.
-
-``python-ckan``
- The CKAN Python library packaged from code at http://bitbucket.org/okfn/ckan
-
-``python-ckanext-*``
- Any CKAN extensions (can be application extensions or library extensions)
-
-``ckan-*``
- Installs a client specific CKAN application
-
-
-
-The ``postinst`` and ``postrm`` files
--------------------------------------
-
-The ``control`` file
---------------------
-
-Extra scripts and permissions
------------------------------
-
-Packaging Python libraries
---------------------------
-
-
-
-Packaging CKAN Extensions
-=========================
-
-There are two types of CKAN extension:
-
-* Client Applications (eg ``ckanext-dgu``, ``ckanext-datanl`` etc)
-* Helpful libraries (eg ``ckanext-qa``, ``ckanext-harvest``, ``ckanext-queue`` etc)
-
-All CKAN extensions (whether client applications or helpful libraries) are
-Python libraries and therefore need packaging. Their ``.deb`` filenames are the
-same as the Python package names but are always prefixed with ``python-`` so
-that ``ckanext-dgu`` becomes ``python-ckanext-dgu`` when packaged as a ``.deb``
-and ``ckanext-harvest`` becomes ``python-ckanext-harvest`` etc.
-
-CKAN extensions which are also client applications generally need to be
-deployed and therefore need require Apache and PostgreSQL to be installed and
-configured correctly too. In addition to the *python* package we therefore also
-create an *application* package for the extension which is named ``ckan-``
-followed by the last part of the extension name. So for ``ckanext-dgu`` two
-packages are created named ``python-ckanext-dgu`` and ``ckan-dgu``. This naming
-may sound slightly inconsistent but it allows a user who wishes to install a
-DGU CKAN instance to just type the command below:
-
-::
-
- sudo apt-get install ckan-dgu
-
-Usually the ``ckan`` package will be a dependency of the your client
-application CKAN extension. When the ``ckan`` package is installed it installs
-``python-ckan`` as a dependency as well as a series of scripts in ``/usr/bin``
-such as:
-
-``ckan-create-instance``
- create a new CKAN instance
-
-``ckan-maintenance-mode``
- put a CKAN intance into or out of maintenence mode (prevent POSTs from
- the web user interface)
-
-In the simple cases, these scripts can then be used in your client application
-CKAN extension's ``postinst`` script to set up the custom instance. In more
-complex cases you may write a ``postinst`` script from scratch. The
-``postinst`` script then forms part of the package and is run by the apt system
-as part of the package installation or upgrade process to configure your CKAN
-instance.
-
-
-
-
-
-
-
-
-
-
-
-
-Before we look at how to actually create an apt repository for your packages
-and how to publish your packages to it, let's understand what a user of your
-package will do to install it.
-
-Understanding How a User Installs from an apt repository
-========================================================
-
-A user will follow the following process:
-
-First create the file ``/etc/apt/sources.list.d/okfn.list`` using this command, replacing ``ubuntu_ckan_dev`` with the correct repo you want to use:
-
-::
-
- echo "deb http://apt.okfn.org/ubuntu_ckan_dev lucid universe" | sudo tee /etc/apt/sources.list.d/okfn.list
-
-Then add the package key to say you trust packages from this repository:
-
-::
-
- sudo apt-get install wget
- wget -qO- http://apt.okfn.org/packages.okfn.key | sudo apt-key add -
- sudo apt-get update
-
-Now you can not install a CKAN extension application, just like any other Debian package:
-
-::
-
- sudo apt-get install ckan-std
-
-At this point you should have a running instance. You may need to copy across
-an existing database if you need your instance pre-populated with data.
-
-
-Setting up a CKAN Apt Repository
-================================
-
-Now you've seen what a user expects to be able to do, let's set up the
-infrastructure to make to make it happen.
-
-
-From Scratch
-------------
-
-Our set up is based on `Joseph Ruscio's set up
-<http://joseph.ruscio.org/blog/2010/08/19/setting-up-an-apt-repository/>`_ and
-will allow us to support multiple operating systems if we want to as well as
-multiple architectures. At the moment we only support Ubuntu Lucid amd64.
-
-To help with repository management we use the ``reprepro`` tool. Despite the fact that the repositories could be set up for different OSs and versions (eg ``lenny``, ``lucid`` etc) we need to make sure that the package names are still unique. This means that we always add the distribution to the version number when we package.
-
-
-The most important detail that AFAIK isn’t covered in any of the tutorials had to do with package naming conventions. The naive assumption (at least on my part) is that you’ll have a different build of your package for each distro/arch combination, and import them into your repository as such. In other words reprepro should track the distro/arch of each import. In actuality, each build’s <PACKAGE>_<VERSION>_<ARCH> must be unique, even though you specify the distro during the includedeb operation.
-
-
-
-
-The Easy Way
-------------
-
-Log into the existing CKAN apt repository server and copy an existing directory
-that already contains the packages you need. For example, to create a
-repository for a new ``ckanext-dgu`` instance you might do:
-
-::
-
- cd /var/packages/
- cp -pr lucid dgu-new
-
-At this point you have a brand new repo, you can add new packages to it like this:
-
-::
-
- cd dgu-new
- sudo reprepro includedeb lucid ~/*.deb
-
-You can remove them like this from the same directory:
-
-::
-
- sudo reprepro remove lucid python-ckan
-
-Any time a change is made you will need to enter the passphrase for the key.
-
-
-Automatic Packaging
-===================
-
-The BuildKit script will build and place Debian packages in your ``missing``
-directory. Make sure there is nothing in there that shouldn't be overwritten by
-this script.
-
-
-Adding a Package to a Repository
-================================
-
-
-Packaging CKAN Itself
-=====================
-
-
-
-
-Why We use ``pip`` rather than ``install_requires``
-===================================================
-
-
-Packaging CKAN and its dependencies for a production install
-============================================================
-
-Installing a Packaged CKAN-based Site
-=====================================
-
-Testing Your Packaging in a VM
-==============================
-
-The Release Process
-===================
-
-
-
-
-
-Creating the CKAN Command
-=========================
-
-Create a directory named ``ckan``. Then within it create a ``DEBIAN`` directory with three files:
-
-``control``:
-
- ::
-
- Package: ckan
- Version: 1.3.2~09
- Architecture: amd64
- Maintainer: James Gardner <james.gardner at okfn.org>
- Installed-Size: 0
- Depends: python-ckan
- Recommends: postgresql, curl
- Section: main/web
- Priority: extra
- Homepage: http://ckan.org
- Description: ckan
- The Data Hub
-
-``postinst``:
-
- ::
-
- #!/bin/sh
- set -e
- # Any commands that happen after install or upgrade go here
-
-``postrm``
-
- ::
-
- #!/bin/sh
- set -e
- # Any commands that happen after removal or before upgrade go here
-
-Then in the ``ckan`` directory you add any files you want copied. In this case
-we want a ``/usr/bin/ckan-create-instance`` script so we create the ``usr``
-directory in the ``ckan`` directory at the same level as the ``DEBIAN``
-directory, then create the ``bin`` directory within it and add the script in
-there.
-
-Finally we want to package up the ``.deb`` file. From within the ``ckan``
-directory run this:
-
-::
-
- dpkg-deb -b . ..
-
-This will create the ``../ckan_1.3.2~09_amd64.deb`` package ready for you to
-upload to the repo.
-
-The ``ckan`` package is already created so in reality you will usually be
-packaging ``ckan-<instance>``. If you make sure your package depends on
-``ckan`` and ``python-ckanext-<instance>`` you can then call the ``ckan``
-package's ``ckan-create-instance`` command in your ``ckan-<instance>``'s
-``postinst`` command to set up Apache and PostgreSQL for the instance
-automatically.
-
-
-Setting up the Repositories
-===========================
-
-Build individual dependencies like this:
-
-::
-
- python -m buildkit.deb . ckanext-importlib 0.1~02 http://ckan.org python-ckan
- python -m buildkit.deb . owslib 0.3.2beta~03 http://ckan.org python-lxml
-
- python -m buildkit.deb . ckanext-inspire 0.1~03 http://ckan.org python-ckan
- python -m buildkit.deb . ckanext-spatial 0.1~04 http://ckan.org python-ckan
- python -m buildkit.deb . ckanext-harvest 0.1~15 http://ckan.org python-ckan python-ckanext-spatial python-carrot
- python -m buildkit.deb . ckanext-csw 0.3~10 http://ckan.org python-ckanext-harvest python-owslib python-ckan
- python -m buildkit.deb . ckanext-dgu 0.2~11 http://ckan.org python-ckan python-ckanext-importlib python-ckanext-dgu python-ckanext-csw python-ckan python-ckanext-spatial python-ckanext-inspire
- python -m buildkit.deb . ckanext-qa 0.1~19 http://ckan.org python-ckan
- python -m buildkit.deb . ckan 1.4~01 http://ckan.org python-routes python-vdm python-pylons python-genshi python-sqlalchemy python-repoze.who python-repoze.who-plugins python-pyutilib.component.core python-migrate python-formalchemy python-sphinx python-markupsafe python-setuptools python-psycopg2 python-licenses python-ckan-deps
-
-There's a dependency on postfix. Choose internet site and the default hostname unless you know better.
-
-Once you have packages you'll want to put them in a repo. You can do that as described here:
-
-* http://joseph.ruscio.org/blog/2010/08/19/setting-up-an-apt-repository/
-
-Then add them like this:
-
-::
-
- cd /var/packages/lucid/
- sudo reprepro includedeb lucid ~/*.deb
-
-You can remove them like this from the same directory:
-
-::
-
- sudo reprepro remove lucid python-ckan
-
-Automatic Packaging
-===================
-
-The BuildKit script will build and place Debian packages in your ``missing``
-directory. Make sure there is nothing in there that shouldn't be overwritten by
-this script.
-
-To package everything automatically, run it like this:
-
-::
-
- cd missing
- bin/python -m buildkit.update_all .
-
-For each package you'll be loaded into ``vim`` to edit the changelog. Save and
-quit when you are done. Names, version numbers and dependencies are
-automatically generated.
-
-You should find all your packages nicely created now.
-
-
-Next Steps
-==========
-
-* Delayed updates
-
-
-Proposed Changes to CKAN
-========================
-
-* Change the config file to support file based logging by default
-* Move who.ini into the config
-* Add a ckan/wsgi.py for standard DGU deployment
-* Modify __init__.py to change
-
-
-* No __init__.py in test directory
-
-
-
-ubuntu at ip-10-226-226-132:/var/packages/dgu-uat$ sudo reprepro includedeb lucid /home/ubuntu/release/2011-04-18_01/*.deb
-/home/ubuntu/release/2011-04-18_01/ckan_1.3.2~10_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan' '1.3.2~10' in 'lucid|universe|amd64', as it has already '1.3.4~03'.
-/home/ubuntu/release/2011-04-18_01/ckan_1.3.2~11_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan' '1.3.2~11' in 'lucid|universe|amd64', as it has already '1.3.4~03'.
-/home/ubuntu/release/2011-04-18_01/ckan_1.3.4~01_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan' '1.3.4~01' in 'lucid|universe|amd64', as it has already '1.3.4~03'.
-/home/ubuntu/release/2011-04-18_01/ckan_1.3.4~02_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan' '1.3.4~02' in 'lucid|universe|amd64', as it has already '1.3.4~03'.
-/home/ubuntu/release/2011-04-18_01/ckan_1.3.4~03_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan' '1.3.4~03' in 'lucid|universe|amd64', as it has already '1.3.4~03'.
-/home/ubuntu/release/2011-04-18_01/ckan-dgu_0.2~06_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan-dgu' '0.2~06' in 'lucid|universe|amd64', as it has already '0.2~15'.
-/home/ubuntu/release/2011-04-18_01/ckan-dgu_0.2~07_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan-dgu' '0.2~07' in 'lucid|universe|amd64', as it has already '0.2~15'.
-/home/ubuntu/release/2011-04-18_01/ckan-dgu_0.2~08_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan-dgu' '0.2~08' in 'lucid|universe|amd64', as it has already '0.2~15'.
-/home/ubuntu/release/2011-04-18_01/ckan-dgu_0.2~09_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan-dgu' '0.2~09' in 'lucid|universe|amd64', as it has already '0.2~15'.
-/home/ubuntu/release/2011-04-18_01/ckan-dgu_0.2~11_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan-dgu' '0.2~11' in 'lucid|universe|amd64', as it has already '0.2~15'.
-/home/ubuntu/release/2011-04-18_01/ckan-dgu_0.2~12_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan-dgu' '0.2~12' in 'lucid|universe|amd64', as it has already '0.2~15'.
-/home/ubuntu/release/2011-04-18_01/ckan-dgu_0.2~13_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan-dgu' '0.2~13' in 'lucid|universe|amd64', as it has already '0.2~15'.
-/home/ubuntu/release/2011-04-18_01/ckan-dgu_0.2~14_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan-dgu' '0.2~14' in 'lucid|universe|amd64', as it has already '0.2~15'.
-/home/ubuntu/release/2011-04-18_01/ckan-dgu_0.2~15_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan-dgu' '0.2~15' in 'lucid|universe|amd64', as it has already '0.2~15'.
-/home/ubuntu/release/2011-04-18_01/python-ckan_1.3.4~01-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckan' '1.3.4~01-1' in 'lucid|universe|amd64', as it has already '1.3.4~02-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckan_1.3.4~02-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckan' '1.3.4~02-1' in 'lucid|universe|amd64', as it has already '1.3.4~02-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-csw_0.3~10-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-csw' '0.3~10-1' in 'lucid|universe|amd64', as it has already '0.3~10-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-dgu_0.2~08-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-dgu' '0.2~08-1' in 'lucid|universe|amd64', as it has already '0.2~11-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-dgu_0.2~09-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-dgu' '0.2~09-1' in 'lucid|universe|amd64', as it has already '0.2~11-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-dgu_0.2~10-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-dgu' '0.2~10-1' in 'lucid|universe|amd64', as it has already '0.2~11-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-dgu_0.2~11-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-dgu' '0.2~11-1' in 'lucid|universe|amd64', as it has already '0.2~11-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-harvest_0.1~13-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-harvest' '0.1~13-1' in 'lucid|universe|amd64', as it has already '0.1~15-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-harvest_0.1~14-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-harvest' '0.1~14-1' in 'lucid|universe|amd64', as it has already '0.1~15-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-harvest_0.1~15-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-harvest' '0.1~15-1' in 'lucid|universe|amd64', as it has already '0.1~15-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-inspire_0.1~01-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-inspire' '0.1~01-1' in 'lucid|universe|amd64', as it has already '0.1~03-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-inspire_0.1~02-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-inspire' '0.1~02-1' in 'lucid|universe|amd64', as it has already '0.1~03-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-inspire_0.1~03-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-inspire' '0.1~03-1' in 'lucid|universe|amd64', as it has already '0.1~03-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-qa_0.1~19-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-qa' '0.1~19-1' in 'lucid|universe|amd64', as it has already '0.1~19-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-spatial_0.1~01-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-spatial' '0.1~01-1' in 'lucid|universe|amd64', as it has already '0.1~04-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-spatial_0.1~03-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-spatial' '0.1~03-1' in 'lucid|universe|amd64', as it has already '0.1~04-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-spatial_0.1~04-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-spatial' '0.1~04-1' in 'lucid|universe|amd64', as it has already '0.1~04-1'.
-/home/ubuntu/release/2011-04-18_01/python-owslib_0.3.2beta~03-1_amd64.deb: component guessed as 'universe'
-ERROR: '/home/ubuntu/release/2011-04-18_01/python-owslib_0.3.2beta~03-1_amd64.deb' cannot be included as 'pool/universe/p/python-owslib/python-owslib_0.3.2beta~03-1_amd64.deb'.
-Already existing files can only be included again, if they are the same, but:
-md5 expected: 3f38d2e844c8d6ec15da6ba51910f3e2, got: ee48427eb11f8152f50f6dc93aeb70d4
-sha1 expected: 87cd7724d8d8f0aaeaa24633abd86e02297771d7, got: 8476b1b0e022892ceb8a35f1848818c31d7441bf
-sha256 expected: 4c9937c78be05dfa5b9dfc85f3a26a51ca4ec0a2d44e8bca530a0c85f12ef400, got: ad3f7458d069a9dd268d144577a7932735643056e45d0a30b7460c38e64057d7
-size expected: 57658, got: 57656
-/home/ubuntu/release/2011-04-18_01/python-owslib_0.3.2beta~04-1_amd64.deb: component guessed as 'universe'
-Exporting indices...
-19A05DDEB16777A2 James Gardner (thejimmyg) <james.gardner at okfn.org> needs a passphrase
-Please enter passphrase:
-Deleting files just added to the pool but not used (to avoid use --keepunusednewfiles next time)
-deleting and forgetting pool/universe/c/ckan/ckan_1.3.2~10_amd64.deb
-deleting and forgetting pool/universe/c/ckan/ckan_1.3.2~11_amd64.deb
-deleting and forgetting pool/universe/c/ckan/ckan_1.3.4~01_amd64.deb
-deleting and forgetting pool/universe/c/ckan/ckan_1.3.4~02_amd64.deb
-deleting and forgetting pool/universe/c/ckan-dgu/ckan-dgu_0.2~06_amd64.deb
-deleting and forgetting pool/universe/c/ckan-dgu/ckan-dgu_0.2~07_amd64.deb
-deleting and forgetting pool/universe/c/ckan-dgu/ckan-dgu_0.2~08_amd64.deb
-deleting and forgetting pool/universe/c/ckan-dgu/ckan-dgu_0.2~09_amd64.deb
-deleting and forgetting pool/universe/c/ckan-dgu/ckan-dgu_0.2~11_amd64.deb
-deleting and forgetting pool/universe/c/ckan-dgu/ckan-dgu_0.2~12_amd64.deb
-deleting and forgetting pool/universe/c/ckan-dgu/ckan-dgu_0.2~13_amd64.deb
-deleting and forgetting pool/universe/c/ckan-dgu/ckan-dgu_0.2~14_amd64.deb
-deleting and forgetting pool/universe/p/python-ckan/python-ckan_1.3.4~01-1_amd64.deb
-deleting and forgetting pool/universe/p/python-ckanext-dgu/python-ckanext-dgu_0.2~08-1_amd64.deb
-deleting and forgetting pool/universe/p/python-ckanext-dgu/python-ckanext-dgu_0.2~09-1_amd64.deb
-deleting and forgetting pool/universe/p/python-ckanext-dgu/python-ckanext-dgu_0.2~10-1_amd64.deb
-deleting and forgetting pool/universe/p/python-ckanext-harvest/python-ckanext-harvest_0.1~13-1_amd64.deb
-deleting and forgetting pool/universe/p/python-ckanext-harvest/python-ckanext-harvest_0.1~14-1_amd64.deb
-deleting and forgetting pool/universe/p/python-ckanext-inspire/python-ckanext-inspire_0.1~01-1_amd64.deb
-deleting and forgetting pool/universe/p/python-ckanext-inspire/python-ckanext-inspire_0.1~02-1_amd64.deb
-deleting and forgetting pool/universe/p/python-ckanext-spatial/python-ckanext-spatial_0.1~01-1_amd64.deb
-deleting and forgetting pool/universe/p/python-ckanext-spatial/python-ckanext-spatial_0.1~03-1_amd64.deb
-Not deleting possibly left over files due to previous errors.
-(To keep the files in the still existing index files from vanishing)
-Use dumpunreferenced/deleteunreferenced to show/delete files without references.
-1 files lost their last reference.
-(dumpunreferenced lists such files, use deleteunreferenced to delete them.)
-There have been errors!
-(reverse-i-search)`delete': cat src/pip-^Clete-this-directory.txt
-ubuntu at ip-10-226-226-132:/var/packages/dgu-uat$ sudo reprepro deleteunreferenced --help
-Error: Too many arguments for command 'deleteunreferenced'!
-Syntax: reprepro deleteunreferenced
-There have been errors!
-ubuntu at ip-10-226-226-132:/var/packages/dgu-uat$ sudo reprepro deleteunreferenced
-deleting and forgetting pool/universe/p/python-owslib/python-owslib_0.3.2beta~03-1_amd64.deb
-ubuntu at ip-10-226-226-132:/var/packages/dgu-uat$
-ubuntu at ip-10-226-226-132:/var/packages/dgu-uat$ sudo reprepro includedeb lucid /home/ubuntu/release/2011-04-18_01/*.deb
-/home/ubuntu/release/2011-04-18_01/ckan_1.3.2~10_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan' '1.3.2~10' in 'lucid|universe|amd64', as it has already '1.3.4~03'.
-/home/ubuntu/release/2011-04-18_01/ckan_1.3.2~11_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan' '1.3.2~11' in 'lucid|universe|amd64', as it has already '1.3.4~03'.
-/home/ubuntu/release/2011-04-18_01/ckan_1.3.4~01_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan' '1.3.4~01' in 'lucid|universe|amd64', as it has already '1.3.4~03'.
-/home/ubuntu/release/2011-04-18_01/ckan_1.3.4~02_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan' '1.3.4~02' in 'lucid|universe|amd64', as it has already '1.3.4~03'.
-/home/ubuntu/release/2011-04-18_01/ckan_1.3.4~03_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan' '1.3.4~03' in 'lucid|universe|amd64', as it has already '1.3.4~03'.
-/home/ubuntu/release/2011-04-18_01/ckan-dgu_0.2~06_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan-dgu' '0.2~06' in 'lucid|universe|amd64', as it has already '0.2~15'.
-/home/ubuntu/release/2011-04-18_01/ckan-dgu_0.2~07_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan-dgu' '0.2~07' in 'lucid|universe|amd64', as it has already '0.2~15'.
-/home/ubuntu/release/2011-04-18_01/ckan-dgu_0.2~08_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan-dgu' '0.2~08' in 'lucid|universe|amd64', as it has already '0.2~15'.
-/home/ubuntu/release/2011-04-18_01/ckan-dgu_0.2~09_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan-dgu' '0.2~09' in 'lucid|universe|amd64', as it has already '0.2~15'.
-/home/ubuntu/release/2011-04-18_01/ckan-dgu_0.2~11_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan-dgu' '0.2~11' in 'lucid|universe|amd64', as it has already '0.2~15'.
-/home/ubuntu/release/2011-04-18_01/ckan-dgu_0.2~12_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan-dgu' '0.2~12' in 'lucid|universe|amd64', as it has already '0.2~15'.
-/home/ubuntu/release/2011-04-18_01/ckan-dgu_0.2~13_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan-dgu' '0.2~13' in 'lucid|universe|amd64', as it has already '0.2~15'.
-/home/ubuntu/release/2011-04-18_01/ckan-dgu_0.2~14_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan-dgu' '0.2~14' in 'lucid|universe|amd64', as it has already '0.2~15'.
-/home/ubuntu/release/2011-04-18_01/ckan-dgu_0.2~15_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan-dgu' '0.2~15' in 'lucid|universe|amd64', as it has already '0.2~15'.
-/home/ubuntu/release/2011-04-18_01/python-ckan_1.3.4~01-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckan' '1.3.4~01-1' in 'lucid|universe|amd64', as it has already '1.3.4~02-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckan_1.3.4~02-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckan' '1.3.4~02-1' in 'lucid|universe|amd64', as it has already '1.3.4~02-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-csw_0.3~10-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-csw' '0.3~10-1' in 'lucid|universe|amd64', as it has already '0.3~10-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-dgu_0.2~08-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-dgu' '0.2~08-1' in 'lucid|universe|amd64', as it has already '0.2~11-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-dgu_0.2~09-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-dgu' '0.2~09-1' in 'lucid|universe|amd64', as it has already '0.2~11-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-dgu_0.2~10-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-dgu' '0.2~10-1' in 'lucid|universe|amd64', as it has already '0.2~11-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-dgu_0.2~11-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-dgu' '0.2~11-1' in 'lucid|universe|amd64', as it has already '0.2~11-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-harvest_0.1~13-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-harvest' '0.1~13-1' in 'lucid|universe|amd64', as it has already '0.1~15-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-harvest_0.1~14-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-harvest' '0.1~14-1' in 'lucid|universe|amd64', as it has already '0.1~15-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-harvest_0.1~15-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-harvest' '0.1~15-1' in 'lucid|universe|amd64', as it has already '0.1~15-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-inspire_0.1~01-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-inspire' '0.1~01-1' in 'lucid|universe|amd64', as it has already '0.1~03-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-inspire_0.1~02-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-inspire' '0.1~02-1' in 'lucid|universe|amd64', as it has already '0.1~03-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-inspire_0.1~03-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-inspire' '0.1~03-1' in 'lucid|universe|amd64', as it has already '0.1~03-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-qa_0.1~19-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-qa' '0.1~19-1' in 'lucid|universe|amd64', as it has already '0.1~19-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-spatial_0.1~01-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-spatial' '0.1~01-1' in 'lucid|universe|amd64', as it has already '0.1~04-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-spatial_0.1~03-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-spatial' '0.1~03-1' in 'lucid|universe|amd64', as it has already '0.1~04-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-spatial_0.1~04-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-spatial' '0.1~04-1' in 'lucid|universe|amd64', as it has already '0.1~04-1'.
-/home/ubuntu/release/2011-04-18_01/python-owslib_0.3.2beta~03-1_amd64.deb: component guessed as 'universe'
-ERROR: '/home/ubuntu/release/2011-04-18_01/python-owslib_0.3.2beta~03-1_amd64.deb' cannot be included as 'pool/universe/p/python-owslib/python-owslib_0.3.2beta~03-1_amd64.deb'.
-Already existing files can only be included again, if they are the same, but:
-md5 expected: 3f38d2e844c8d6ec15da6ba51910f3e2, got: ee48427eb11f8152f50f6dc93aeb70d4
-sha1 expected: 87cd7724d8d8f0aaeaa24633abd86e02297771d7, got: 8476b1b0e022892ceb8a35f1848818c31d7441bf
-sha256 expected: 4c9937c78be05dfa5b9dfc85f3a26a51ca4ec0a2d44e8bca530a0c85f12ef400, got: ad3f7458d069a9dd268d144577a7932735643056e45d0a30b7460c38e64057d7
-size expected: 57658, got: 57656
-/home/ubuntu/release/2011-04-18_01/python-owslib_0.3.2beta~04-1_amd64.deb: component guessed as 'universe'
-Exporting indices...
-19A05DDEB16777A2 James Gardner (thejimmyg) <james.gardner at okfn.org> needs a passphrase
-Please enter passphrase:
-Deleting files just added to the pool but not used (to avoid use --keepunusednewfiles next time)
-deleting and forgetting pool/universe/c/ckan/ckan_1.3.2~10_amd64.deb
-deleting and forgetting pool/universe/c/ckan/ckan_1.3.2~11_amd64.deb
-deleting and forgetting pool/universe/c/ckan/ckan_1.3.4~01_amd64.deb
-deleting and forgetting pool/universe/c/ckan/ckan_1.3.4~02_amd64.deb
-deleting and forgetting pool/universe/c/ckan-dgu/ckan-dgu_0.2~06_amd64.deb
-deleting and forgetting pool/universe/c/ckan-dgu/ckan-dgu_0.2~07_amd64.deb
-deleting and forgetting pool/universe/c/ckan-dgu/ckan-dgu_0.2~08_amd64.deb
-deleting and forgetting pool/universe/c/ckan-dgu/ckan-dgu_0.2~09_amd64.deb
-deleting and forgetting pool/universe/c/ckan-dgu/ckan-dgu_0.2~11_amd64.deb
-deleting and forgetting pool/universe/c/ckan-dgu/ckan-dgu_0.2~12_amd64.deb
-deleting and forgetting pool/universe/c/ckan-dgu/ckan-dgu_0.2~13_amd64.deb
-deleting and forgetting pool/universe/c/ckan-dgu/ckan-dgu_0.2~14_amd64.deb
-deleting and forgetting pool/universe/p/python-ckan/python-ckan_1.3.4~01-1_amd64.deb
-deleting and forgetting pool/universe/p/python-ckanext-dgu/python-ckanext-dgu_0.2~08-1_amd64.deb
-deleting and forgetting pool/universe/p/python-ckanext-dgu/python-ckanext-dgu_0.2~09-1_amd64.deb
-deleting and forgetting pool/universe/p/python-ckanext-dgu/python-ckanext-dgu_0.2~10-1_amd64.deb
-deleting and forgetting pool/universe/p/python-ckanext-harvest/python-ckanext-harvest_0.1~13-1_amd64.deb
-deleting and forgetting pool/universe/p/python-ckanext-harvest/python-ckanext-harvest_0.1~14-1_amd64.deb
-deleting and forgetting pool/universe/p/python-ckanext-inspire/python-ckanext-inspire_0.1~01-1_amd64.deb
-deleting and forgetting pool/universe/p/python-ckanext-inspire/python-ckanext-inspire_0.1~02-1_amd64.deb
-deleting and forgetting pool/universe/p/python-ckanext-spatial/python-ckanext-spatial_0.1~01-1_amd64.deb
-deleting and forgetting pool/universe/p/python-ckanext-spatial/python-ckanext-spatial_0.1~03-1_amd64.deb
-Not deleting possibly left over files due to previous errors.
-(To keep the files in the still existing index files from vanishing)
-Use dumpunreferenced/deleteunreferenced to show/delete files without references.
-1 files lost their last reference.
-(dumpunreferenced lists such files, use deleteunreferenced to delete them.)
-There have been errors!
-(reverse-i-search)`delete': cat src/pip-^Clete-this-directory.txt
-ubuntu at ip-10-226-226-132:/var/packages/dgu-uat$ sudo reprepro deleteunreferenced --help
-Error: Too many arguments for command 'deleteunreferenced'!
-Syntax: reprepro deleteunreferenced
-There have been errors!
-ubuntu at ip-10-226-226-132:/var/packages/dgu-uat$ sudo reprepro deleteunreferenced
-deleting and forgetting pool/universe/p/python-owslib/python-owslib_0.3.2beta~03-1_amd64.deb
-ubuntu at ip-10-226-226-132:/var/packages/dgu-uat$
-
--- a/doc/deployment.rst Thu Jul 28 16:45:02 2011 +0100
+++ /dev/null Thu Jan 01 00:00:00 1970 +0000
@@ -1,280 +0,0 @@
-Production Deployment
-=====================
-
-Here's an example for deploying CKAN to http://demo.ckan.net/ via Apache.
-
-1. Ideally setup the server with Ubuntu.
-
-
-2. Ensure these packages are installed:
- (e.g. sudo apt-get install <package-name>)
-
- ===================== ============================================
- Package Notes
- ===================== ============================================
- mercurial Source control
- python-dev Python interpreter v2.5 - v2.7 and dev headers
- apache2 Web server
- libapache2-mod-python Apache module for python
- libapache2-mod-wsgi Apache module for WSGI
- postgresql PostgreSQL database
- libpq-dev PostgreSQL library
- python-psycopg2 PostgreSQL python module
- python-setuptools Python package management
- python-libxml2 Python XML library
- python-libxslt1 Python XSLT library
- libxml2-dev XML library development files
- libxslt1-dev XSLT library development files
- git-core Git source control (for getting MarkupSafe src)
- subversion Subversion source control (for pyutilib)
- ===================== ============================================
-
- Now use easy_install (which comes with python-setuptools) to install
- these python packages:
- (e.g. sudo easy_install <package-name>)
-
- ===================== ============================================
- Python Package Notes
- ===================== ============================================
- virtualenv Python virtual environment sandboxing
- pip Python installer
- ===================== ============================================
-
- Check that you received:
-
- * virtualenv v1.3 or later
- * pip v0.4 or later
-
-
-NB: Instead of using these manual instructions, steps 3 to 10 can be achieved
-automatically on a remote server by running the fabric deploy script on
-your local machine. You need fabric and python-dev modules installed locally.
-If you don't have the ckan repo checked out locally then download the
-fabfile.py using::
-
- $ wget https://bitbucket.org/okfn/ckan/raw/default/fabfile.py
-
-Now you can then do the deployment with something like::
-
- $ fab config_0:demo.ckan.net,hosts_str=someserver.net,db_pass=my_password deploy
-
-
-3. Setup a PostgreSQL database
-
- List existing databases::
-
- $ sudo -u postgres psql -l
-
- It is advisable to ensure that the encoding of databases is 'UTF8', or
- internationalisation may be a problem. Since changing the encoding of Postgres
- may mean deleting existing databases, it is suggested that this is fixed before
- continuing with the CKAN install.
-
- Create a database user if one doesn't already exist::
-
- $ sudo -u postgres createuser -S -D -R -P <user>
-
- Replace <user> with the unix username whose home directory has the ckan install.
- It should prompt you for a new password for the CKAN data in the database.
-
- Now create the database::
-
- $ sudo -u postgres createdb -O <user> ckandemo
-
-
-4. Create a python virtual environment
-
- In a general user's home directory::
-
- $ mkdir -p ~/var/srvc/demo.ckan.net
- $ cd ~/var/srvc/demo.ckan.net
- $ virtualenv pyenv
- $ . pyenv/bin/activate
-
-
-5. Create the Pylons WSGI script
-
- Create a file ~/var/srvc/demo.ckan.net/pyenv/bin/demo.ckan.net.py as follows (editing the first couple of variables as necessary)::
-
- import os
- instance_dir = '/home/USER/var/srvc/demo.ckan.net'
- config_file = 'demo.ckan.net.ini'
- pyenv_bin_dir = os.path.join(instance_dir, 'pyenv', 'bin')
- activate_this = os.path.join(pyenv_bin_dir, 'activate_this.py')
- execfile(activate_this, dict(__file__=activate_this))
- from paste.deploy import loadapp
- config_filepath = os.path.join(instance_dir, config_file)
- from paste.script.util.logging_config import fileConfig
- fileConfig(config_filepath)
- application = loadapp('config:%s' % config_filepath)
-
-
-6. Install code and dependent packages into the environment
-
- Decide which release of CKAN you want to install. The CHANGELOG.txt has details on the releases. You'll need the exact tag name, and these are listed on the bitbucket page: https://bitbucket.org/okfn/ckan/src and hover over tags to see the options, e.g. ``ckan-1.4``. ::
-
- $ wget https://bitbucket.org/okfn/ckan/raw/ckan-1.4/pip-requirements.txt
-
- Or for the bleeding edge use::
-
- $ wget https://bitbucket.org/okfn/ckan/raw/default/pip-requirements.txt
-
- And now install::
-
- $ pip -E pyenv install -r pip-requirements.txt
-
- If everything goes correctly then you'll finally see: ``Successfully installed``.
-
-
-7. Create CKAN config file
-
- ::
-
- $ paster make-config ckan demo.ckan.net.ini
-
-
-8. Configure CKAN
-
- Edit 'demo.ckan.net.ini' and change the default values as follows:
-
- 8.1. sqlalchemy.url
-
- Set the sqlalchemy.url database connection information using values from step 3.
-
- 8.2. licenses_group_url
-
- Set the licenses_group_url to point to a licenses service. Options
- include: ::
-
- http://licenses.opendefinition.org/2.0/ckan_original
- http://licenses.opendefinition.org/2.0/all_alphabetical
-
- For information about creating your own licenses services, please refer to
- the Python package called 'licenses' (http://pypi.python.org/pypi/licenses).
-
- 8.3. loggers
-
- CKAN can make a log file if you change the ``[loggers]`` section to this::
-
- [loggers]
- keys = root, ckan
-
- [handlers]
- keys = file
-
- [formatters]
- keys = generic
-
- [logger_root]
- level = INFO
- handlers = file
-
- [logger_ckan]
- level = DEBUG
- handlers = file
- qualname = ckan
-
- [handler_file]
- class = handlers.RotatingFileHandler
- formatter = generic
- level = NOTSET
- args = ('/var/log/ckan/demo.ckan.log', 'a', 2048, 3)
-
- [formatter_generic]
- format = %(asctime)s %(levelname)-5.5s [%(name)s] %(message)s
-
-
-9. Initialise database
-
- ::
-
- $ . pyenv/bin/activate
- $ paster --plugin ckan db init --config demo.ckan.net.ini
-
-
-10. Set some permissions for Pylons
-
- Whilst still in the ~/var/srvc/demo.ckan.net directory::
-
- $ mkdir data sstore
- $ chmod g+w -R data sstore
- $ sudo chgrp -R www-data data sstore
- $ ln -s pyenv/src/ckan/who.ini ./
-
- Also edit the who.ini configuration file to set a secret for the auth_tkt plugin.
-
-
-11. Setup Apache with Ckan
-
- Create file /etc/apache2/sites-available/demo.ckan.net as follows::
-
- <VirtualHost *:80>
- ServerName demo.ckan.net
- ServerAlias demo.ckan.net
-
- WSGIScriptAlias / /home/USER/var/srvc/demo.ckan.net/pyenv/bin/demo.ckan.net.py
- # pass authorization info on (needed for rest api)
- WSGIPassAuthorization On
-
- ErrorLog /var/log/apache2/demo.ckan.net.error.log
- CustomLog /var/log/apache2/demo.ckan.net.custom.log combined
- </VirtualHost>
-
-
-12. Enable site in Apache
-
- ::
-
- $ sudo a2ensite demo.ckan.net
-
-
-13. Restart Apache
-
- ::
-
- $ sudo /etc/init.d/apache2 restart
-
-
-14. Browse CKAN website at http://demo.ckan.net/ (assuming you have the DNS setup for this server). Should you have problems, take a look at the log files specified in your apache config and ckan oconfig. e.g. ``/var/log/apache2/demo.ckan.net.error.log`` and ``/var/log/ckan/demo.ckan.log``.
-
-
-Upgrade
--------
-
-Ideally production deployments are upgraded with fabric, but here are the manual instructions.
-
-1. Activate the virtual environment for your install::
-
- $ cd ~/var/srvc/demo.ckan.net
- $ . pyenv/bin/activate
-
-2. It's probably wise to backup your database::
-
- $ paster --plugin=ckan db dump demo_ckan_backup.pg_dump --config=demo.ckan.net.ini
-
- If you get a message about the command being 'mothballed' then you have a particularly old ckan! In this case, use pg_dump directly, specifying the database details from your config file.
-
- $ grep -i sqlalchemy.url demo.ckan.net.ini
- sqlalchemy.url = postgres://okfn:testpassword@psql.okfn.org/demo.okfn.org
- $ pg_dump -U okfn -h psql.okfn.org >demo_ckan_backup.pg_dump
-
-3. Get a version of pip-requirements.txt for the new version you want to install (see info on finding a suitable tag name above)::
-
- $ wget https://bitbucket.org/okfn/ckan/raw/ckan-1.4/pip-requirements.txt
-
-4. Update all the modules::
-
- $ pip -E pyenv install -r pip-requirements.txt
-
-5. Upgrade the database::
-
- $ paster --plugin ckan db upgrade --config {config.ini}
-
-6. Restart apache (so modpython has the latest code)::
-
- $ sudo /etc/init.d/apache2 restart
-
-7. You could manually try CKAN in a browser, or better still run the smoke tests found in ckanext/blackbox. To do this, install ckanext and run ckanext from another machine - see ckanext README.txt for instructions: https://bitbucket.org/okfn/ckanext and then run::
-
- $ python blackbox/smoke.py blackbox/ckan.net.profile.json
-
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/doc/extensions.rst Thu Jul 28 17:18:48 2011 +0100
@@ -0,0 +1,66 @@
+==============
+Add Extensions
+==============
+
+This is where it gets interesting! The CKAN software can be customised with 'extensions'. These are a simple way to extend core CKAN functions.
+
+Extensions allow you to customise CKAN for your own requirements, without interfering with the basic CKAN system.
+
+.. warning:: This is an advanced topic. At the moment, you need to have prepared your system to work with extensions, as described in :doc:`prepare-extensions`. We are working to make the most popular extensions more easily available as Debian packages.
+
+Finding Extensions
+------------------
+
+Many CKAN extensions are listed on the CKAN wiki's `List of Extensions <http://wiki.ckan.net/List_of_Extensions>`_. All CKAN extensions can be found at `OKFN's bitbucket page <https://bitbucket.org/okfn/>`_, prefaced with ``ckanext-``.
+
+Some popular extensions include:
+
+* `ckanext-admin <https://bitbucket.org/okfn/ckanext-admin>`_: Admin web interface for CKAN.
+* `ckanext-apps <https://bitbucket.org/okfn/ckanext-apps>`_: Apps and ideas catalogue extension for CKAN.
+* `ckanext-deliverance <https://bitbucket.org/okfn/ckanext-deliverance>`_: Extends CKAN to use the Deliverance HTTP proxy, which can request and render web pages from * an external site (e.g. a CMS like Drupal or Wordpress).
+* `ckanext-disqus <https://bitbucket.org/okfn/ckanext-disqus>`_: Allows users to comment on package pages with Disqus.
+* `ckanext-follower <https://bitbucket.org/okfn/ckanext-follower>`_: Allow users to follow packages.
+* `ckanext-googleanalytics <https://bitbucket.org/okfn/ckanext-googleanalytics>`_: Integrates Google Analytics data into CKAN. Gives download stats on package pages, list * of most popular packages, etc.
+* `ckanext-qa <https://bitbucket.org/okfn/ckanext-qa>`_: Provides link checker, 5 stars of openness and other Quality Assurance features.
+* `ckanext-rdf <https://bitbucket.org/okfn/ckanext-rdf>`_: Consolidated handling of RDF export and import for CKAN.
+* `ckanext-stats <https://bitbucket.org/okfn/ckanext-stats>`_: Statistics (and visuals) about the datasets in a CKAN instance.
+* `ckanext-wordpresser <https://bitbucket.org/okfn/ckanext-wordpresser>`_: CKAN plugin / WSGI middleware for combining CKAN with a Wordpress site.
+
+Installing an Extension
+-----------------------
+
+You can install an extension on a CKAN instance as follows.
+
+1. First, ensure you are working within your virtualenv (see :doc:`prepare-extensions` if you are not sure what this means)::
+
+ . /home/ubuntu/pyenv/bin/activate
+
+2. Install the extension package code using ``pip``.
+
+ For example, to install the Disqus extension, which allows users to comment on datasets::
+
+ pip install -E ~/var/srvc/ckan.net/pyenv hg+http://bitbucket.org/okfn/ckanext-disqus
+
+ The ``-E`` parameter is for your CKAN Python environment (e.g. ``~/var/srvc/ckan.net/pyenv``).
+
+ Prefix the source URL with the repo type (``hg+`` for Mercurial, ``git+`` for Git).
+
+ The dependency you've installed will appear in the ``src/`` directory under your Python environment.
+
+3. Add the names of any plugin implementations the extension uses to the CKAN
+config file. You can find these in the plugin's ``setup.py`` file under ``[ckan.plugins]``.
+
+ The config plugins variable is in the '[app:main]' section under 'ckan.plugins'. e.g.::
+
+ [app:main]
+ ckan.plugins = disqus
+
+ If your extension implements multiple different plugin interfaces, separate them with spaces::
+
+ ckan.plugins = disqus amqp myplugin
+
+4. If necessary, restart WSGI, which usually means restarting Apache::
+
+ sudo /etc/init.d/apache2 restart
+
+Your extension should now be installed.
--- a/doc/feeds.rst Thu Jul 28 16:45:02 2011 +0100
+++ /dev/null Thu Jan 01 00:00:00 1970 +0000
@@ -1,111 +0,0 @@
-==========
-CKAN Feeds
-==========
-
-Introduction
-============
-
-Access to the CKAN metadata is provided in the RESTful API, but to help users in touch with changes to the data, CKAN also provides feeds. The main feed provides a list of recent changes to all packages, and there are also individual feeds for each package.
-
-Requests
-========
-
-The main feed is at::
-
-/revision/list?format=atom&days=1
-
-For example, to see all the changes made to packages on ckan.net in the past day, browse address: `<http://ckan.net/revision/list?format=atom&days=1>`_
-
-In addition, each package has a feed at::
-
-/package/history/PACKAGENAME?format=atom&days=7
-
-You might want to keep track of developments of the Ordnance Survey open data seen here `<http://ckan.net/package/ordnance_survey>`_, for example. Here is the corresponding URL you'd add to your RSS/Atom Reader: `<http://ckan.net/package/history/ordnance_survey?format=atom&days=7>`_
-
-Format
-======
-
-The feeds are in Atom format.
-
-Each 'entry' in the feed corresponds to a Revision, which is a set of changes.
-
-The Package feed is a subset of the Main Feed, with only the Revisions that reflect a change in that particular package.
-
-The details given for each Revision are:
-
-+-----------+-------------------------------------+
-| Field | Description |
-+===========+=====================================+
-| updated / | Date & time of the change, given |
-| published | in ISO format. |
-+-----------+-------------------------------------+
-| author | 'name' is login name or IP address. |
-+-----------+-------------------------------------+
-| link | Link to Revision in the web |
-| | interface. |
-+-----------+-------------------------------------+
-| summary | Log message for the revision. Also, |
-| | for the main feed, details of |
-| | packages changes, created and |
-| | deleted. See below. |
-+-----------+-------------------------------------+
-
-Example main feed::
-
- <?xml version="1.0" encoding="utf-8"?>
- <feed xmlns="http://www.w3.org/2005/Atom" xml:lang="None">
- <title>CKAN Package Revision History</title>
- <link href="/revision/list/" rel="alternate"></link>
- <id>/revision/list/</id>
- <updated>2010-04-18T21:17:57Z</updated>
- <entry>
- <title>rdbc6b54d-1fd1-4b13-b250-340a01646909 [pgcanada:]</title>
- <link href="/revision/read/dbc6b54d-1fd1-4b13-b250-340a01646909" rel="alternate"></link>
- <updated>2010-04-18T21:17:57Z</updated>
- <author><name>208.65.246.156</name></author>
- <id>tag:,2010-04-18:/revision/read/dbc6b54d-1fd1-4b13-b250-340a01646909</id>
- <summary type="html">Packages affected: [pgcanada:created].</summary>
- </entry>
- <!-- ...other entries... -->
- </feed>
-
-Example package feed for package 'water_voles'::
-
- <?xml version="1.0" encoding="utf-8"?>
- <feed xmlns="http://www.w3.org/2005/Atom" xml:lang="None">
- <title>CKAN Package Revision History</title>
- <link href="/revision/read/water_voles" rel="alternate"></link>
- <id>/revision/read/water_voles</id>
- <updated>2010-04-19T15:32:28Z</updated>
- <entry>
- <title>Creating test data.</title>
- <link href="/revision/read/2bad6982-2394-4187-b289-2ed77ad25d65" rel="alternate"></link>
- <updated>2010-04-19T15:30:00Z</updated>
- <published>2010-04-19T15:30:00Z</published>
- <author><name>http://someone.somecompany.com</name></author>
- <id>tag:,2010-04-19:/revision/read/2bad6982-2394-4187-b289-2ed77ad25d65</id>
- <summary type="html">Log message: Added a test package.</summary>
- </entry>
- <!-- ...other entries... -->
- </feed>
-
-Main feed summary
-=================
-
-The main field provides a little more detail to the edits made to packages. In each entry summary, there is a list of packages affected, and the change to each is described as 'created', 'updated' or 'deleted'. For example::
-
- <summary type="html>Packages affected: [hospital_performance:created].</summary>
-
-For package updates, some further information is provided if there are changes to the package's resources, or the 'date_updated' extra field. For example::
-
- Packages affected: [water_voles:updated:resources].
-
-or::
-
- Packages affected: [water_voles:updated:date_updated].
-
-(The date_updated field is highlighted for a particular use of CKAN and this feature may be generalised in the future.)
-
-A revision may have changes to several packages, particularly if it was created using the API. In this case, the package list is space separated::
-
- Packages affected: [hospital_performance:created water_voles:updated bus_stops:updated:resources bus_stats:updated:resources:date_updated].</summary>
--- a/doc/form-integration.rst Thu Jul 28 16:45:02 2011 +0100
+++ b/doc/form-integration.rst Thu Jul 28 17:18:48 2011 +0100
@@ -1,30 +1,30 @@
================
-Form integration
+Form Integration
================
-CKAN provides facility for integrating the package editing forms into another front end
+CKAN allows you to integrate its Edit Package and New Package forms forms into an external front-end. To that end, CKAN also provides a simple way to redirect these forms back to the external front-end upon submission.
-Form completion redirect
-========================
+Redirecting CKAN Forms
+======================
-It is simple enough for the external front end to link to CKAN's package creation or edit pages, but once the form is submitted the user needs to be redirected back to the external front end, instead of CKAN's package read page. This is achieved with a parameter to the CKAN URL.
+It is obviously simple enough for an external front-end to link to CKAN's Edit Package and New Package forms, but once the forms are submitted, it would be desirable to redirect the user back to the external front-end, rather than CKAN's package read page.
-The 'return URL' can be specified in two places:
+This is achieved with a parameter to the CKAN URL. The 'return URL' can be specified in two places:
- 1. passed as a URL encoded value with the parameter "return_to" in the link to CKAN's form page.
+ 1. Passed as a URL-encoded value with the parameter ``return_to`` in the link to CKAN's form page.
- 2. specified in the CKAN config key 'package_new_return_url' and 'package_edit_return_url'.
+ 2. Specified in the CKAN config keys ``package_new_return_url`` and ``package_edit_return_url`` (see :ref:`config-package-urls`).
-(If the 'return URL' is given in both ways then the first takes precedence.)
+(If the 'return URL' is supplied in both places, then the first takes precedence.)
-Since the 'return URL' may need to include the package name, which could be set in the form, CKAN replaces a known placeholder "<NAME>" with this value on redirect.
+Since the 'return URL' may need to include the package name, which could be changed by the user, CKAN replaces a known placeholder ``<NAME>`` with this value on redirect.
-Note that the downside of specifying the 'return URL' in the CKAN config is that the CKAN web interface is less usable on its own, since the user is hampered by the redirects to the external interface.
+.. note:: Note that the downside of specifying the 'return URL' in the CKAN config is that the CKAN web interface becomes less usable on its own, since the user is hampered by the redirects to the external interface.
Example
-------
-An external front end displays a package 'ontariolandcoverv100' here::
+An external front-end displays a package 'ontariolandcoverv100' here::
http://datadotgc.ca/dataset/ontariolandcoverv100
@@ -32,27 +32,23 @@
http://ca.ckan.net/package/edit/ontariolandoverv100
-On first thought, the return link is::
-
- http://datadotgc.ca/dataset/ontariolandcoverv100
-
-But when the user edit's this package, the name may change. So the return link needs to be::
+At first, it may seem that the return link should be ``http://datadotgc.ca/dataset/ontariolandcoverv100``. But when the user edits this package, the name may change. So the return link needs to be::
http://datadotgc.ca/dataset/<NAME>
-This is URL encoded to be::
+And this is URL-encoded to become::
http%3A%2F%2Fdatadotgc.ca%2Fdataset%2F%3CNAME%3E
-So the edit link becomes::
+So, in summary, the edit link becomes::
http://ca.ckan.net/package/edit/ontariolandoverv100?return_to=http%3A%2F%2Fdatadotgc.ca%2Fdataset%2F%3CNAME%3E
-During editing the package, the user changes the name to `canadalandcover`, presses 'preview' and finally 'commit'. The user is now redirected back to the external front end at::
+During editing the package, the user changes the package name to `canadalandcover`, presses 'preview' and finally 'commit'. The user is now redirected back to the external front-end at::
http://datadotgc.ca/dataset/canadalandcover
-This same functionality could be achieved by this line in the config (ca.ckan.net.ini)::
+The same functionality could be achieved by this line in the config file (``ca.ckan.net.ini``)::
...
--- a/doc/forms.rst Thu Jul 28 16:45:02 2011 +0100
+++ b/doc/forms.rst Thu Jul 28 17:18:48 2011 +0100
@@ -1,56 +1,55 @@
-Forms Using Templates
-=====================
+=================
+Customizing Forms
+=================
-The forms used to edit Packages and Groups in CKAN can be customised. This makes it easier to help users input data, helping them choose from sensible options or to use different data formats. This document sets out to show how this is achieved, without getting embroilled in the main CKAN code.
+The forms used to edit packages and groups in CKAN can be customized. This lets you tailor them to your needs, helping your users choose from sensible options or use different data formats.
-Note that this section deals with the form used to *edit* packages and groups, not the way they are displayed. Display is done in the CKAN templates.
+This document explains how to customize the package and group forms you offer to your users, without getting embroiled in the core CKAN code.
-Location of related code
-------------------------
-
- * ckan/controllers/package.py
- * ckan/templates/package/new_package_form.html
+.. note:: This section deals with the form used to *edit* packages and groups, not the way they are displayed. For information on customizing the display of forms, see :doc:`theming`.
-Building a package form
+.. warning:: This is an advanced topic. Ensure you are familiar with :doc:`extensions` before attempting to customize forms.
+
+Building a Package Form
-----------------------
-Basics
-^^^^^^
+The Best Way: Extensions
+^^^^^^^^^^^^^^^^^^^^^^^^
-You will firstly need to make a new controller in your extension. This should subclass PackageController like::
+The best way to build a package form is by using a CKAN extension.
+
+You will firstly need to make a new controller in your extension. This should subclass PackageController as follows::
from ckan.controllers.package import PackageController
class PackageNew(PackageController):
package_form = 'custom_package_form.html'
-The package_form variable in the subclass will be used as the new form template.
+The ``package_form`` variable in the subclass will be used as the new form template.
-It is recommended that you copy the package form (named new_package_form.html) and make modifications to it. However, it is possible to start from scratch.
+It is recommended that you copy the package form (``new_package_form.html``) and make modifications to it. However, it is possible to start from scratch.
-In order for you to get this new controller pointed at correctly, your extension should look like the following::
+To point at this new controller correctly, your extension should look like the following::
class CustomForm(SingletonPlugin):
-
implements(IRoutes)
implements(IConfigurer)
-
def before_map(self, map):
map.connect('/package/new', controller='ckanext.extension_name.controllers.PackageNewController:PackageNew', action='new')
map.connect('/package/edit/{id}', controller='ckanext.extension_name.controllers.PackageNewController:PackageNew', action='edit')
return map
-
def after_map(self, map):
- return map
-
+ return map
def update_config(self, config):
configure_template_directory(config, 'templates')
-Replace extension_name with the name of your extension. This also assumes that custom_package_form.html is located in templates subdirectory of your extension i.e ckanext/extension_name/templates/custom_package_form.html.
+Replace ``extension_name`` with the name of your extension.
+
+This also assumes that ``custom_package_form.html`` is located in the ``templates`` subdirectory of your extension i.e ``ckanext/extension_name/templates/custom_package_form.html``.
Advanced Use
^^^^^^^^^^^^
-The PackageController has a more hooks to customize how and what data is displayed. These functions can be overridden subclass of PackageController::
+The PackageController has more hooks to customize the displayed data. These functions can be overridden in a subclass of PackageController::
_setup_template_variables(self, context)
@@ -64,185 +63,4 @@
This defines a navl schema to customize conversion from the database to the form.
-A complicated example of the use of these hooks can be found in extension ckanext-dgu.
-
-
-Forms using FormAlchemy (depreciated)
-=====================================
-
-This is now depreciated in order to get this to work your extention must implement IRoutes and have a before_map like the following::
-
- class Plugin(SingletonPlugin):
- implements(IRoutes)
-
- def before_map(self, map):
- map.connect('/package/new', controller='package_formalchemy', action='new')
- map.connect('/package/edit/{id}', controller='package_formalchemy', action='edit')
- return map
-
-Location of related code
-------------------------
-
-In the CKAN code, the majority of forms code is in ckan/forms.
-
- * ckan/forms/common.py - A common place for fields used by standard forms
- * ckan/forms/builder.py - The FormBuilder class, which provides an easy way to define a form. It creates a FormAlchemy fieldset used in the CKAN controller code.
- * ckan/forms/package.py - This contains the 'standard' form, which is a good example and is useful to derive custom forms from.
- * ckan/forms/package_gov.py - Contains the a government form, which serves as an example of using lots of extra fields for dates, geographical coverage, etc.
-
-
-Building a package form
------------------------
-
-Basics
-^^^^^^
-
-The *PackageFormBuilder* class initialises with the basic package form which we can then configure::
-
- builder = PackageFormBuilder()
-
-All the basic package fields are added to the form automatically - this currently includes: name, title, version, url, author, author_email, maintainer, maintainer_email, notes and license_id. In case this changes in future, consult the fields for table 'Package' in ckan/model/core.py.
-
-To provide editing of other fields beyond the basic ones, you need to use *add_field* and select either an existing *ConfiguredField* in common.py, or define your own. For example, the autocompleting tag entry field is defined as a field in common.py and it is added to the standard form like this::
-
- builder.add_field(common.TagField('tags'))
-
-The basic fields (name, title, etc) and a few more (license, tags, resources) are defined for all packages. Additional information can be stored on each package in the 'extra' fields. Often we want to provide a nicer interface to these 'extra' fields to help keep consistency in format between the packages. For example, in the government form (package_gov.py) we have added a field for the release date. This is stored as a Package 'extra' with key 'date_released' and by using the DateExtraField, in the form the user is asked for a date.::
-
- builder.add_field(common.DateExtraField('date_released'))
-
-You can configure existing fields using the usual `FormAlchemy Field options <http://docs.formalchemy.org/fields.html#fields>`_. For example, here we add a validator to a standard field::
-
- builder.set_field_option('name', 'validate', package_name_validator)
-
-Options are given keyword parameters by passing a dictionary. For example, this is how we set the notes field's size::
-
- builder.set_field_option('notes', 'textarea', {'size':'60x15'})
-
-Fields in package forms are grouped together. You should specify which fields are displayed in which groups and in which order like this::
-
- from sqlalchemy.util import OrderedDict
- builder.set_displayed_fields_in_groups(OrderedDict([
- ('Basic information', ['name', 'title', 'version', 'url']),
- ('Resources', ['resources']),
- ('Detail', ['author', 'author_email'])]))
-
-To complete the form design you need to return the fieldset object. Ensure this is executed once - when your python form file is imported::
-
- my_fieldset = builder.get_fieldset()
-
-
-Field labels
-^^^^^^^^^^^^
-
-The field labels are derived from the model key using a 'prettify' function. The default munge capitalises the first letter and changes underscores to spaces. You can write a more advanced function depending on your needs. Here is the template for a prettify function::
-
- def prettify(field_name):
- return field_name.replace('_', ' ').capitalize())
-
-If you write a new one, you tell the builder about it like this::
-
- builder.set_label_prettifier(prettify)
-
-
-Templates
-^^^^^^^^^
-
-Package forms by default use the Genshi template *ckan/package/form.html*. If you want to use a modified one then specify it for example like this::
-
- builder.set_form_template('package/my_form')
-
-
-Hidden labels
-^^^^^^^^^^^^^
-
-A couple of common fields (ResourceField and ExtrasField currently) are designed to go in their own field group (see below) and without the usual field label. To hide the label, add these fields like this::
-
- builder.add_field(common.ResourcesField('resources', hidden_label=True))
-
-Instead of starting with just the basic fields, many people will want to edit the standard form, which already contains the resources, extra fields and customise that further. To achieve that you import the builder object like this::
-
- import ckan.forms.package as package
- builder = package.build_package_form()
-
-
-Defining custom fields
-----------------------
-
-If you want to define a completely new field then here is a useful template::
-
- class MyField(common.ConfiguredField):
- def get_configured(self):
- return self.MyField(self.name).with_renderer(self.MyRenderer).validate(self.my_validator)
-
- class MyField(formalchemy.Field):
- def sync(self):
- # edit self.model with using value self._deserialize()
-
- class MyRenderer(formalchemy.fields.FieldRenderer):
- def render(self, **kwargs):
- # return html of field editor based on self._value
-
- def _serialized_value(self):
- # take self._params and serialize them ready for rendering
- # or self.deserialize() into python value that can be saved
- # on a sync.
-
- def my_validator(self, val, field):
- if not ...:
- raise formalchemy.ValidationError('Invalid value')
-
-More examples are in common.py and further information can be obtained from the `FormAlchemy documentation <http://docs.formalchemy.org/>`_.
-
-
-Using a custom form
--------------------
-
-To register your new form with CKAN you need to do three things.
-
-1. In your form you need a function that returns your new form's field set.
-
- For example you might add below your form code::
-
- my_fieldset = builder.get_fieldset()
-
- def get_fieldset(is_admin=False):
- return my_fieldset
-
- (The *is_admin* parameter can be considered if you wish to return a different fieldset for administrator users.)
-
-2. You need to provide an 'entry point' into your code package so that CKAN can access your new form.
-
- It is anticipated that your form code will live in a python package outside the CKAN main code package, managed by setuptools. The entry points are listed in the python package's setup.py and you just need to add a category [ckan.forms] and list the function that returns::
-
- from setuptools import setup, find_packages
- setup(
- ...
-
- entry_points="""
- [ckan.forms]
- my_form = my_module.forms.my_form:get_fieldset
- """,
- )
-
- For this change to have an effect, you need to recreate the egg information, so run::
-
- $ python setup.py egg_info
-
-3. Change an option in your CKAN pylons config file to switch to using the new form.
-
- For example, your pylons config file will probably be 'development.ini' during development, when you 'paster serve' your CKAN app for testing.
-
- You need to change the 'package_form' setting in the '[app:main]' section to the name defined int he entry point. For example::
-
- [app:main]
- ...
- package_form = my_form
- group_form = my_group_form
- package_group_form = my_package_group_form
-
- For this to have an effect you may need to restart the pylons (either by restarting the 'serve' command or the Apache host). Now go and edit a package and try out the new form!
-
- You can also override the config file setting with a URL parameter in your browser. For example you might browse:
-
- http://eco.ckan.net/package/edit/water-voles?package_form=my_form
+A complex example of the use of these hooks can be found in the ``ckanext-dgu`` extension.
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/doc/i18n.rst Thu Jul 28 17:18:48 2011 +0100
@@ -0,0 +1,143 @@
+=====================
+Internationalize CKAN
+=====================
+
+CKAN is used in many countries, and adding a new language is a simple process.
+
+Supported Languages
+===================
+
+CKAN already supports numerous languages. To check whether your language is supported, look in the source at ``ckan/i18n`` for translation files. Languages are named using two-letter ISO language codes (e.g. ``es``, ``de``).
+
+If your language is present, you can switch language simply by setting the ``lang`` option in your CKAN config file, as described in :ref:`config-i18n`. For example, to switch to German::
+
+ lang=de
+
+If your language is not supported yet, the remainder of this section section provides instructions on how to prepare a translation file and add it to CKAN.
+
+Adding a New Language
+=====================
+
+If you want to add an entirely new language to CKAN, you have two options.
+
+* :ref:`i18n-manual`. Creating translation files manually.
+* :ref:`i18n-transifex`. Creating translation files using Transifex, the open source translation software.
+
+
+.. _i18n-manual:
+
+Manual Setup
+------------
+
+If you prefer not to use Transifex, you can create translation files manually.
+
+All the English strings in CKAN are extracted into the ``ckan.pot`` file, which can be found in ``ckan/i18n``.
+
+.. note:: For information, the pot file was created with the ``babel`` command ``python setup.py extract_messages``.
+
+0. Install Babel
+++++++++++++++++
+
+You need Python's ``babel`` library (Debian package ``python-pybabel``). Install it as follows with pip::
+
+ pip -E pyenv install babel
+
+1. Create a 'po' File for Your Language
++++++++++++++++++++++++++++++++++++++++
+
+First, grab the CKAN i18n repository::
+
+ hg clone http://bitbucket.org/bboissin/ckan-i18n/
+
+Then create a translation file for your language (a po file) using the pot file::
+
+ python setup.py init_catalog --locale YOUR_LANGUAGE
+
+Replace ``YOUR_LANGUAGE`` with the two-letter ISO language code (e.g. ``es``, ``de``).
+
+In future, when the pot file is updated, you can update the strings in your po file, while preserving your po edits, by doing::
+
+ python setup.py update_catalog --locale YOUR-LANGUAGE
+
+2. Do the Translation
++++++++++++++++++++++
+
+Edit the po file and translate the strings. For more information on how to do this, see `the Pylons book <http://pylonsbook.com/en/1.1/internationalization-and-localization.html>`_.
+
+We recommend using a translation tool, such as `poedit <http://www.poedit.net/>`_, to check the syntax is correct. There are also extensions for editors such as emacs.
+
+3. Commit the Translation
+++++++++++++++++++++++++++
+
+When the po is complete, commit it to the CKAN i18n repo::
+
+ hg add ckan/i18n/YOUR_LANGUAGE/LC_MESSAGES/ckan.po
+ hg ci -m '[i18n]: New language po added: YOUR_LANGUAGE' ckan/i18n/YOUR_LANGUAGE/LC_MESSAGES/ckan.po
+ hg push
+
+.. note:: You need to be given credentials to do this - to request these, contact the `ckan-discuss <http://lists.okfn.org/mailman/listinfo/ckan-discuss>`_ list.
+
+4. Compile a Translation
+++++++++++++++++++++++++
+
+Once you have created a translation (either with Transifex or manually) you can build the po file into a ``mo`` file, ready for deployment.
+
+With either method of creating the po file, it should be found in the CKAN i18n repository: ``ckan/i18n/YOUR_LANGUAGE/LC_MESSAGES/ckan.po``
+
+In this repo, compile the po file like this::
+
+ python setup.py compile_catalog --locale YOUR_LANGUAGE
+
+As before, replace ``YOUR_LANGUAGE`` with your language short code, e.g. ``es``, ``de``.
+
+This will result in a binary 'mo' file of your translation at ``ckan/i18n/YOUR_LANGUAGE/LC_MESSAGES/ckan.mo``.
+
+5. (optional) Deploy the Translation
+++++++++++++++++++++++++++++++++++++
+
+This section explains how to deploy your translation automatically to your host, if you are using a remote host.
+
+It assumes a standard layout on the server (you may want to check before you upload!) and that you are deploying to ``hu.ckan.net`` for language ``hu``.
+
+Once you have a compiled translation file, for automated deployment to your host do::
+
+ fab config_0:hu.ckan.net upload_i18n:hu
+
+See the ``config_0`` options if more configuration is needed e.g. of host or location.
+
+Alternatively, if you do not want to use fab, you can just scp::
+
+ scp ckan.mo /home/okfn/var/srvc/hu.ckan.net/pyenv/src/ckan/ckan/i18n/hu/LC_MESSAGES/ckan.mo
+
+6. Configure the Language
++++++++++++++++++++++++++
+
+Finally, once the mo file is in place, you can switch between the installed languages using the ``lang`` option in the CKAN config file, as described in :ref:`config-i18n`.
+
+
+.. _i18n-transifex:
+
+Transifex Setup
+---------------
+
+Transifxes, the open translation platform, provides a simple web interface for writing translations and is widely used for CKAN internationalization.
+
+Using Transifex makes it easier to handle collaboration, with an online editor that makes the process more accessible.
+
+Existing CKAN translation projects can be found at: https://www.transifex.net/projects/p/ckan/teams/
+
+Updated translations are automatically pushed to https://bitbucket.org/bboissin/ckan-i18n and these can be compiled and placed on CKAN servers by the server administrators.
+
+Transifex Administration
+++++++++++++++++++++++++
+
+The Transifex workflow is as follows:
+
+* Install transifex command-line utilities
+* ``tx init`` in CKAN to connect to Transifex
+* Run ``python setup.py extract_messages`` on the CKAN source
+* Upload the local .pot file via command-line ``tx push``
+* Get people to complete translations on Transifex
+* Pull locale .po files via ``tx pull``
+* ``python setup.py compile_catalog``
+* Commit and push po and mo files
--- a/doc/importer.rst Thu Jul 28 16:45:02 2011 +0100
+++ /dev/null Thu Jan 01 00:00:00 1970 +0000
@@ -1,98 +0,0 @@
-================
-Package Importer
-================
-
-
-Introduction
-============
-
-This part of the CKAN web interface provides an easy way to import packages
-into CKAN in bulk via a spreadsheet. It fills the gap between entry via a
-simple form (/package/new) and the RESTful API (/api/rest).
-
-** NB: This feature is not currently available. **
-
-
-Details
-=======
-
-Importing a package with the same name as one that exists in the CKAN database results in the new package overwriting the existing one. There is a warning for this.
-
-To perform an import, the user must be logged in. To add a package to a group, the user must have priviledges to edit the particular group.
-
-Format
-======
-
-The details of the packages should be stored in an Excel spreadsheet or CSV file. In Excel format, the package details should be the first (or only) sheet of a workbook.
-
-The importer looks for a header row (which must contain 'name' or 'title') and below that all the rows are the package details. The header row can contain any or all of the field names, but must include 'name' or 'title'. If the 'name' is not specified then a unique name will be generated from the title.
-
-Example
-=======
-
-+-------+-----------+-----------+--------------------------------+------------------------+---+
-| | A | B | C | D | E |
-+=======+===========+===========+================================+========================+===+
-| **1** | Packages | | | | |
-+-------+-----------+-----------+--------------------------------+------------------------+---+
-| **2** | | | | | |
-+-------+-----------+-----------+--------------------------------+------------------------+---+
-| **3** | name | title | resource-0-url | tags | |
-+-------+-----------+-----------+--------------------------------+------------------------+---+
-| **4** | wikipedia | Wikipedia | http://download.wikimedia.org/ | encyclopedia reference | |
-+-------+-----------+-----------+--------------------------------+------------------------+---+
-| **5** | tviv | TV IV | http://tviv.org/Category:Grids | tv encyclopaedia | |
-+-------+-----------+-----------+--------------------------------+------------------------+---+
-| **6** | | | | | |
-+-------+-----------+-----------+--------------------------------+------------------------+---+
-
-Fields
-======
-
-Each package has many fields.
-
-+------------------------+----------------------------------------+------------------------------------+
-| Name | Example value | Notes |
-+========================+========================================+====================================+
-| name | wikipedia-blind | |
-+------------------------+----------------------------------------+------------------------------------+
-| title | Wikipedia for the Blind | |
-+------------------------+----------------------------------------+------------------------------------+
-| notes | Maintained until 2008 | |
-+------------------------+----------------------------------------+------------------------------------+
-| url | http://blind.wikipedia.org/ | |
-+------------------------+----------------------------------------+------------------------------------+
-| resource-0-url | http://blind.wikipedia.org/dump-en.csv | Number resources from 0. |
-+------------------------+----------------------------------------+------------------------------------+
-| resource-0-format | csv | |
-+------------------------+----------------------------------------+------------------------------------+
-| resource-0-description | English version | |
-+------------------------+----------------------------------------+------------------------------------+
-| resource-0-hash | e0d123e5f31 | Hash of the resource |
-+------------------------+----------------------------------------+------------------------------------+
-| resource-1-url | http://blind.wikipedia.org/dump-fr.csv | |
-+------------------------+----------------------------------------+------------------------------------+
-| resource-1-format | csv | |
-+------------------------+----------------------------------------+------------------------------------+
-| resource-1-description | French version | |
-+------------------------+----------------------------------------+------------------------------------+
-| resource-1-hash | 78bfdf5a008 | |
-+------------------------+----------------------------------------+------------------------------------+
-| tags | encyclopedia blind format-csv | Space separated list |
-+------------------------+----------------------------------------+------------------------------------+
-| author | John Doe | |
-+------------------------+----------------------------------------+------------------------------------+
-| author_url | john at doe.com | |
-+------------------------+----------------------------------------+------------------------------------+
-| maintainer | John Doe | |
-+------------------------+----------------------------------------+------------------------------------+
-| maintainer_url | john at doe.com | |
-+------------------------+----------------------------------------+------------------------------------+
-| license | OKD Compliant::UK Click Use PSI | License name |
-+------------------------+----------------------------------------+------------------------------------+
-| [arbitrary] | | Any field name and a string value. |
-+------------------------+----------------------------------------+------------------------------------+
-| groups | blind-picks | Space separated list of group |
-| | | names to add package to. |
-+------------------------+----------------------------------------+------------------------------------+
-
--- a/doc/index.rst Thu Jul 28 16:45:02 2011 +0100
+++ b/doc/index.rst Thu Jul 28 17:18:48 2011 +0100
@@ -1,34 +1,40 @@
-==========================================
-Welcome to CKAN's Developer Documentation!
-==========================================
+=======================================
+Welcome to CKAN's Administration Guide
+=======================================
-This Developer Documentation provides more advanced help for `CKAN <http://ckan.org>`_ development and administration. The latest released version of this documentation is found at: http://packages.python.org/ckan/
+This Administration Guide covers how to set up and manage `CKAN <http://ckan.org>`_ (Comprehensive Knowledge Archive Network) software.
+* The first two sections cover your two options for installing CKAN: package or source install.
+* The rest of the first half of the Guide, up to :doc:`authorization`, cover setup and basic admin.
+* The second half of the Guide, from :doc:`prepare-extensions` onwards, covers advanced tasks, including extensions and forms.
+
+For high-level information on what CKAN is, see the `CKAN website <http://ckan.org>`_.
Contents:
.. toctree::
:maxdepth: 2
- README
- deployment
- configuration
+ install-from-package
+ install-from-source
+ post-installation
+ theming
+ loading_data
paster
- api
+ authorization
+ prepare-extensions
+ extensions
plugins
- feeds
- loader_scripts
- importer
forms
form-integration
- model
- authentication
- authorization
- load_testing
database_dumps
- deb
- vm
+ upgrade
+ i18n
+ configuration
+ api
+ test
buildbot
+ about
Other material
==============
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/doc/install-from-package.rst Thu Jul 28 17:18:48 2011 +0100
@@ -0,0 +1,429 @@
+==============================
+Option 1: Package Installation
+==============================
+
+This section describes how to install CKAN from packages. This is the recommended and by far the easiest way to install CKAN.
+
+Package install requires you to use 64-bit Ubuntu 10.04: either locally, through a virtual machine or Amazon EC2. Your options are as follows:
+
+* Using 64-bit Ubuntu 10.04 directly.
+* :ref:`using-virtualbox`. This is suitable if you want to host your CKAN instance on a machine running any other OS.
+* :ref:`using-amazon`. This is suitable if you want to host your CKAN instance in the cloud, on a ready-made Ubuntu OS.
+
+.. note:: We recommend you use package installation unless you are a core CKAN developer or have no access to Ubuntu 10.04 through any of the methods above, in which case, you should use :doc:`install-from-source`.
+
+For support during installation, please contact `the ckan-dev mailing list <http://lists.okfn.org/mailman/listinfo/ckan-dev>`_.
+
+Prepare your System
+--------------------
+
+CKAN runs on 64-bit Ubuntu 10.04. If you are already using Ubuntu 10.04, you can continue straight to :ref:`run-package-installer`.
+
+However, if you're not, you can either use VirtualBox to set up an Ubuntu VM on Windows, Linux, Macintosh and Solaris. Alternatively, you can use an Amazon EC2 instance.
+
+.. _using-virtualbox:
+
+Option A: Using VirtualBox
+++++++++++++++++++++++++++
+
+This option is suitable if you want to install CKAN on a machine running an OS other than Ubuntu 10.04. `VirtualBox <http://www.virtualbox.org>`_ lets you set up a virtual machine to run Ubuntu 10.04.
+
+Pre-requisites and Downloads
+****************************
+
+First, check your machine meets `the pre-requisites for VirtualBox <http://www.virtualbox.org/wiki/End-user_documentation>`_. These include a fairly recent processor and some spare memory.
+
+Then download the installation files.
+
+* `Download the VirtualBox installer <http://www.virtualbox.org/wiki/Downloads>`_.
+* `Download the Ubuntu image <http://www.ubuntu.com/download/ubuntu/download>`_ - make sure you choose Ubuntu 10.04 and 64-bit.
+
+Install VirtualBox
+******************
+
+.. note::
+
+ This tutorial is for a Mac, but you can find instructions for installing VirtualBox on any OS `in the VirtualBox Manual <http://www.virtualbox.org/manual/ch02.html>`_.
+
+To install, double-click on the VirtualBox installer:
+
+.. image:: images/virtualbox1-package.png
+ :width: 807px
+ :alt: The VirtualBox installer - getting started
+
+Click Continue to begin the installation process. Enter your password when required, and wait for the installation to finish.
+
+Create Your Virtual Machine
+***************************
+
+Go to Applications and open VirtualBox, then click New:
+
+.. image:: images/virtualbox4-newvm.png
+ :width: 807px
+ :alt: The VirtualBox installer - the New Virtual Machine Wizard
+
+Give your VM a name - we'll call ours ``ubuntu_ckan``. Under **OS Type**, choose **Linux** and **Ubuntu 64-bit**.
+
+.. image:: images/virtualbox5-vmtype.png
+ :width: 807px
+ :alt: The VirtualBox installer - choosing your operating system
+
+Leave the memory size as 512MB, and choose **Create new hard disk**. This will open a new wizard:
+
+.. image:: images/virtualbox6-vmloc.png
+ :width: 807px
+ :alt: The VirtualBox installer - creating a new hard disk
+
+You can leave the defaults unchanged here too - click **Continue**, and then **Done**, and **Done** again, to create a new VM.
+
+Next, choose your VM from the left-hand menu, and click **Start**:
+
+.. image:: images/virtualbox7-startvm.png
+ :width: 807px
+ :alt: Starting your new VM
+
+This will open the First Run Wizard:
+
+.. image:: images/virtualbox8-firstrun.png
+ :width: 807px
+ :alt: The VirtualBox First Run Wizard
+
+After clicking **Continue**, you'll see **Select Installation Media**. This is where we need to tell our VM to boot from Ubuntu. Click on the file icon, and find your Ubuntu ``.iso`` file:
+
+.. image:: images/virtualbox9-iso.png
+ :width: 807px
+ :alt: When you get to Select Installation Media, choose your Ubuntu .iso file
+
+Click **Done**, wait for a few seconds, and you will see your Ubuntu VM booting.
+
+Set Up Ubuntu
+*************
+
+During boot, you will be asked if you want to try Ubuntu, or install it. Choose **Install Ubuntu**:
+
+.. image:: images/virtualbox11-ubuntu.png
+ :width: 807px
+ :alt: Booting Ubuntu - choose the Install Ubuntu option
+
+You can then follow the usual Ubuntu installation process.
+
+After Ubuntu is installed, from the main menu, choose **System > Administration > Update Manager**. You'll be asked if you want to install updates - say yes.
+
+When all the updates have been downloaded and installed, you'll be prompted to reboot Ubuntu.
+
+At this point, you can proceed to :ref:`run-package-installer`.
+
+.. _using-amazon:
+
+Option B: Using Amazon EC2
+++++++++++++++++++++++++++
+
+If you prefer to run your CKAN package install in the cloud, you can use an Amazon EC2 instance, which is a fairly cheap and lightweight way to set up a server.
+
+Create an Amazon Account
+************************
+
+If you don't already have an Amazon AWS account you'll need to create one first. You can `create an Amazon AWS account for EC2 here <http://aws.amazon.com/ec2/>`_.
+
+Configure EC2
+*************
+
+Once you have an EC2 account, you'll need to configure settings for your CKAN instance.
+
+Start by logging into your `Amazon AWS Console <https://console.aws.amazon.com/s3/home>`_ and click on the EC2 tab.
+
+Select the region you want to run your CKAN instance in - the security group you set up is region-specific. In this tutorial, we use EU West, so it will be easier to follow if you do too.
+
+.. image :: images/1.png
+
+Set up a Security Group
+^^^^^^^^^^^^^^^^^^^^^^^
+
+Click the **Security Groups** link in the **My Resources** section in the right-hand side of the dashboard.
+
+.. image :: images/2.png
+ :width: 807px
+
+Create a security group called ``web_test`` that gives access to ports 22, 80 and 5000 as shown below. This is needed so that you'll actually be able to access your server once it is created. You can't change these settings once the instance is running, so you need to do so now.
+
+.. image :: images/3a.png
+ :width: 807px
+
+.. image :: images/3b.png
+ :width: 807px
+
+Create a Keypair
+^^^^^^^^^^^^^^^^
+
+Now create a new keypair ``ckan_test`` to access your instance:
+
+.. image :: images/4.png
+ :width: 807px
+
+When you click **Create**, your browser will prompt you to save a keypair called ``ckan_test.pem``:
+
+.. image :: images/5.png
+ :width: 807px
+
+In this tutorial, we save the keypair in ``~/Downloads/ckan_test.pem``, but you should save it
+somewhere safe.
+
+.. note :: If you plan to boot your EC2 instance from the command line, you need to remember where you've put this file.
+
+
+Boot the EC2 Image
+******************
+
+CKAN requires Ubuntu 10.04 to run (either the i386 or amd64
+architectures). Luckily Canonical provide a `range of suitable images <http://uec-images.ubuntu.com/releases/10.04/release/>`_.
+
+The cheapest EC2 instance is the micro one, but that isn't very powerful, so in this tutorial,
+we'll use the 32-bit small version.
+
+We're in ``eu-west-1`` and we'll use an instance-only image (i.e. all the data will be lost when you shut it down) so we need the `ami-3693a542 <https://console.aws.amazon.com/ec2/home?region=eu-west-1#launchAmi=ami-3693a542>`_ AMI.
+
+.. note ::
+
+ There are more recent Ubuntu images at http://cloud.ubuntu.com/ami/ but we need the older 10.04 LTS release.
+
+At this point, you can either boot this image from the AWS
+console or launch it from the command line.
+
+
+Option 1: Boot the EC2 Image AMI via the AWS Console
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+From the EC2 dashboard, choose **Launch instance >**:
+
+.. image :: images/2.png
+ :width: 807px
+ :alt: Choose launch instance from the EC2 dashboard
+
+Now work through the wizard as shown in the following screenshots.
+
+In the first step search for ``ami-3693a542`` and select it from the results (it may take a few seconds for Amazon to find it).
+
+.. warning ::
+
+ No image other than ``ami-3693a542`` will work with CKAN.
+
+.. image :: images/i1.png
+ :width: 807px
+ :alt: Search for image ami-3693a542
+
+You can keep the defaults for all of the following screens:
+
+.. image :: images/i2.png
+ :width: 807px
+ :alt: Keep the defaults while setting up your instance
+.. image :: images/i3.png
+ :width: 807px
+ :alt: Keep the defaults while setting up your instance
+.. image :: images/i4.png
+ :width: 807px
+ :alt: Keep the defaults while setting up your instance
+.. image :: images/i5.png
+ :width: 807px
+ :alt: Keep the defaults while setting up your instance
+
+Choose the ``web_test`` security group you created earlier:
+
+.. image :: images/i6.png
+ :width: 807px
+ :alt: Choose the web_test security group you created earlier
+
+Then finish the wizard:
+
+.. image :: images/i7.png
+ :width: 807px
+ :alt: Finish the wizard
+
+Finally click the **View your instances on the Instances page** link:
+
+.. image :: images/i8.png
+ :width: 807px
+ :alt: View your instance
+
+After a few seconds you'll see your instance has booted. Now skip to :ref:`log-in-to-instance`.
+
+Option 2: Boot the EC2 Image AMI from the Command Line
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+[You can skip this section if you've just booted from the AWS console and go straight to :ref:`log-in-to-instance`]
+
+To boot from the command line you still need the same information but you enter it in one command. I'll show you now.
+
+Install The EC2 Tools Locally
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you are on Linux, you can just install the tools like this:
+
+::
+
+ sudo apt-get install ec2-ami-tools
+ sudo apt-get install ec2-api-tools
+
+If you are on Windows or Mac you'll need to `download them from the Amazon website <http://aws.amazon.com/developertools/351>`_.
+
+Once the software is installed you can use the files you've just downloaded to do create your instance.
+
+Get Security Certificates
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Next click on the **Account** link, right at the top of the screen, and you'll see this screen:
+
+.. image :: images/6.png
+ :width: 807px
+ :alt: The Account screen
+
+From this screen choose **Security Credentials** from the left hand side. Once
+the page has loaded scroll down and you'll see the **Access Credentials**
+section. Click on the **X.509 Certificate** tab:
+
+.. image :: images/7.png
+ :width: 807px
+ :alt: The Access Credentials screen
+
+Here you'll be able to create an X.509 certificate and private key.
+
+.. tip ::
+
+ You can only have two X.509 certificates at any given time, so you might need
+ to inactivate an old one first and then delete it before you are allowed to
+ create a new one, as in the screenshot above.
+
+Once you click the **Create New Certificate** link you get a popup which allows
+you to download the certificate and private key - do this. Once again, ours are in
+``~/Downloads``, but you should save it somewhere safe.
+
+.. image :: images/8.png
+ :width: 807px
+ :alt: Download your certificate
+
+.. tip ::
+
+ Amazon will only give you a private key file once when you create it so
+ although you can always go back to get a copy of the certificate, you can only
+ get the private key once. Make sure you save it in a safe place.
+
+You now have:
+
+* Your private key (``pk-[ID].pem``)
+* Your certificate file (``cert-[ID].pem``)
+* Your new keypair (``ckan-test.pem``)
+
+The private key and the certificate files have the same name in the ``ID`` part.
+
+Create an Ubuntu Instance
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Once the tools are installed, run this command:
+
+::
+
+ ec2-run-instances ami-3693a542 --instance-type m1.small --region eu-west-1 --group web_test \
+ --key ckan_test \
+ --private-key ~/Downloads/pk-[ID].pem \
+ --cert ~/Downloads/cert-[ID].pem
+
+
+.. note ::
+
+ The ``--key`` argument is the name of the keypair (``ckan_test``), not the certificate
+ itself (``ckan_test.pem``).
+
+.. warning ::
+
+ Amazon charge you for a minimum of one hour usage, so you shouldn't create and
+ destroy lots of EC2 instances unless you want to be charged a lot.
+
+.. _log-in-to-instance:
+
+Log in to the Instance
+**********************
+
+Once your instance has booted, you will need to find out its public DNS. Give it
+a second or two for the instance to load then browse to the running instance in
+the AWS console. If you tick your instance you'll be able to find the public
+DNS by scrolling down to the bottom of the **Description** tag.
+
+.. image :: images/8a.png
+ :width: 807px
+ :alt: Find the public DNS
+
+Here you can see that our public DNS is
+``ec2-79-125-86-107.eu-west-1.compute.amazonaws.com``. The private DNS only works
+from other EC2 instances so isn't any use to us.
+
+Once you've found your instance's public DNS, ensure the key has the correct permissions:
+
+::
+
+ chmod 0600 "ckan_test.pem"
+
+You can then log in like this:
+
+::
+
+ ssh -i ~/Downloads/ckan_test.pem ubuntu at ec2-46-51-149-132.eu-west-1.compute.amazonaws.com
+
+The first time you connect you'll see this, choose ``yes``:
+
+::
+
+ RSA key fingerprint is 6c:7e:8d:a6:a5:49:75:4d:9e:05:2e:50:26:c9:4a:71.
+ Are you sure you want to continue connecting (yes/no)? yes
+ Warning: Permanently added 'ec2-79-125-86-107.eu-west-1.compute.amazonaws.com,79.125.86.107' (RSA) to the list of known hosts.
+
+When you log in you'll see a welcome message. You can now proceed to :ref:`run-package-installer`.
+
+
+.. note ::
+
+ If this is a test install of CKAN, when you have finished using CKAN, you can shut down your EC2 instance through the AWS console.
+
+.. warning ::
+
+ Shutting down your EC2 instance will lose all your data. Also, Amazon charge you for a minimum usage of one hour, so don't create and destroy lots of EC2 instances unless you want to be charged a lot!
+
+
+.. _run-package-installer:
+
+Run the Package Installer
+-------------------------
+
+On your Ubuntu 10.04 system, open a terminal window and switch to the root user:
+
+::
+
+ sudo -s
+
+Install the CKAN packages as follows:
+
+::
+
+ echo 'deb http://apt.okfn.org/ubuntu_ckan-std_dev lucid universe' > /etc/apt/sources.list.d/okfn.list
+ wget -qO- http://apt.okfn.org/packages.okfn.key | sudo apt-key add -
+ apt-get update
+ apt-get install ckan-std
+
+Wait for the output to finish, then create your CKAN instance:
+
+::
+
+ ckan-std-install
+
+If you are using Amazon EC2, you will additionally need to set the hostname of your server. To do this, run the command below, replacing ``ec2-46-51-149-132.eu-west-1.compute.amazonaws.com`` with the public DNS of your EC2 instance. Leave the ``/`` at the end, as it is part of the ``sed`` command. Then restart Apache. You can skip this if installing on VirtualBox or a local server.
+
+::
+
+ sudo sed -e "s/ServerAlias \(.*\)/ServerAlias ec2-46-51-149-132.eu-west-1.compute.amazonaws.com/" \
+ -i /etc/apache2/sites-available/std.common
+ sudo /etc/init.d/apache2 restart
+
+Finally visit your CKAN instance - either at your Amazon EC2 hostname, or at http://localhost. You'll be redirected to the login screen because you won't have set up any permissions yet, so the welcome screen will look something like this.
+
+.. image :: images/9.png
+ :width: 807px
+
+You can now proceed to :doc:`post-installation`.
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/doc/install-from-source.rst Thu Jul 28 17:18:48 2011 +0100
@@ -0,0 +1,270 @@
+=============================
+Option 2: Install from Source
+=============================
+
+This section describes how to install CKAN from source. This removes the requirement for Ubuntu 10.04 that exists with :doc:`install-from-package`.
+
+.. warning:: This option is more complex than :doc:`install-from-package`, so we suggest only doing it this way if you plan to work on CKAN core, or have no access to Ubuntu 10.04 via any of the suggested methods.
+
+For support during installation, please contact `the ckan-dev mailing list <http://lists.okfn.org/mailman/listinfo/ckan-dev>`_.
+
+Install the Source
+------------------
+
+These are instructions to get developing with CKAN.
+
+Before you start, it may be worth `checking CKAN has passed the auto build and
+tests <http://buildbot.okfn.org/waterfall>`_.
+
+
+1. Ensure the required packages are installed.
+
+ If you have access to ``apt-get``, you can install these packages as follows:
+
+ ::
+
+ sudo apt-get install build-essential libxml2-dev libxslt-dev
+ sudo apt-get install wget mercurial postgresql libpq-dev git-core
+ sudo apt-get install python-dev python-psycopg2 python-virtualenv
+ sudo apt-get install subversion
+
+ Otherwise, you should install these packages from source.
+
+ ===================== ===============================================
+ Package Description
+ ===================== ===============================================
+ mercurial `Source control <http://mercurial.selenic.com/>`_
+ python `Python v2.5-2.7 <http://www.python.org/getit/>`_
+ postgresql `PostgreSQL database <http://www.postgresql.org/download/>`_
+ libpq `PostgreSQL library <http://www.postgresql.org/docs/8.1/static/libpq.html>`_
+ psycopg2 `PostgreSQL python module <http://initd.org/psycopg/install/>`_
+ libxml2 `XML library development files <http://xmlsoft.org/>`_
+ libxslt `XSLT library development files <http://www.linuxfromscratch.org/blfs/view/6.3/general/libxslt.html>`_
+ virtualenv `Python virtual environments <http://pypi.python.org/pypi/virtualenv>`_
+ wget `Command line tool for downloading from the web <http://www.gnu.org/s/wget/>`_
+ build-essential Tools for building source code (or up-to-date Xcode on Mac)
+ git `Git source control (for getting MarkupSafe src) <http://book.git-scm.com/2_installing_git.html>`_
+ subversion `Subversion source control (for pyutilib) <http://subversion.apache.org/packages.html>`_
+ ===================== ===============================================
+
+
+
+2. Create a Python virtual environment.
+
+ In your home directory run the command below. It is currently important to
+ call your virtual environment ``pyenv`` so that the automated deployment tools
+ work correctly.
+
+ ::
+
+ cd ~
+ virtualenv pyenv
+
+ .. tip ::
+
+ If you don't have a ``python-virtualenv`` package in your distribution
+ you can get a ``virtualenv.py`` script from within the
+ `virtualenv source distribution <http://pypi.python.org/pypi/virtualenv/>`_
+ and then run ``python virtualenv.py pyenv`` instead.
+
+ To help with automatically installing CKAN dependencies we use a tool
+ called ``pip``. Make sure you have activated your environment (see step 3)
+ and then install it from an activated shell like this:
+
+ ::
+
+ easy_install pip
+
+3. Activate your virtual environment.
+
+ To work with CKAN it is best to adjust your shell settings so that your
+ shell uses the virtual environment you just created. You can do this like
+ so:
+
+ ::
+
+ . pyenv/bin/activate
+
+ When your shell is activated you will see the prompt change to something
+ like this:
+
+ ::
+
+ (pyenv)[ckan at host ~/]$
+
+ An activated shell looks in your virtual environment first when choosing
+ which commands to run. If you enter ``python`` now it will actually
+ run ``~/pyenv/bin/python`` which is what you want.
+
+4. Install CKAN code and required Python packages into the new environment.
+
+ First you'll need to install CKAN. For the latest version run:
+
+ ::
+
+ pip install --ignore-installed -e hg+http://bitbucket.org/okfn/ckan#egg=ckan
+
+ CKAN has a set of dependencies it requires which you should install too:
+
+ ::
+
+ pip install --ignore-installed -r pyenv/src/ckan/requires/lucid_missing.txt -r pyenv/src/ckan/requires/lucid_conflict.txt
+
+ The ``--ignore-installed`` option ensures ``pip`` installs software into
+ this virtual environment even if it is already present on the system.
+
+ If you are using Ubuntu Lucid you can install the rest of the dependencies
+ from the system versions like this:
+
+ ::
+
+ sudo apt-get install python-psycopg2 python-lxml python-sphinx
+ sudo apt-get install python-pylons python-formalchemy python-repoze.who
+ sudo apt-get install python-repoze.who-plugins python-tempita python-zope.interface
+
+ If you are not using Ubuntu Lucid you'll still need to install all the
+ dependencies that would have been met in the ``apt-get install`` command
+ at the start. You can do so like this:
+
+ ::
+
+ pip install --ignore-installed -r pyenv/src/ckan/requires/lucid_present.txt
+
+ This will take a **long** time. Particularly the install of the ``lxml``
+ package.
+
+ At this point you will need to deactivate and then re-activate your
+ virtual environment to ensure that all the scripts point to the correct
+ locations:
+
+ ::
+
+ deactivate
+ . pyenv/bin/activate
+
+5. Setup a PostgreSQL database.
+
+ List existing databases:
+
+ ::
+
+ psql -l
+
+ It is advisable to ensure that the encoding of databases is 'UTF8', or
+ internationalisation may be a problem. Since changing the encoding of PostgreSQL
+ may mean deleting existing databases, it is suggested that this is fixed before
+ continuing with the CKAN install.
+
+ Next you'll need to create a database user if one doesn't already exist.
+
+ .. tip ::
+
+ If you choose a database name, user or password which are different from those
+ suggested below then you'll need to update the configuration file you'll create in
+ the next step.
+
+ Here we choose ``ckantest`` as the database and ``ckanuser`` as the user:
+
+ ::
+
+ sudo -u postgres createuser -S -D -R -P ckantest
+
+ It should prompt you for a new password for the CKAN data in the database.
+ It is suggested you enter ``pass`` for the password.
+
+ Now create the database, which we'll call ``ckantest`` (the last argument):
+
+ ::
+
+ sudo -u postgres createdb -O ckantest ckantest
+
+6. Create a CKAN config file.
+
+ Make sure you are in an activated environment (see step 3) so that Python
+ Paste and other modules are put on the python path (your command prompt will
+ start with ``(pyenv)`` if you have) then change into the ``ckan`` directory
+ which will have been created when you installed CKAN in step 4 and create the
+ config file ``development.ini`` using Paste:
+
+ ::
+
+ cd pyenv/src/ckan
+ paster make-config ckan development.ini
+
+ You can give your config file a different name but the tests will expect you
+ to have used ``development.ini`` so it is strongly recommended you use this
+ name, at least to start with.
+
+ If you used a different database name or password when creating the database
+ in step 5 you'll need to now edit ``development.ini`` and change the
+ ``sqlalchemy.url`` line, filling in the database name, user and password you used.
+
+ ::
+
+ sqlalchemy.url = postgresql://ckantest:pass@localhost/ckantest
+
+ If you're using a remote host with password authentication rather than SSL authentication, use::
+
+ sqlalchemy.url = postgresql://<user>:<password>@<remotehost>/ckan?sslmode=disable
+
+ .. caution ::
+
+ Advanced users: If you are using CKAN's fab file capability you currently need to create
+ your config file as ``pyenv/ckan.net.ini`` so you will probably have
+ ignored the advice about creating a ``development.ini`` file in the
+ ``pyenv/src/ckan`` directory. This is fine but CKAN probably won't be
+ able to find your ``who.ini`` file. To fix this edit ``pyenv/ckan.net.ini``,
+ search for the line ``who.config_file = %(here)s/who.ini`` and change it
+ to ``who.config_file = who.ini``.
+
+ We are moving to a new deployment system where this incompatibility
+ will be fixed.
+
+7. Create database tables.
+
+ Now that you have a configuration file that has the correct settings for
+ your database, you'll need to create the tables. Make sure you are still in an
+ activated environment with ``(pyenv)`` at the front of the command prompt and
+ then from the ``pyenv/src/ckan`` directory run this command:
+
+ ::
+
+ paster db init
+
+ You should see ``Initialising DB: SUCCESS``. If you are not in the
+ ``pyenv/src/ckan`` directory or you don't have an activated shell, the command
+ will not work.
+
+ If the command prompts for a password it is likely you haven't set up the
+ database configuration correctly in step 6.
+
+8. Create the cache directory.
+
+ You need to create the Pylon's cache directory specified by 'cache_dir'
+ in the config file.
+
+ (from the ``pyenv/src/ckan`` directory):
+
+ ::
+
+ mkdir data
+
+
+9. Run the CKAN webserver.
+
+ NB If you've started a new shell, you'll have to activate the environment
+ again first - see step 3.
+
+ (from the ``pyenv/src/ckan`` directory):
+
+ ::
+
+ paster serve development.ini
+
+10. Point your web browser at: http://127.0.0.1:5000/
+
+ The CKAN homepage should load.
+
+Finally, make sure that tests pass, as described in :ref:`basic-tests`.
+
+You can now proceed to :doc:`post-installation`.
--- a/doc/load_testing.rst Thu Jul 28 16:45:02 2011 +0100
+++ /dev/null Thu Jan 01 00:00:00 1970 +0000
@@ -1,70 +0,0 @@
-=================
-Load Testing CKAN
-=================
-
-What we did
-===========
-
-1. Put db in some standard state and start server::
-
- paster db clean && paster db create && paster create-test-data search
- # alternatively we could use a dump of the live db
- # psql -h localhost --user tester ckantest < ....
-
- # may want to create a production config
- paster serve development.ini
-
-2. Do tests::
-
- # 5 results for this query
- ab -n 100 -c 10 -e myresults.csv "http://localhost:5000/api/search/package?q=government&all_fields=1"
- # remote test host
- # ab -n 100 -c 10 -e myresults.csv "http://test.ckan.net/api/search/package?q=geo&all_fields=1"
-
-3. Examine results, fix, and repeat!
-
-
-Research
-========
-
-Testing Tools
-+++++++++++++
-
-Apache Benchmarking Tool
-------------------------
-
- * http://httpd.apache.org/docs/2.0/programs/ab.html
- * Well known and standard.
- * Can run against local or remote but local preferred
- * ab -n 100 -c 10 -e myresults.csv http://www.okfn.org/
-
-funkload
---------
-
- * Mature and python based
- * http://funkload.nuxeo.org/
- * Functional testing primarily but with load testing support
-
-Individual scripts
-------------------
-
- * http://code.google.com/appengine/articles/load_test.html
-
-
-High Performance Servers and Caching Utilities
-----------------------------------------------
-
-memcached
-+++++++++
-
- * http://memcached.org/
- * In memory caching with access via a port
- * Very heavily used
-
-Tornado
-+++++++
-
- * http://www.tornadoweb.org/
- * Built by friendfeed and open-sourced
- * Asynchronous
-
--- a/doc/loader_scripts.rst Thu Jul 28 16:45:02 2011 +0100
+++ /dev/null Thu Jan 01 00:00:00 1970 +0000
@@ -1,31 +0,0 @@
-==============
-Loader scripts
-==============
-
-Introduction
-============
-
-'Loader scripts' provide a simple way to take any format metadata and bulk upload it to a remote CKAN instance. Essentially each custom script has to convert the metadata to the standard 'package' dictionary format, and the loader does the work of loading it into CKAN in an intelligent way.
-
-
-Structure
-=========
-
-First you need an importer that derives from `PackageImporter` (ckan/lib/importer.py). This takes whatever format the metadata is in and sorts it into records of type `DataRecord`. Then each DataRecord is converted into the correct fields for a package using the `record_2_package` method. This results in package dictionaries.
-
-Note: for CSV and Excel formats, there is already the `SpreadsheetPackageImporter` (ckan/lib/spreadsheet_importer.py) which wraps the file in `SpreadsheetData` before extracting the records into `SpreadsheetDataRecords`.
-
-The `PackageLoader` takes the package dictionaries and loads them onto a CKAN instance using the ckanclient. There are various settings to determine:
- * how to identify the same package, previously been loaded into CKAN. This canbe simply by name or by an identifier stored in another field.
- * how to merge in changes to an existing packages. It can simply replace it or maybe merge in resources etc.
-
-Loaders generally go into the `ckanext` repository.
-
-The loader shoud be given a command line interface using the `Command` base class (ckanext/command.py). You need to add a line to the setup.py `[console_scripts]` and when you run ``python setup.py develop`` it creates a script for you in your python environment.
-
-
-Example
-=======
-
-To get a flavour of what these scripts look like, take a look at the ONS scripts: https://bitbucket.org/okfn/ckanext-dgu/src/default/ckanext/dgu/ons/
-
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/doc/loading_data.rst Thu Jul 28 17:18:48 2011 +0100
@@ -0,0 +1,99 @@
+=============
+Load Datasets
+=============
+
+You can upload individual datasets through the CKAN front-end, but for importing datasets on masse, you have two choices:
+
+* :ref:`load-data-api`. You can use the `CKAN API <api.html>`_ to script import. To simplify matters, we offer provide standard loading scripts for Google Spreadsheets, CSV and Excel.
+
+* :ref:`load-data-harvester`. The `CKAN harvester extension <https://bitbucket.org/okfn/ckanext-harvest/>`_ provides web and command-line interfaces for larger import tasks.
+
+If you need advice on data import, `contact the ckan-dev mailing list <http://lists.okfn.org/mailman/listinfo/ckan-dev>`_.
+
+.. note :: If loading your data requires scraping a web page regularly, you may find it best to write a scraper on `ScraperWiki <http://www.scraperwiki.com>`_ and combine this with either of the methods above.
+
+.. _load-data-api:
+
+Import Data with the CKAN API
+-----------------------------
+
+You can use the `CKAN API <api.html>`_ to upload datasets directly into your CKAN instance.
+
+The Simplest Approach - ckanclient
+++++++++++++++++++++++++++++++++++
+
+The most basic way to automate package loading is with a Python script using the `ckanclient library <http://pypi.python.org/pypi/ckanclient>`_. You will need to register for an API key first.
+
+You can install ckanclient with::
+
+ pip install ckanclient
+
+Here is an example script to register a new package::
+
+ import ckanclient
+ # Instantiate the CKAN client.
+ ckan = ckanclient.CkanClient(api_key=my_api_key, base_location="http://myckaninstance.com/api")
+ # Describe the package.
+ package_entity = {
+ 'name': my_package_name,
+ 'url': my_package_url,
+ 'download_url': my_package_download_url,
+ 'tags': my_package_keywords,
+ 'notes': my_package_long_description,
+ }
+ # Register the package.
+ ckan.package_register_post(package_entity)
+
+Loader Scripts
+++++++++++++++
+
+'Loader scripts' provide a simple way to take any format metadata and bulk upload it to a remote CKAN instance.
+
+Essentially each set of loader scripts converts the dataset metadata to the standard 'package' format, and then loads it into CKAN.
+
+Loader scripts are generally stored into the `ckanext` repository. To get a flavour of what loader scripts look like, take a look at `the ONS scripts <https://bitbucket.org/okfn/ckanext-dgu/src/default/ckanext/dgu/ons/>`_.
+
+Loader Scripts for CSV and Excel
+********************************
+
+For CSV and Excel formats, the `SpreadsheetPackageImporter` (found in ``ckan/lib/spreadsheet_importer.py``) loader script wraps the file in `SpreadsheetData` before extracting the records into `SpreadsheetDataRecords`.
+
+SpreadsheetPackageImporter copes with multiple title rows, data on multiple sheets, dates. The loader can reload packages based on a unique key column in the spreadsheet, choose unique names for packages if there is a clash, add/merge new resources for existing packages and manage package groups.
+
+Loader Scripts for Google Spreadsheets
+**************************************
+
+The `GoogleSpreadsheetReader` class (found in ``ckanclient.loaders``) simplifies the process of loading data from Google Spreadsheets.
+
+`This script <https://bitbucket.org/okfn/ckanext/src/default/bin/ckanload-italy-nexa>`_ has a simple example of loading data from Google Spreadsheets.
+
+Write Your Own Loader Script
+****************************
+
+## this needs work ##
+
+First, you need an importer that derives from `PackageImporter` (found in ``ckan/lib/importer.py``). This takes whatever format the metadata is in and sorts it into records of type `DataRecord`.
+
+Next, each DataRecord is converted into the correct fields for a package using the `record_2_package` method. This results in package dictionaries.
+
+The `PackageLoader` takes the package dictionaries and loads them onto a CKAN instance using the ckanclient. There are various settings to determine:
+
+ * ##how to identify the same package, previously been loaded into CKAN.## This can be simply by name or by an identifier stored in another field.
+ * how to merge in changes to an existing packages. It can simply replace it or maybe merge in resources etc.
+
+The loader should be given a command-line interface using the `Command` base class (``ckanext/command.py``).
+
+You need to add a line to the CKAN ``setup.py`` (under ``[console_scripts]``) and when you run ``python setup.py develop`` it creates a script for you in your Python environment.
+
+.. _load-data-harvester:
+
+Import Data with the Harvester Extension
+----------------------------------------
+
+The `CKAN harvester extension <https://bitbucket.org/okfn/ckanext-harvest/>`_ provides useful tools for more advanced data imports.
+
+These include a command-line interface and a web user interface for running harvesting jobs.
+
+To use the harvester extension, derive from the `base class of the harvester extension <https://bitbucket.org/okfn/ckanext-harvest/src/61844c8d2374/ckanext/harvest/harvesters/base.py>`_ and then write a custom ``_create_or_update_package`` method for your data.
+
+For more information on working with extensions, see :doc:`extensions`.
--- a/doc/model.rst Thu Jul 28 16:45:02 2011 +0100
+++ /dev/null Thu Jan 01 00:00:00 1970 +0000
@@ -1,96 +0,0 @@
-=====
-Model
-=====
-
-The structure of the CKAN data is described in the 'model'. This is in the code at::
-
- ckan/model
-
-Many of the domain objects are Revisioned and some are Stateful. These are concepts introduced by vdm.
-
-Migration
-=========
-
-When edits are made to the model code, then before the code can be used on a CKAN instance with existing data, the existing data has to be migrated. This is achieved with a migration script. CKAN currently uses `SqlAlchemy Migrate <http://code.google.com/p/sqlalchemy-migrate/>`_ to manage these scripts.
-
-When you deploy new code to a CKAN instance, as part of the process you run any required migration scripts with::
-
- paster db upgrade
-
-The scripts give their model version numbers in their filenames and are stored here::
-
- ckan/migration/versions/
-
-The current version the database is migrated to is also stored in the database. When you run the upgrade, as each migration script is run it prints to the console something like ``11->12``. If no upgrade is required because it is up to date, then nothing is printed.
-
-Creating a new migration script
-===============================
-
-A migration script should be checked into CKAN at the same time as the model changes it is related to. Before pushing the changes, ensure the tests pass when running against the migrated model, which requires the ``--ckan-migration`` setting - see `<README.html#migrationtesting>`_.
-
-To create a new migration script, create a python file in ckan/migration/versions/ and name it with a prefix numbered one higher than the previous one and some words describing the change.
-
-You need to use the special engine provided by the SqlAlchemy Migrate. Here is the standard header for your migrate script::
-
- from sqlalchemy import *
- from migrate import *
-
-
-The migration operations go in the upgrade function::
-
- def upgrade(migrate_engine):
- metadata = MetaData()
- metadata.bind = migrate_engine
-
-The following process should be followed when doing a migration. This process is here to make the process easier and to validate if any mistakes have been made.
-
-1. Get a dump of the database schema before you add your new migrate scripts.
-
- paster db clean
- paster db upgrade
- pg_dump -h host -s -f old.sql dbname
-
-2. Get a dump of the database as you have specified it in the model.
-
- paster db clean
- paster db create-test #this makes the database as defined in the model
- pg_dump -h host -s -f new.sql dbname
-
-3. Get agpdiff (apt-get it). It produces sql it thinks that you need to run on the database in order to get it to the updated schema.
-
- apgdiff old.sql new.sql > upgrade.diff
- (or if you don't want to install java use http://apgdiff.startnet.biz/diff_online.php)
-
-4. The upgrade.diff file created will have all the changes needed in sql. Delete the drop index lines as they are not created in the model.
-
-5. Put the resulting sql in your migrate script.
-
- eg migrate_engine.execute('''update table .........; update table ....''')
-
-6. Do a dump again, then a diff again to see if the the only thing left are drop index statements.
-
-7. run nosetests with --ckan-migration flag.
-
-Its that simple. Well almost..
-
-* If you are doing any table/field renaming adding that to your new migrate script first and use this as a base for your diff (i.e add a migrate script with these renaming before 1). This way the resulting sql wont try and drop and recreate the field/table!!
-* It sometimes drops the foreign key constraints in the wrong order causing an error so you may need to rearrange the order in the resulting upgrade.diff.
-* If you need to do any data transfer in the migrations then do it between the dropping of the constraints and adding of new ones.
-* May need to add some tests if you are doing data migrations.
-
-An example of a script doing it this way is 034_resource_group_table.py. This script copies the definitions of the original tables in order to do the renaming the tables/fields.
-
-In order to do some basic data migration testing extra assertions should be added to the migration script.
-
-Examples of this can also be found in 034_resource_group_table.py for example.
-
-This statement is run at the top of the migration script to get the count of rows::
-
- package_count = migrate_engine.execute('''select count(*) from package''').first()[0]
-
-And the following is run after to make sure that row count is the same::
-
- resource_group_after = migrate_engine.execute('''select count(*) from resource_group''').first()[0]
- assert resource_group_after == package_count
-
-
--- a/doc/paster.rst Thu Jul 28 16:45:02 2011 +0100
+++ b/doc/paster.rst Thu Jul 28 17:18:48 2011 +0100
@@ -1,16 +1,81 @@
-Paster commands
-+++++++++++++++
+=================
+Common CKAN Tasks
+=================
-Paster commands provide a number of useful ways to administer a CKAN installation. These are run on the command line on the server running CKAN.
+The majority of common CKAN administration tasks are carried out using the **paster** script.
-Commands
-========
+Paster is run on the command line on the server running CKAN. This section covers:
+
+* :ref:`paster-understanding`. Understanding paster syntax and getting help.
+* :ref:`paster-tasks`. How to carry out common CKAN admin tasks using paster.
+
+.. _paster-understanding:
+
+Understanding Paster
+====================
+
+The basic paster format is::
+
+ paster --plugin=ckan <ckan commands> --config=<config file>
+
+For example, to initialise a database::
+
+ paster --plugin=ckan db init --config=/etc/ckan/std/std.ini
+
+
+.. _paster-help:
+
+Getting Help on Paster
+----------------------
+
+To get a full list of paster commands (i.e. including CKAN commands)::
+
+ paster --plugin=ckan --help
+
+And to get more detailed help on each command (e.g. on ``db``)::
+
+ paster --plugin=ckan --help db
+
+
+Position of Paster Parameters
+-----------------------------
+
+The position of paster parameters matters.
+
+``--plugin`` is a parameter to paster, so needs to come before the CKAN command. To do this, the first parameter to paster is normally ``--plugin=ckan``.
+
+.. note:: The default value for ``--plugin`` is ``setup.py`` in the current directory. If you are running paster from the directory where CKAN's ``setup.py`` file is located, you don't need to specify the plugin parameter..
+
+Meanwhile, ``--config`` is a parameter to CKAN, so needs to come after the CKAN command. This specifies the CKAN config file for the instance you want to use, e.g. ``--config=/etc/ckan/std/std.ini``
+
+.. note:: The default value for ``--config`` is ``development.ini`` in the current directory. If you are running a package install of CKAN (as described in :doc:`install-from-package`), you should explicitly specify ``std.ini``.
+
+The position of the CKAN command itself is less important, as longs as it follows ``--plugin``. For example, both the following commands have the same effect:::
+
+ paster --plugin=ckan db --config=development.ini init
+ paster --plugin=ckan db init --config=development.ini
+
+
+Running a Paster Shell
+----------------------
+
+If you want to run a "paster shell", which can be useful for development, then the plugin is pylons. e.g. ``paster --plugin=pylons shell``.
+
+Often you will want to run this as the same user as the web application, to ensure log files are written as the same user. And you'll also want to specify a config file (note that this is not specified using the ``--config`` parameter, but simply as the final argument). For example::
+
+ sudo -u www-data paster --plugin=pylons shell std.ini
+
+
+.. _paster-tasks:
+
+Common Tasks Using Paster
+=========================
+
+The following tasks are supported by paster.
================= ==========================================================
- changes Distribute changes
create-test-data Create test data in the database.
db Perform various tasks on the database.
- notify Send out modification notifications.
ratings Manage the ratings stored in the db
rights Commands relating to per-object and system-wide access rights.
roles Commands relating to roles and actions.
@@ -20,55 +85,98 @@
================= ==========================================================
-Running paster commands
-=======================
+For the full list of tasks supported by paster, you can run::
+
+ paster --plugin=ckan --help
-Basically the format is::
- paster --plugin=ckan <ckan commands> --config=<config file>
+create-test-data: Create test data
+----------------------------------
-e.g.::
+As the name suggests, this command lets you load test data when first setting up CKAN. See :ref:`create-test-data` for details.
- paster --plugin=ckan db init --config=myckan.ini
-
-To get a list of paster commands (i.e. including CKAN commands)::
-
- paster --plugin=ckan --help
-
-And you can get help for each command. e.g.::
-
- paster --plugin=ckan --help db
-
-
-``--plugin`` parameter
+db: Manage databases
--------------------
-Paster is a system-wide command, so you need to tell it to find the commands defined in the ckan code. So first parameter to paster is normally ``--plugin=ckan``.
+Lets you initialise, upgrade, and dump database files in various formats.
-NB: If your current directory is where CKAN's setup.py is, you don't need to specify this parameter.
+For example, to initialise the CKAN database, creating the tables that CKAN uses (note that you don't need to do this during setup if you have run ``create-test-data``)::
-If you want to run a "paster shell" then the plugin is pylons. e.g. ``paster --plugin=pylons shell``. Often you will want to run this as the same user as the web application, to ensure log files are written as the same user. And you'll also want to specify a config file. e.g.::
+ paster --plugin=ckan db init --config=/etc/ckan/std/std.ini
- sudo -u www-data paster --plugin=pylons shell myckan.ini
+When you upgrade CKAN software by any method *other* than the package update described in :doc:`upgrade`, before you restart it, you should run 'db upgrade', to migrate the database tables if necessary::
+ paster --plugin=ckan db upgrade --config=/etc/ckan/std/std.ini
-``--config`` parameter
-----------------------
+For information on using ``db`` to create dumpfiles, see :doc:`database_dumps`.
-Each server can have multiple CKAN instances running, so use this parameter to specify the CKAN config file for the instance you want to operate on. e.g. ``--config=myckan.ini``
-NB: Developers tend to use development.ini, and this is the default value (looking in the current directory), so in this situation you don't need to specify this parameter.
+ratings: Manage package ratings
+-------------------------------
+Manages the ratings stored in the database, and can be used to count ratings, remove all ratings, or remove only anonymous ratings.
-Position of paster parameters
+For example, to remove anonymous ratings from the database::
+
+ paster --plugin=ckan ratings clean-anonymous --config=/etc/ckan/std/std.ini
+
+
+rights: Set user permissions
+----------------------------
+
+Sets the authorization roles of a specific user on a given object within the system.
+
+For example, to give the user named 'bar' the 'admin' role on the package 'foo'::
+
+ paster --plugin=ckan rights make bar admin package:foo --config=/etc/ckan/std/std.ini
+
+To list all the rights currently specified::
+
+ paster --plugin=ckan rights list --config=/etc/ckan/std/std.ini
+
+For more information and examples, see :doc:`authorization`.
+
+
+roles: Manage system-wide permissions
+--------------------------------------
+
+This important command gives you fine-grained control over CKAN permissions, by listing and modifying the assignment of actions to roles.
+
+The ``roles`` command has its own section: see :doc:`authorization`.
+
+
+search-index: Rebuild search index
+----------------------------------
+
+Rebuilds the search index defined in the :ref:`config-search-backend` config setting. This is useful to prevent search indexes from getting out of sync with the main database.
+
+For example::
+
+ paster --plugin=ckan search-index --config=/etc/ckan/std/std.ini
+
+
+sysadmin: Give sysadmin rights
+------------------------------
+
+Gives sysadmin rights to a named user. This means the user can perform any action on any object.
+
+For example, to make a user called 'admin' into a sysadmin::
+
+ paster --plugin=ckan sysadmin add admin --config=/etc/ckan/std/std.ini
+
+
+.. _paster-user:
+
+user: Create and manage users
-----------------------------
-``--plugin`` is a paster parameter, so needs to come before the CKAN command, but <code>--config</code> is a ckan parameter so needs to come after the CKAN command.
+Lets you create, remove, list and manage users.
-Here are three ways to express the same thing::
+For example, to create a new user called 'admin'::
- paster db init
- paster --plugin=ckan db --config=development.ini init
- paster --plugin=ckan db init --config=development.ini
+ paster --plugin=ckan user add admin --config=/etc/ckan/std/std.ini
+To delete the 'admin' user::
+
+ paster --plugin=ckan user delete admin --config=/etc/ckan/std/std.ini
--- a/doc/plugins.rst Thu Jul 28 16:45:02 2011 +0100
+++ b/doc/plugins.rst Thu Jul 28 17:18:48 2011 +0100
@@ -1,44 +1,24 @@
-CKAN Extensions, Plugins Interfaces and Workers
-+++++++++++++++++++++++++++++++++++++++++++++++
+===============================
+Understand and Write Extensions
+===============================
-**Note: the terms "extension", "plugin interface" and "worker"
-now have very precise meanings and that the use of the generic word "plugin" to
-describe any way in which CKAN might be extended is deprecated.**
+If you want to extend CKAN core functionality, the best way to do so is by writing extensions.
-.. contents ::
+Extensions allow you to customise CKAN for your own requirements, without interfering with the basic CKAN system.
-Background
-----------
-
-The CKAN codebase is open source so that different organisations can customise
-it for their own needs. As CKAN has grown in popularity it has become
-especially important to organise the code in a way that can accommodate the
-customisations which different organisations require without those changes
-interfering with customizations required by other organisations.
-
-To meet this need we have introduced the concepts of CKAN extensions, plugin
+To meet the need to customize CKAN efficiently, we have introduced the concepts of CKAN extensions, plugin
interfaces and workers. These work together to provide a simple mechanism to
extend core CKAN functionality.
-Let's start by looking at extensions.
+.. warning:: This is an advanced topic. At the moment, you need to have prepared your system to work with extensions, as described in :doc:`prepare-extensions`. We are working to make the most popular extensions more easily available as Debian packages.
+
+.. note:: The terms **extension**, **plugin interface** and **worker** have very precise meanings: the use of the generic word **plugin** to describe any way in which CKAN might be extended is deprecated.
+
+.. contents ::
CKAN Extensions
---------------
-CKAN currently has the following extensions:
-
-* disqus_ extension for user comments
-* Solr extension for full text search and result faceting
-* Asyncronous queue extension based on AMQP for model update, harvesting and other operations
-* Custom form extensions for different governments
-
-.. note ::
-
- The form extension does not currently behave quite the same way as the other
- extensions, it will do soon.
-
-All CKAN extensions have package names of the form ``ckanext-name`` so the
-queue extension code is stored in the package ``ckanext-queue`` for example.
Extensions are implemented as *namespace packages* under the ``ckanext``
package which means that they can be imported like this:
@@ -47,48 +27,11 @@
$ python
>>> import ckanext.queue
-Extensions are used for any code that is not required for the core CKAN code to
-operate but which are nethertheless and important part of one or more CKAN
-instances.
-
Individual CKAN *extensions* may implement one or more *plugin interfaces* or
*workers* to provide their functionality. You'll learn about these later on.
-All CKAN extensions are described on the CKAN public wiki at
-http://wiki.okfn.org/ckan/extensions. If you write an extension then do share
-details of it there but also document the plugin interfaces the extension
-implements so that when people install it, they will know which plugin
-interfaces they need to set up in their config file.
-Installing an extension
-~~~~~~~~~~~~~~~~~~~~~~~
-
-To install an extension on a CKAN instance:
-
-1. Install the extension package code using pip. The -E parameter is for your
-CKAN python environment (e.g. ``~/var/srvc/ckan.net/pyenv``). Prefix the source
-url with the repo type (``hg+`` for Mercurial, ``git+`` for Git). For example::
-
- $ pip install -E ~/var/srvc/ckan.net/pyenv hg+http://bitbucket.org/okfn/ckanext-disqus
-
-2. Add the names of any plugin implementations the extension uses to the CKAN
-config. The config file may have a filepath something like:
-``~/var/srvc/ckan.net/ckan.net.ini``. The plugins variable is in the '[app:main]'
-section under 'ckan.plugins'. e.g.::
-
- [app:main]
- ckan.plugins = disqus
-
- If your extension implemented multiple different plugin interfaces, separate them with spaces::
-
- ckan.plugins = disqus amqp myplugin
-
-3. Restart WSGI, which usually means restarting Apache::
-
- $ sudo /etc/init.d/apache2 restart
-
-
-Creating your own extension
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Create Your Own Extension
+~~~~~~~~~~~~~~~~~~~~~~~~~
All CKAN extensions must start with the name ``ckanext-``. You can create your
own CKAN extension like this:
@@ -146,9 +89,35 @@
To build useful extensions you need to be able to "hook into" different parts
of CKAN in order to extend its functionality. You do this using CKAN's plugin
-architeture. We'll look at this in the next section. If you do write a CKAN
-extension you may well want to publish it publicly so others can use it too.
-See the `Publishing your extension`_ section below to find out how.
+architecture. We'll look at this in the next section.
+
+Testing Extensions
+``````````````````
+
+CKAN extensions ordinarily have their own ``test.ini`` that refers to the CKAN ``test.ini``, so you can run them in exactly the same way. For example::
+
+ cd ckanext-dgu
+ nosetests ckanext/dgu/tests --ckan
+ nosetests ckanext/dgu/tests --ckan --with-pylons=test-core.ini
+
+To test your changes you'll need to use the ``paster serve`` command from the ``ckan`` directory:
+
+::
+
+ cd /home/ubuntu/pyenv/src/ckan
+ . ../../bin/activate
+ paster make-config ckan development.ini
+
+Then make any changes to the ``development.ini`` file that you need before continuing:
+
+::
+
+ paster db upgrade
+ paster serve --reload
+
+You should also make sure that your CKAN installation passes the developer tests, as described in :doc:`test`.
+
+Finally, if you write a CKAN, extension you may well want to publish it so others can use it too. See the `Publishing your extension`_ section below for details.
Plugins
-------
@@ -232,11 +201,11 @@
.. tip ::
- This example is based on real code used to implement the ``ckanext-discus`` plugin
+ This example is based on real code used to implement the ``ckanext-disqus`` plugin
to add commenting to packages. You can see the latest version of this code at
http://bitbucket.org/okfn/ckanext-disqus/src/tip/ckanext/plugins/disqus/__init__.py.
-First we set up logging and some helpers we'll need from Genshi to transfor the stream:
+First we set up logging and some helpers we'll need from Genshi to transfer the stream:
::
@@ -256,11 +225,11 @@
from ckan.plugins.interfaces import IConfigurable, IGenshiStreamFilter
In this case we are implementing both the ``IConfigurable`` and
-``IGenshiStreamFilter`` plugin interfaces in out plugin class. The
+``IGenshiStreamFilter`` plugin interfaces in our plugin class. The
``IConfigurable`` plugin interface defines a ``configure()`` method which will
be is called on out plugin to let it know about configuration options. The
``IGenshiStreamFilter`` plugin interface defines a ``filter()`` method which
-will be called on the plugin to give it the oppurtunity to change the template
+will be called on the plugin to give it the opportunity to change the template
before the HTML is returned to the browser.
Let's have a look at the code:
@@ -337,7 +306,7 @@
interfaces to define your own "hooks". Before we can use the ``Disqus`` plugin
there is one more thing to do: add it to the extension and set an *entry point*.
-Setting the entry point
+Setting the Entry Point
~~~~~~~~~~~~~~~~~~~~~~~
Imagine the code above was saved into a file named ``disqus.py`` in the
@@ -359,7 +328,6 @@
CKAN finds plugins by searching for entry points in the group ``ckan.plugin``.
-
Entry points are defined in a package's ``setup.py`` file. If you look in the
``setup.py`` file for the ``ckanext-myname`` extension you'll see these
lines commented out towards the end:
@@ -411,10 +379,10 @@
``python setup.py develop`` if needed). Your users will then need to add the
new entry point name to their ``ckan.plugins`` config option too.
-Writing a database plugin
+Writing a Database Plugin
~~~~~~~~~~~~~~~~~~~~~~~~~
-You've seen how to use ``IConfigurable`` and ``IGenshiStreamFilter``, here's
+You've seen how to use ``IConfigurable`` and ``IGenshiStreamFilter``. Here's
another example which implements ``IMapperExtension`` to log messages after any
record is inserted into the database.
@@ -447,10 +415,11 @@
Publishing Your Extension
~~~~~~~~~~~~~~~~~~~~~~~~~
-At this point you might want to share your extension with the public. First
-check you have chosen an open source license (`MIT
-<http://opensource.org/licenses/mit-license.html>`_ license is fine for
-example) and then update the ``long_description`` variable in ``setup.py`` to
+At this point you might want to share your extension with the public.
+
+First check you have chosen an open source licence (e.g. the `MIT
+<http://opensource.org/licenses/mit-license.html>`_ licence) and then
+update the ``long_description`` variable in ``setup.py`` to
explain what the extension does and which entry point names a user of the
extension will need to add to their ``ckan.plugins`` configuration.
@@ -462,14 +431,14 @@
python setup.py register
python setup.py sdist upload
-You'll then see your extension at http://pypi.python.org/pypi. Others will then
+You'll then see your extension at http://pypi.python.org/pypi. Others will
be able to install your plugin with ``pip``.
-You should also add a summary of your extension and its entry points to
-http://wiki.okfn.org/ckan/extensions. You can create a new account if you need
-to.
+Finally, please also add a summary of your extension and its entry points to the Extensions page on
+http://wiki.ckan.net.
-Writing a plugin interface
+
+Writing a Plugin Interface
~~~~~~~~~~~~~~~~~~~~~~~~~~
This describes how to add a plugin interface to make core CKAN code pluggable.
@@ -509,11 +478,11 @@
your class via ``self.plugin``.
See the pyutilib_ documentation for more information on creating interfaces and
-plugins. Be aware though that pyutilib uses slightly different terminology. It
+plugins. However, be aware that pyutilib uses slightly different terminology. It
calls ``PluginImplementations`` ``ExtensionPoint`` and it calls instances of a
plugin object a *service*.
-Testing plugins
+Testing Plugins
~~~~~~~~~~~~~~~
When writing tests for your plugin code you will need setup and teardown code
@@ -552,23 +521,22 @@
cls.app = paste.fixture.TestApp(wsgiapp)
At this point you should be able to write your own plugins and extensions
-together with their tests. Over time we hope to move more functionality out
-into CKAN extensions.
+together with their tests.
-Ordering of extensions
+Ordering of Extensions
~~~~~~~~~~~~~~~~~~~~~~
.. caution ::
- The order that extensions are initially loaded is **different** to the order that their plugins are run.
+ The order in which extensions are initially loaded is **different** to the order that their plugins are run.
-The order that extensions are initially loaded is as follows:
+The order in which extensions are initially loaded is as follows:
1. System plugins (in setup.py under ``ckan.system_plugins``).
2. In order of the plugins specified in the config file: ``plugins =``.
-3. If more than one module has a plug-in with the same name specified in the config, then all those are loaded, in the order the modules appear in ``sys.path``.
+3. If more than one module has a plugin with the same name specified in the config, then all those are loaded, in the order the modules appear in ``sys.path``.
The order that a plugins are run in, for example the order that IRoutes extensions have their ``before_map`` method run, is alphabetical by the plugin class.
@@ -577,7 +545,7 @@
(This alphabetical ordering is done by ``pyutilib.component.core:ExtensionPoint.extensions()``)
-Plugin API documentation
+Plugin API Documentation
~~~~~~~~~~~~~~~~~~~~~~~~
.. automodule:: ckan.plugins.core
@@ -595,8 +563,8 @@
The Queue Extension
-------------------
-** Note: the queue extension currently isn't working correctly. These docs
-may not work for you**
+.. warning:: The queue extension is currently under development. These docs may not work for you.
+
Certain tasks that CKAN performs lend themselves to the use of a queue system.
Queue systems can be very simple. At their heart is the idea that you have two
@@ -641,7 +609,7 @@
yum install ncurses-devel flex.x86_64 m4.x86_64 openssl-devel.x86_64 unixODBC-devel.x86_64
-Install erlang like this:
+Install Erlang like this:
::
@@ -661,7 +629,7 @@
wget http://www.rabbitmq.com/releases/rabbitmq-server/v2.2.0/rabbitmq-server-2.2.0-1.noarch.rpm
rpm -Uhv --no-deps rabbitmq-server-2.2.0-1.noarch.rpm
-Finally edit the ``/etc/init.d/rabbitmq-server`` script so that it uses the correct path for your erlang install. Change this line
+Finally edit the ``/etc/init.d/rabbitmq-server`` script so that it uses the correct path for your Erlang install. Change this line
::
@@ -691,7 +659,7 @@
Working Directly with Carrot
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-As you learned earlier, CKAN uses carrot with the ``pyamqplib`` backend. Carrot is well documented at this URL:
+As you learned earlier, CKAN uses carrot with the ``pyamqplib`` backend. Carrot is well-documented at this URL:
http://ask.github.com/carrot/introduction.html
@@ -778,7 +746,7 @@
Working with CKANext Queue
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~
Rather than working with carrot publishers and consumers directly,
``ckanext-queue`` provides two useful Python objects to help you:
@@ -799,16 +767,16 @@
``ckanext.queue.worker.Worker``
This is a base class which you can inherit from. You can override its
- ``consume()`` method to asyncronously pull items from the queue to do useful
+ ``consume()`` method to asynchronously pull items from the queue to do useful
things
.. note ::
- To use the queue extension you don't need to implenent any new plugin
+ To use the queue extension you don't need to implement any new plugin
interfaces, you just need to use the ``get_publisher(config).send()`` method and the
``Worker`` class. Of course your own extension might use plugins to hook into
- other parts of CKAN to get information to put or retireve from the queue.
+ other parts of CKAN to get information to put or retrieve from the queue.
The worker implementation runs outside the CKAN server process, interacting
directly with both the AMQP queue and the CKAN API (to get CKAN data). The
@@ -862,7 +830,7 @@
You'll see that once again the consumer picks up the message.
-Writing a worker
+Writing a Worker
````````````````
Now let's replace the consumer with a worker.
@@ -1000,17 +968,3 @@
Now that you know about extensions, plugins and workers you should be able to
extend CKAN in lots of new and interesting ways.
-
-.. Links
-.. -----
-..
-.. Etherpad discussion: http://ckan.okfnpad.org/plugins
-..
-.. Existing plugin implementations (using the old API), comments from are pudo:
-..
-.. - Comments: http://bitbucket.org/pudo/ckandisqus
-.. - Weird stuff: http://bitbucket.org/pudo/ckanextdeliverance
-.. - Shouldn't be a plugin, but typical for localized versions: http://bitbucket.org/pudo/offenedaten
-.. - and probably the largest yet least plugin-ish: http://bitbucket.org/okfn/ckanextiati
-.. - this is what I want to avoid: http://bitbucket.org/okfn/ckanextiati/src/tip/ckanext/iati/authz.py
-
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/doc/post-installation.rst Thu Jul 28 17:18:48 2011 +0100
@@ -0,0 +1,63 @@
+========================
+Post-Installation Setup
+========================
+
+After you have completed installation (from either package or source), follow this section for instructions on setting up an initial user, loading test data, and notes on deploying CKAN.
+
+.. _create-admin-user:
+
+Create an Admin User
+====================
+
+By default, CKAN has a set of locked-down permissions. To begin
+working with it you need to set up a user and some permissions.
+
+First create an admin account from the command line (you must be root, ``sudo -s``):
+
+::
+
+ paster --plugin=ckan user add admin --config=/etc/ckan/std/std.ini
+
+When prompted, enter a password - this is the password you will use to log in to CKAN. In the resulting output, note that you will also get assigned a CKAN API key.
+
+.. note :: This command is your first introduction to some important CKAN concepts. **paster** is the script used to run CKAN commands. **std.ini** is the CKAN config file. You can change options in this file to configure CKAN.
+
+For exploratory purposes, you might was well make the ``admin`` user a
+sysadmin. You obviously wouldn't give most users these rights as they would
+then be able to do anything. You can make the ``admin`` user a sysadmin like
+this:
+
+::
+
+ paster --plugin=ckan sysadmin add admin --config=/etc/ckan/std/std.ini
+
+You can now login to the CKAN frontend with the username ``admin`` and the password you set up.
+
+.. _create-test-data:
+
+Load Test Data
+==============
+
+It can be handy to have some test data to start with. You can get test data like this:
+
+::
+
+ paster --plugin=ckan create-test-data --config=/etc/ckan/std/std.ini
+
+You now have a CKAN instance that you can log in to, with some test data to check everything
+works.
+
+.. _deployment-notes:
+
+Deployment
+==========
+
+You may want to deploy your CKAN instance at this point, to share with others.
+
+If you have installed CKAN from packages, then Apache and WSGI deployment scripts are already configured for you in standard locations.
+
+If you have installed CKAN from source, then the standard production deployment of CKAN is Apache and WSGI, which you will need to configure yourself. For more information, see http://wiki.ckan.net/Deployment
+
+CKAN has been successfully deployed by a variety of other methods including Apache reverse proxy + paster, nginx reverse proxy + paster, and nginx + uwsgi.
+
+You can now proceed to :doc:`theming`.
\ No newline at end of file
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/doc/prepare-extensions.rst Thu Jul 28 17:18:48 2011 +0100
@@ -0,0 +1,31 @@
+=========================
+Prepare to Use Extensions
+=========================
+
+If you are running a package installation of CKAN, before you start using and testing extensions (described in :doc:`extensions`) you need to prepare your system.
+
+Firstly, you'll need to set up and enter a virtual Python environment, as follows:
+
+::
+
+ sudo apt-get install virtualenv pip mercurial
+ virtualenv /home/ubuntu/pyenv
+ . /home/ubuntu/pyenv/bin/activate
+
+Then, you need to install the CKAN source into your virtual environment. You can install CKAN like this:
+
+::
+
+ pip install -e hg+http://bitbucket.org/okfn/ckan#egg=ckan
+
+Your new CKAN developer install will be running on http://localhost:5000/
+
+When you start using extensions, you should install any of the developer versions of the CKAN extensions you want to work on like this (using the appropriate URL):
+
+::
+
+ pip install -e hg+http://bitbucket.org/okfn/<dependency-name>@<version>#egg=<egg-name>
+
+The dependency you've installed will appear in ``/home/ubuntu/pyenv/src/`` where you can work on it.
+
+After working on extensions, you should make sure that your deployment passes the tests, as described in :doc:`test`.
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/doc/test.rst Thu Jul 28 17:18:48 2011 +0100
@@ -0,0 +1,118 @@
+======================
+Testing for Developers
+======================
+
+If you are installing CKAN from source, or developing extensions, then you need to know how to run CKAN tests.
+
+This section describes testing topics for developers, including basic tests, migration testing and testing against PostgreSQL.
+
+.. _basic-tests:
+
+Basic Tests
+-----------
+
+After completing your source installation of CKAN, you should check that tests pass. You should also check this before checking in changes to CKAN code.
+
+Make sure you've created a config file at ``pyenv/ckan/development.ini``. Then activate the Python environment::
+
+ . pyenv/bin/activate
+
+Install nose into your virtual environment::
+
+ pip install --ignore-installed nose
+
+At this point you will need to deactivate and then re-activate your
+virtual environment to ensure that all the scripts point to the correct
+locations:
+
+::
+
+ deactivate
+ . pyenv/bin/activate
+
+Then run the quick development tests::
+
+ cd pyenv/src/ckan
+ nosetests ckan/tests --ckan
+
+You *must* run the tests from the CKAN directory as shown above, otherwise the
+``--ckan`` plugin won't work correctly.
+
+.. warning ::
+
+ By default, the test run is 'quick and dirty' - only good enough as an initial check.
+
+
+Testing against PostgreSQL
+--------------------------
+
+The default way to run tests is defined in ``test.ini`` (which is the default config file for nose - change it with option ``--with-pylons``). This specifies using SQLite and sets ``faster_db_test_hacks``, which are compromises.
+
+::
+
+ cd pyenv/src/ckan
+ nosetests ckan/tests --ckan
+
+Although SQLite is useful for testing a large proportion of CKAN, actually in deployment, CKAN must run with PostgreSQL.
+
+Running the tests against PostgreSQL is slower but more thorough for two reasons:
+
+ 1. You test subtleties of PostgreSQL
+ 2. CKAN's default search relies on PostgreSQL's custom full-text search, so these (100 or so) tests are skipped when running against SQLite.
+
+So when making changes to anything involved with search or closely related to the database, it is wise to test against PostgreSQL.
+
+To test against PostgreSQL:
+
+ 1. Edit your local ``development.ini`` to specify a PostgreSQL database with the ``sqlalchemy.url`` parameter.
+ 2. Tell nose to use ``test-core.ini`` (which imports settings from ``development.ini``)
+
+::
+
+ nosetests ckan/tests --ckan --with-pylons=test-core.ini
+
+The test suite takes a long time to run against standard PostgreSQL (approx. 15 minutes, or close to an hour on Ubuntu/10.04 Lucid).
+
+This can be improved to between 5 and 15 minutes by running PostgreSQL in memory and turning off durability, as described `in the PostgreSQL documentation <http://www.postgresql.org/docs/9.0/static/non-durability.html>`_.
+
+.. _migrationtesting:
+
+Migration Testing
+-----------------
+
+If your changes require a model change, you'll need to write a migration script. To ensure this is tested as well, you should instead run the tests this way::
+
+ nosetests ckan/tests --ckan --ckan-migrate --with-pylons=test-core.ini
+
+By default, tests are run using the model defined in ``ckan/model``, but by using the ``--ckan-migrate`` option the tests will run using a database that has been created using the migration scripts, which is the way the database is created and upgraded in production. These tests are the most thorough and will take around 20 minutes.
+
+.. caution ::
+
+ Ordinarily, you should set ``development.ini`` to specify a PostgreSQL database
+ so these also get used when running ``test-core.ini``, since ``test-core.ini``
+ inherits from ``development.ini``. If you were to change the ``sqlalchemy.url``
+ option in your ``development.ini`` file to use SQLite, the command above would
+ actually test SQLite rather than PostgreSQL, so always check the setting in
+ ``development.ini`` to ensure you are running the full tests.
+
+.. warning ::
+
+ A common error when wanting to run tests against a particular database is to change ``sqlalchemy.url`` in ``test.ini`` or ``test-core.ini``. The problem is that these are versioned files and people have checked in these by mistake, creating problems for other developers and the CKAN buildbot. This is easily avoided by only changing ``sqlalchemy.url`` in your local ``development.ini`` and testing ``--with-pylons=test-core.ini``.
+
+Common problems running tests
+-----------------------------
+
+* `nose.config.ConfigError: Error reading config file 'setup.cfg': no such option 'with-pylons'`
+
+ This error can result when you run nosetests for two reasons:
+
+ 1. Pylons nose plugin failed to run. If this is the case, then within a couple of lines of running `nosetests` you'll see this warning: `Unable to load plugin pylons` followed by an error message. Fix the error here first.
+
+ 2. The Python module 'Pylons' is not installed into you Python environment. Confirm this with::
+
+ python -c "import pylons"
+
+* `OperationalError: (OperationalError) no such function: plainto_tsquery ...`
+
+ This error usually results from running a test which involves search functionality, which requires using a PostgreSQL database, but another (such as SQLite) is configured. The particular test is either missing a `@search_related` decorator or there is a mixup with the test configuration files leading to the wrong database being used.
+
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/doc/theming.rst Thu Jul 28 17:18:48 2011 +0100
@@ -0,0 +1,101 @@
+=============
+Customization
+=============
+
+After installing CKAN, the next step is probably to re-theme the site with your own logo, site name, and CSS.
+
+Site Name and Description
+=========================
+
+You can change the name and logo of the site by setting options in the CKAN config file.
+
+This is the file called ``std.ini`` that you first encountered in :ref:`create-admin-user`. It is usually located at ``/etc/ckan/std/std.ini``.
+
+Open this file, and change the following options::
+
+ ckan.site_title = My CKAN Site
+ ckan.site_description = The easy way to get, use and share data
+
+After you've edited these options, restart Apache::
+
+ sudo /etc/init.d/apache2 restart
+
+Refresh your home page (clearing the cache if necessary) and you should see your new title and description.
+
+More Advanced Customization
+===========================
+
+If you want to make broader changes to the look and feel of your CKAN site, we offer ways to add custom CSS and over-ride the default CKAN templates.
+
+Custom CSS and Templates
+------------------------
+
+You can add custom CSS, templates, scripts, images etc to your site using the ``extra_template_paths`` and ``extra_public_paths`` options in the CKAN config file::
+
+ extra_template_paths = %(here)s/my-templates
+ extra_public_paths = %(here)s/my-public
+
+All contents of the public directory is mounted directly into the URL space of the site (taking precedence over existing files of the same name).
+
+Furthermore, you can supply multiple public directories, which will be searched in order.
+
+For example, if you set the following option in the CKAN config file::
+
+ extra_public_paths = /path/to/mypublicdir
+
+And then add a file called ``myhtmlfile.html`` in ``/path/to/mypublicdir``, the file would appear on http://yourckan.org/ at ``http://yourckan.org/myhtmlfile.html``.
+
+If you create a file with the same path as one in the main CKAN public directory, your file will override the default CKAN file. For example, if you add ``mypublicdir/css/ckan.css``, then ``http://yourckan.org/css/ckan.css`` will be your file.
+
+Adding a New Logo
+^^^^^^^^^^^^^^^^^
+
+One example is introducing your own logo, which you can do with a new file and a CKAN config option.
+
+Add a logo file at ``mypublicdir/images/mylogo.png``, then set options in the CKAN config file (``/etc/ckan/std/std.ini``) as follows::
+
+ extra_public_paths = /path/to/mypublicdir
+ ckan.site_logo = /images/mylogo.png
+
+
+Adding a New Stylesheet
+^^^^^^^^^^^^^^^^^^^^^^^
+
+Lots of visual changes can be made simply by changing the stylesheet.
+
+The easiest way to override the default CKAN style is to create one or more custom CSS files and load them in the ``layout.html`` template.
+
+Use the 'public' directory as described in the previous section, then add a new file at ``mypublicdir/css/mycss.css``.
+
+Next, copy the ``layout.html`` template and add a reference to the new CSS file. Here is an example of the edited ``layout.html`` template::
+
+ <html xmlns="http://www.w3.org/1999/xhtml"
+ xmlns:i18n="http://genshi.edgewall.org/i18n"
+ xmlns:py="http://genshi.edgewall.org/"
+ xmlns:xi="http://www.w3.org/2001/XInclude"
+ py:strip="">
+ <py:def function="optional_head">
+ <link rel="stylesheet" href="${g.site_url}/css/mycss.css" />
+ </py:def>
+ <xi:include href="layout_base.html" />
+ </html>
+
+Retheming the Site with Templates
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Template files are used as source templates for rendered pages on the site. These templates are just an HTML page but with variables, such as the page title set by each page: ``${page_title}``.
+
+To over-ride a template, set the ``extra_template_paths`` directory as described above, then copy and rewrite the template file you wish to over-ride.
+
+Commonly modified templates are:
+
+ * ``layout_base.html`` - base customizationlayout template for whole site
+ * ``layout.html`` - empty by default
+ * ``home/index.html`` - the home page of the site
+ * ``home/about.html`` - the about page
+
+If you are re-theming the site, we recommend you over-ride ``layout.html``, which is empty but inherits from ``layout_base.html``. This will mean you can upgrade the site more easily in the future.
+
+.. note::
+
+ For more information on the syntax of the CKAN templates, refer to the `Genshi documentation <http://genshi.edgewall.org/wiki/Documentation>`_.
\ No newline at end of file
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/doc/upgrade.rst Thu Jul 28 17:18:48 2011 +0100
@@ -0,0 +1,25 @@
+============
+Upgrade CKAN
+============
+
+Any time you want to upgrade CKAN to a newer version, first back up your database:
+
+::
+
+ paster --plugin=ckan db dump demo_ckan_backup.pg_dump --config=demo.ckan.net.ini
+
+Then run this as root:
+
+::
+
+ apt-get update
+ apt-get dist-upgrade
+ ckan-std-install
+
+You need to use ``dist-upgrade`` rather than ``upgrade`` because it is possible
+more recent versions of CKAN will add new dependencies rather than just
+updating existing ones, and apt will only install new dependencies if you use
+``dist-upgrade``.
+
+The ``ckan-std-install`` will also upgrade your existing CKAN
+instance, keeping a backup of your old CKAN config file.
--- a/doc/vm.rst Thu Jul 28 16:45:02 2011 +0100
+++ /dev/null Thu Jan 01 00:00:00 1970 +0000
@@ -1,247 +0,0 @@
-Testing CKAN in a VM
-++++++++++++++++++++
-
-.. WARNING::
- This document is still under development, use only if you are a member
- of the CKAN team who wishes to be an early adopter and are interested in
- experimenting with virtual machines.
-
-If you aren't running Lucid, you may need to test in a VM.
-
-Repository cache
-================
-
-First set up a cache of the repositories so that you don't need to fetch packages each time you
-build a VM:
-
-::
-
- $ sudo apt-get install apt-proxy
- $ sudo apt-get install uml-utilities
-
-Now find your local host's ethernet port name, IP address, netmask::
-
- $ ifconfig
- eth0 Link encap:Ethernet HWaddr 1c:6f:65:8a:0a:d8
- inet addr:50.56.101.208 Bcast:50.56.101.255 Mask:255.255.255.0
-
-i.e.::
-
- ethernet port name: eth0
- IP address: 50.56.101.208
- netmask: 255.255.255.0
-
-These instructions assume these values, so replace them with yours.
-
-Now edit your apt-get config::
-
- $ sudo vim /etc/apt-proxy/apt-proxy.conf
-
-And set these two lines. The first is your host's IP address::
-
- ;; Server IP to listen on
- address = 50.56.101.208
-
- ;; Server port to listen on
- port = 9998
-
-Now restart it::
-
- $ sudo /etc/init.d/apt-proxy restart
-
-Once this is complete, your (empty) proxy is ready for use on
-http://<host-machine-ip>:9998 and will find Ubuntu repository under ``/ubuntu``.
-
-See also:
-
-* https://help.ubuntu.com/community/AptProxy
-
-
-VM creation
-===========
-
-Install the VM program::
-
- sudo apt-get install python-vm-builder
- sudo apt-get install kvm-pxe
-
-Now create a directory ``~/Vms`` for your virtual machines.
-
-::
-
- mkdir ~/Vms
-
-
-We'll use manual bridging and networking rather than relying on the magic provided by ``libvirt``. Out virtual network for the VMs will be 192.168.100.xxx. You can use any number from 2-253 inclusive for the last bit of the IP. This first machine will have the IP address 192.168.100.2. Each virtual machine afterwards must have a unique IP address.
-
-First set some variables (use your own IP address for HOST_IP):
-
-::
-
- export THIS_IP="4"
- export HOST_IP="50.56.101.208"
-
-Check your CPU has native VM support::
-
- $ kvm-ok
-
-If your CPU doesn't support KVM extensions:
-
-1. change ``kvm`` to ``qemu`` in the vmbuilder step coming up next
-2. use ``/usr/bin/qemu`` instead of ``/usr/bin/kvm``
-
-Now create the VM:
-
-::
-
- cd ${HOME}/Vms
- export BASE_IP="192.168.100"
- sudo vmbuilder kvm ubuntu \
- --mem 512 \
- --cpus 6 \
- --domain ckan_${THIS_IP} \
- --dest ckan_${THIS_IP} \
- --flavour virtual \
- --suite lucid \
- --arch amd64 \
- --hostname ckan${THIS_IP} \
- --user ubuntu \
- --pass ubuntu \
- --rootpass ubuntu \
- --debug -v \
- --ip ${BASE_IP}.${THIS_IP} \
- --mask 255.255.255.0 \
- --net ${BASE_IP}.0 \
- --bcast ${BASE_IP}.255 \
- --gw ${BASE_IP}.254 \
- --dns ${BASE_IP}.254 \
- --proxy http://${HOST_IP}:9998/ubuntu \
- --components main,universe \
- --addpkg vim \
- --addpkg openssh-server \
- --addpkg wget
-
-This assumes you already have an apt mirror set up on port 9998 as described
-above and that you are putting everything in ``~/Vms``.
-
-Now for the networking.
-
-First check you have forwarding enabled on the host:
-
-::
-
- sudo -s
- echo "1" > /proc/sys/net/ipv4/ip_forward
- exit
-
-Now save this as ``~/Vms/start.sh``:
-
-::
-
- #!/bin/bash
-
- if [ "X$1" = "X" ] || [ "X$2" = "X" ] || [ "X$3" = "X" ] || [ "X$4" = "X" ] || [ "X$5" = "X" ]; then
- echo "ERROR: call this script with network device name, tunnel name, amount of memory, number of CPUs and path to the image e.g."
- echo " $0 eth0 qtap0 512M 4 /home/Vms/ckan_2/tmpKfAdeU.qcow2 [extra args to KVM]"
- exit 1
- fi
-
- NETWORK_DEVICE=$1
- TUNNEL=$2
- MEM=$3
- CPUS=$4
- IMAGE=$5
- EXTRA=$6
- MACADDR="52:54:$(dd if=/dev/urandom count=1 2>/dev/null | md5sum | sed 's/^\(..\)\(..\)\(..\)\(..\).*$/\1:\2:\3:\4/')";
-
- echo "Creating bridge..."
- sudo iptables -t nat -A POSTROUTING -o ${NETWORK_DEVICE} -j MASQUERADE
- sudo brctl addbr br0
- sudo ifconfig br0 192.168.100.254 netmask 255.255.255.0 up
- echo "done."
- echo "Creating tunnel..."
- sudo modprobe tun
- sudo tunctl -b -u root -t ${TUNNEL}
- sudo brctl addif br0 ${TUNNEL}
- sudo ifconfig ${TUNNEL} up 0.0.0.0 promisc
- echo "done."
- echo "Starting VM ${IMAGE} on ${TUNNEL} via ${NETWORK_DEVICE} with MAC ${MACADDR}..."
- sudo /usr/bin/kvm -M pc-0.12 -enable-kvm -m ${MEM} -smp ${CPUS} -name dev -monitor pty -boot c -drive file=${IMAGE},if=ide,index=0,boot=on -net nic,macaddr=${MACADDR} -net tap,ifname=${TUNNEL},script=no,downscript=no -serial none -parallel none -usb ${EXTRA}
-
-
-Make it executable:
-
-::
-
- chmod a+x ~/Vms/start.sh
-
-Now you can start it along the lines of this:
-
-::
-
- ./start.sh eth0 qtap0 512M 1 /home/james/Vms/ckan_4/tmpuNIv2h.qcow2
-
-Now login:
-
-::
-
- ssh ubuntu@${BASE_IP}.${THIS_IP}
-
-Once in you'll need some more configuration.
-
-Edit ``/etc/resolv.conf`` to contain just this (the Google DNS servers, handy
-to used a fixed IP so that you don't have to update your ``resolve.conf`` each
-time you move to a different network):
-
-::
-
- nameserver 8.8.8.8
-
-Then change ``/etc/apt/apt.conf`` to comment out the proxy line, you may as
-well get updates directly now.
-
-Finally, run this (swapping the repository name for the one you want to test)
-to allow yourself to install CKAN:
-
-::
-
- sudo apt-get install wget
- wget -qO- http://apt-alpha.ckan.org/packages.okfn.key | sudo apt-key add -
- echo "deb http://apt-alpha.ckan.org/debian lucid universe" | sudo tee /etc/apt/sources.list.d/okfn.list
- sudo apt-get update
-
-Now that you have the repo added you can install and test CKAN as normal.
-
-Here's how mine look:
-
-::
-
- ubuntu at ckan4:~$ cat /etc/network/interfaces
- # This file describes the network interfaces available on your system
- # and how to activate them. For more information, see interfaces(5).
-
- # The loopback network interface
- auto lo
- iface lo inet loopback
-
- # The primary network interface
- auto eth0
- iface eth0 inet static
- address 192.168.100.4
- netmask 255.255.255.0
- network 192.168.100.0
- broadcast 192.168.100.255
- gateway 192.168.100.254
- dns 192.168.100.254
- ubuntu at ckan4:~$ cat /etc/resolv.conf
- nameserver 8.8.8.8
-
-
-VNC access
-==========
-
-To add ability to VNC into the VM as it runs (useful for debug), add this line to the start.sh command::
-
- -vnc :1
-
-Then you can vnc to it when running using a VNC Client.
\ No newline at end of file
http://bitbucket.org/okfn/ckan/changeset/32bc33aad950/
changeset: 32bc33aad950
user: dread
date: 2011-07-28 18:20:40
summary: [merge] from release-v1.4.3.
affected #: 72 files (90.3 KB)
--- a/README.txt Thu Jul 28 16:26:50 2011 +0100
+++ b/README.txt Thu Jul 28 17:20:40 2011 +0100
@@ -1,443 +1,10 @@
README
++++++
-Introduction
-============
+Welcome to CKAN.
-Comprehensive Knowledge Archive Network (CKAN) Software.
+For installation instructions, see http://docs.ckan.org/install-from-source.html
-See :mod:`ckan.__long_description__` for more information.
+For background information and developer notes, see http://wiki.ckan.net
-
-Developer Installation
-======================
-
-These are instructions to get developing with CKAN. Instructions for deploying
-CKAN to a server are at: :doc:`deployment` (doc/deployment.rst).
-
-Before you start it may be worth checking CKAN has passed the auto build and
-tests. See: http://buildbot.okfn.org/waterfall
-
-
-1. Ensure these packages are installed:
-
- ===================== ===============================================
- Package Description
- ===================== ===============================================
- mercurial Source control
- python-dev Python interpreter v2.5 - v2.7 and dev headers
- postgresql PostgreSQL database
- libpq-dev PostgreSQL library
- python-psycopg2 PostgreSQL python module
- libxml2-dev XML library development files
- libxslt-dev XSLT library development files
- python-virtualenv Python virtual environments
- wget Command line tool for downloading from the web
- build-essential Tools for building source code
- git-core Git source control (for getting MarkupSafe src)
- subversion Subversion source control (for pyutilib)
- ===================== ===============================================
-
- For Ubuntu you can install these like so:
-
- ::
-
- sudo apt-get install build-essential libxml2-dev libxslt-dev
- sudo apt-get install wget mercurial postgresql libpq-dev git-core
- sudo apt-get install python-dev python-psycopg2 python-virtualenv
- sudo apt-get install subversion
-
-2. Create a python virtual environment
-
- In your home directory run the command below. It is currently important to
- call your virtual environment ``pyenv`` so that the automated deployment tools
- work correctly.
-
- ::
-
- cd ~
- virtualenv pyenv
-
- .. tip ::
-
- If you don't have a ``python-virtualenv`` package in your distribution
- you can get a ``virtualenv.py`` script from within the
- `virtualenv source distribution <http://pypi.python.org/pypi/virtualenv/>`_
- and then run ``python virtualenv.py pyenv`` instead.
-
- To help with automatically installing CKAN dependencies we use a tool
- called ``pip``. Make sure you have activated your environment (see step 3)
- and then install it from an activated shell like this:
-
- ::
-
- easy_install pip
-
-3. Activate your virtual environment
-
- To work with CKAN it is best to adjust your shell settings so that your
- shell uses the virtual environment you just created. You can do this like
- so:
-
- ::
-
- . pyenv/bin/activate
-
- When your shell is activated you will see the prompt change to something
- like this:
-
- ::
-
- (pyenv)[ckan at host ~/]$
-
- An activated shell looks in your virtual environment first when choosing
- which commands to run. If you enter ``python`` now it will actually
- run ``~/pyenv/bin/python`` which is what you want.
-
-4. Install CKAN code and required Python packages into the new environment
-
- First you'll need to install CKAN. For the latest version run:
-
- ::
-
- pip install --ignore-installed -e hg+http://bitbucket.org/okfn/ckan#egg=ckan
-
- CKAN has dependent packages it requires you install. They are listed in the three pyenv/src/ckan/requires files.
-
- If you are running Ubuntu Lucid then it is recommended you take advantage of the Lucid packages to cover the dependencies listed in lucid_present.txt. In this case you should install the Lucid present packages::
-
- sudo apt-get install python-psycopg2 python-lxml python-sphinx
- sudo apt-get install python-pylons python-formalchemy python-repoze.who
- sudo apt-get install python-repoze.who-plugins python-tempita python-zope.interface
-
- And the remaining (non-Lucid) packages are installed like this::
-
- pip install --ignore-installed -r pyenv/src/ckan/requires/lucid_missing.txt -r pyenv/src/ckan/requires/lucid_conflict.txt
-
- If you are not using Ubuntu Lucid then should install all three sets of dependencies with this single command::
-
- pip install --ignore-installed -r pyenv/src/ckan/requires/lucid_present.txt -r pyenv/src/ckan/requires/lucid_missing.txt -r pyenv/src/ckan/requires/lucid_conflict.txt
-
- This will take a **long** time. Particularly the install of the ``lxml``
- package.
-
- The ``--ignore-installed`` option ensures ``pip`` installs software into
- this virtual environment even if it is already present on the system. Because of this, to ensure you get a mutually agreeable version of WebOb installed, you shouldn't install each requirements file on its own with this option.
-
- At this point you will need to deactivate and then re-activate your
- virtual environment to ensure that all the scripts point to the correct
- locations:
-
- ::
-
- deactivate
- . pyenv/bin/activate
-
-5. Setup a PostgreSQL database
-
- List existing databases:
-
- ::
-
- psql -l
-
- It is advisable to ensure that the encoding of databases is 'UTF8', or
- internationalisation may be a problem. Since changing the encoding of PostgreSQL
- may mean deleting existing databases, it is suggested that this is fixed before
- continuing with the CKAN install.
-
- Next you'll need to create a database user if one doesn't already exist.
-
- .. tip ::
-
- If you choose a database name, user or password which are different from those
- suggested below then you'll need to update the configuration file you'll create in
- the next step.
-
- Here we choose ``ckantest`` as the database and ``ckanuser`` as the user:
-
- ::
-
- sudo -u postgres createuser -S -D -R -P ckantest
-
- It should prompt you for a new password for the CKAN data in the database.
- It is suggested you enter ``pass`` for the password.
-
- Now create the database, which we'll call ``ckantest`` (the last argument):
-
- ::
-
- sudo -u postgres createdb -O ckantest ckantest
-
-6. Create a CKAN config file
-
- Make sure you are in an activated environment (see step 3) so that Python
- Paste and other modules are put on the python path (your command prompt will
- start with ``(pyenv)`` if you have) then change into the ``ckan`` directory
- which will have been created when you installed CKAN in step 4 and create the
- config file ``development.ini`` using Paste:
-
- ::
-
- cd pyenv/src/ckan
- paster make-config ckan development.ini
-
- You can give your config file a different name but the tests will expect you
- to have used ``development.ini`` so it is strongly recommended you use this
- name, at least to start with.
-
- If you used a different database name or password when creating the database
- in step 5 you'll need to now edit ``development.ini`` and change the
- ``sqlalchemy.url`` line, filling in the database name, user and password you used.
-
- ::
-
- sqlalchemy.url = postgresql://ckantest:pass@localhost/ckantest
-
- Other configuration, such as setting the language of the site or editing the
- visual theme are described in :doc:`configuration` (doc/configuration.rst)
-
- .. note ::
-
- If the paster command gives you ``ImportError: cannot import name UnicodeMultiDict`` then you have gotten a version of WebOb that is too new for our Pylons. This usually occurs if you install Remove WebOb and reinstall version 1.0.8.
-
- .. caution ::
-
- Advanced users: If you are using CKAN's fab file capability you currently need to create
- your config file as ``pyenv/ckan.net.ini`` so you will probably have
- ignored the advice about creating a ``development.ini`` file in the
- ``pyenv/src/ckan`` directory. This is fine but CKAN probably won't be
- able to find your ``who.ini`` file. To fix this edit ``pyenv/ckan.net.ini``,
- search for the line ``who.config_file = %(here)s/who.ini`` and change it
- to ``who.config_file = who.ini``.
-
- We are moving to a new deployment system where this incompatibility
- will be fixed.
-
-7. Create database tables
-
- Now that you have a configuration file that has the correct settings for
- your database, you'll need to create the tables. Make sure you are still in an
- activated environment with ``(pyenv)`` at the front of the command prompt and
- then from the ``pyenv/src/ckan`` directory run this command:
-
- ::
-
- paster db init
-
- You should see ``Initialising DB: SUCCESS``. If you are not in the
- ``pyenv/src/ckan`` directory or you don't have an activated shell, the command
- will not work.
-
- If the command prompts for a password it is likely you haven't set up the
- database configuration correctly in step 6.
-
-8. Create the cache directory
-
- You need to create the Pylon's cache directory specified by 'cache_dir'
- in the config file.
-
- (from the ``pyenv/src/ckan`` directory):
-
- ::
-
- mkdir data
-
-
-9. Run the CKAN webserver
-
- NB If you've started a new shell, you'll have to activate the environment
- again first - see step 3.
-
- (from the pyenv/src/ckan directory):
-
- ::
-
- paster serve development.ini
-
-10. Point your web browser at: http://127.0.0.1:5000/
-
- The CKAN homepage should load without problem.
-
-If you ever want to upgrade to a more recent version of CKAN, read the
-``UPGRADE.txt`` file in ``pyenv/src/ckan/``.
-
-Test
-====
-
-Setting up to test
-------------------
-
-Make sure you've created a config file: ``pyenv/ckan/development.ini``
-
-Ensure you have activated the environment::
-
- . pyenv/bin/activate
-
-
-Install nose into your virtual environment if you haven't already::
-
- pip install --ignore-installed nose
-
-At this point you will need to deactivate and then re-activate your
-virtual environment to ensure that all the scripts point to the correct
-locations:
-
-::
-
- deactivate
- . pyenv/bin/activate
-
-
-Running developer tests
------------------------
-
-Here's how you start the quick development tests::
-
- cd pyenv/src/ckan
- nosetests ckan/tests --ckan
-
-You *must* run the tests from the CKAN directory as shown above, otherwise the
-``--ckan`` plugin won't work correctly.
-
-.. caution ::
-
- By default, the test run is 'quick and dirty' - only good enough as a check
- before committing code. See the next section for improved ways of running tests.
-
-
-Test configurations
--------------------
-
-The default way to run tests is defined in test.ini (which is the default config file for nose - change it with option "--with-pylons"). This specifies to use Sqlite and sets faster_db_test_hacks, which are compromises.
-
-::
-
- cd pyenv/src/ckan
- nosetests ckan/tests --ckan
-
-Although Sqlite is useful for testing a large proportion of CKAN, actually in deployment, CKAN must run with PostgreSQL. Running the tests against PosgreSQL is slower but more thorough for two reasons:
- 1. You test subtleties of PostgreSQL
- 2. CKAN's default search relies on PostgreSQL's custom Full Text Search, so these (100 or so) tests are skipped when running against Sqlite.
-
-So when making changes to anything involved with search or closely related to the database, it is wise to test against PostgreSQL.
-
-To test against PosgreSQL:
- 1. Edit your local development.ini to specify a PostgreSQL database with the `sqlalchemy.url` parameter.
- 2. Tell nose to use test-core.ini (which imports settings from development.ini)
-
-::
-
- nosetests ckan/tests --ckan --with-pylons=test-core.ini
-
-The test suite takes a long time to run against standard PostgreSQL (approx. 15 minutes, or close to an hour on Ubuntu/10.04 Lucid).
-
-This can be improved to between 5 and 15 minutes by running PostgreSQL in memory and turning off durability, as described at <http://www.postgresql.org/docs/9.0/static/non-durability.html>.
-
-.. _migrationtesting:
-
-If your changes require a model change, you'll need to write a migration script. To ensure this is tested as well, you should instead run the tests this way::
-
- nosetests ckan/tests --ckan --ckan-migrate --with-pylons=test-core.ini
-
-By default, tests are run using the model defined in ckan/model, but by using the ``--ckan-migrate`` option the tests will run using a database that has been created using the migration scripts, which is the way the database is created and upgraded in production. These tests are the most thorough and will take around 20 minutes.
-
-.. caution ::
-
- Ordinarily, you should set ``development.ini`` to specify a PostgreSQL database
- so these also get used when running ``test-core.ini``, since ``test-core.ini``
- inherits from ``development.ini``. If you were to change the ``sqlalchemy.url``
- option in your ``development.ini`` file to use SQLite, the command above would
- actually test SQLite rather than PostgreSQL so always check the setting in
- ``development.ini`` to ensure you are running the full tests.
-
-.. note ::
-
- A common error when wanting to run tests against a particular database is to change the sqlalchemy.url in test.ini or test-core.ini. The problem is that these are versioned files and people have checked in these by mistake, creating problems for all other developers and the buildbot. This is easily avoided by only changing the sqlalchemy.url in your local development.ini and testing --with-pylons=test-core.ini.
-
-Common problems running tests
------------------------------
-
-* `nose.config.ConfigError: Error reading config file 'setup.cfg': no such option 'with-pylons'`
-
- This error can result when you run nosetests for two reasons:
-
- 1. Pylons nose plugin failed to run. If this is the case, then within a couple of lines of running `nosetests` you'll see this warning: `Unable to load plugin pylons` followed by an error message. Fix the error here first.
-
- 2. The Python module 'Pylons' is not installed into you Python environment. Confirm this with::
-
- python -c "import pylons"
-
-* `OperationalError: (OperationalError) no such function: plainto_tsquery ...`
-
- This error usually results from running a test which involves search functionality, which requires using a PostgreSQL database, but another (such as SQLite) is configured. The particular test is either missing a `@search_related` decorator or there is a mixup with the test configuration files leading to the wrong database being used.
-
-
-Testing extensions
-------------------
-
-CKAN extensions ordinarily have their own test.ini that refers to the ckan test.ini, so you can run them in exactly the same way. For example::
-
- cd ckanext-dgu
- nosetests ckanext/dgu/tests --ckan
- nosetests ckanext/dgu/tests --ckan --with-pylons=test-core.ini
-
-
-Development
-===========
-
-CKAN is an open source project and contributions are welcome!
-
-There are a number of stakeholders in the direction of the project, so we discuss large changes and new features on the ckan-discuss list: http://lists.okfn.org/mailman/listinfo/ckan-discuss
-
-New developers should aquaint themselves with the documentation (see below). Proposed changes should be made on a personal CKAN fork (on BitBucket for example). Request merging with the mainline via the ckan-discuss list.
-
-We have policies for check-ins that ensure the build doesn't break etc. on http://ckan.org/#ProjectProcessesandPolicies which should be followed unless someone builds concensus to change it.
-
-
-Documentation
-=============
-
-The home page for the CKAN project is: http://ckan.org
-
-This README file is part of the Developer Documentation, viewable at:
-http://packages.python.org/ckan/ and stored in the CKAN
-repo at ``ckan/doc``.
-
-The Developer Docs are built using `Sphinx <http://sphinx.pocoo.org/>`_:
-
- python setup.py build_sphinx
-
-(An admin might upload the resulting html to packages.python.org/ckan/ by doing: `easy_install sphinx-pypi-upload` and `python setup.py upload_sphinx`)
-
-(The docs are also uploaded via dav to http://knowledgeforge.net/ckan/doc/ckan/for backwards compatability).
-
-
-Contributors
-============
-
- * Rufus Pollock <rufus [at] rufuspollock [dot] org>
- * David Read
- * John Bywater
- * Nick Stenning (css and js)
-
-Also especial thanks to the following projects without whom this would not have
-been possible:
-
- * CKAN logo: "angry hamster" http://www.maedelmaedel.com/ and
- http://www.villainous.biz/
- * famfamfam.com for silk icons <http://www.famfamfam.com/lab/icons/silk/>
- * Pylons: <http://pylonshq.com/>
- * Python: <http://www.python.org>
-
-
-Copying and License
-===================
-
-This material is copyright (c) 2006-2011 Open Knowledge Foundation.
-
-It is open and licensed under the GNU Affero General Public License (AGPL) v3.0
-whose full text may be found at:
-
-<http://www.fsf.org/licensing/licenses/agpl-3.0.html>
-
+For help during installation, contact ckan-dev at lists.okfn.org
\ No newline at end of file
--- a/ckan/logic/action/get.py Thu Jul 28 16:26:50 2011 +0100
+++ b/ckan/logic/action/get.py Thu Jul 28 17:20:40 2011 +0100
@@ -2,7 +2,6 @@
from sqlalchemy import or_, func, desc
from ckan.logic import NotFound, check_access
-from ckan.model import Session
from ckan.plugins import (PluginImplementations,
IGroupController,
IPackageController)
@@ -31,9 +30,7 @@
api = context.get("api_version", '1')
ref_package_by = 'id' if api == '2' else 'name'
- query = Session.query(model.PackageRevision)
- query = query.filter(model.PackageRevision.state=='active')
- query = query.filter(model.PackageRevision.current==True)
+ query = ckan.authz.Authorizer().authorized_query(user, model.Package)
packages = query.all()
return [getattr(p, ref_package_by) for p in packages]
@@ -42,7 +39,7 @@
user = context["user"]
limit = data_dict.get("limit")
- q = Session.query(model.PackageRevision)
+ q = ckan.authz.Authorizer().authorized_query(user, model.PackageRevision)
q = q.filter(model.PackageRevision.state=='active')
q = q.filter(model.PackageRevision.current==True)
--- a/doc/README.rst Thu Jul 28 16:26:50 2011 +0100
+++ /dev/null Thu Jan 01 00:00:00 1970 +0000
@@ -1,1 +0,0 @@
-../README.txt
\ No newline at end of file
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/doc/_templates/layout.html Thu Jul 28 17:20:40 2011 +0100
@@ -0,0 +1,20 @@
+{% extends '!layout.html' %}
+
+{% block footer %}
+ <div class="footer">
+ {% if hasdoc('copyright') %}
+ {% trans path=pathto('copyright'), copyright=copyright|e %}© <a href="{{ path }}">Copyright</a> {{ copyright }}.{% endtrans %}
+ {% else %}
+ {% trans copyright=copyright|e %}© Copyright {{ copyright }}.{% endtrans %}
+ {% endif %}
+ {% if last_updated %}
+ {% trans last_updated=last_updated|e %}Last updated on {{ last_updated }}.{% endtrans %}
+ {% endif %}
+ {% if show_sphinx %}
+ {% trans sphinx_version=sphinx_version|e %}Created using <a href="http://sphinx.pocoo.org/">Sphinx</a> {{ sphinx_version }}.{% endtrans %}
+ {% endif %}
+ <br/>
+ Please contact the <a href="http://lists.okfn.org/mailman/listinfo/ckan-dev">ckan-dev mailing list</a>
+ with any questions.
+ </div>
+{% endblock %}
\ No newline at end of file
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/doc/about.rst Thu Jul 28 17:20:40 2011 +0100
@@ -0,0 +1,33 @@
+About CKAN
+===========
+
+Development
+-----------
+
+CKAN is an open source project and contributions are welcome!
+
+We discuss large changes and new features on the `ckan-discuss <http://lists.okfn.org/mailman/listinfo/ckan-discuss>`_ mailing list, and technical issues on the `ckan-dev <http://lists.okfn.org/mailman/listinfo/ckan-dev>`_ mailing list. Please join us there.
+
+You can find developer resources and links to our ticketing system on the `CKAN wiki <http://wiki.ckan.net/Main_Page>`_.
+
+Acknowledgements
+----------------
+
+Thanks to the following projects, without which CKAN would not have
+been possible:
+
+ * `Python <http://www.python.org>`_
+ * `Pylons <http://pylonshq.com/>`_
+ * CKAN logo: "angry hamster" by http://www.maedelmaedel.com/ and
+ http://www.villainous.biz/
+ * `FamFamFam silk icons <http://www.famfamfam.com/lab/icons/silk/>`_
+
+Copying and Licence
+-------------------
+
+This material is copyright (c) 2006-2011 Open Knowledge Foundation.
+
+It is open and licensed under the GNU Affero General Public License (AGPL) v3.0
+whose full text may be found at:
+
+<http://www.fsf.org/licensing/licenses/agpl-3.0.html>
\ No newline at end of file
--- a/doc/api.rst Thu Jul 28 16:26:50 2011 +0100
+++ b/doc/api.rst Thu Jul 28 17:20:40 2011 +0100
@@ -16,8 +16,11 @@
.. #|version_3_doc| replace:: :doc:`api/version3`
-|site| |api|
-======================
+.. index:: API
+
+===========================
+Reference: The |site| |api|
+===========================
.. toctree::
:hidden:
@@ -25,6 +28,8 @@
The |site| |api| presents the catalog of metadata behind |site|.
+.. note:: This section contains general information about the |site| API. If you are looking for API documentation, please see |version_2_doc|.
+
.. include:: api/overview.rst.inc
.. include:: api/versions.rst.inc
--- a/doc/api/clients.rst.inc Thu Jul 28 16:26:50 2011 +0100
+++ b/doc/api/clients.rst.inc Thu Jul 28 17:20:40 2011 +0100
@@ -3,5 +3,5 @@
There are also some code modules (Python, PHP, Drupal, Perl etc.) that provide
convenient wrappers around much of the CKAN API. For full details of these,
-please consult: http://trac.ckan.org/wiki/API
+please consult http://wiki.ckan.net/API
--- a/doc/api/purpose.rst.inc Thu Jul 28 16:26:50 2011 +0100
+++ b/doc/api/purpose.rst.inc Thu Jul 28 17:20:40 2011 +0100
@@ -1,2 +1,1 @@
-This document describes |version| of the |main_doc|.
-
+This document describes |version| of the |main_doc|.
\ No newline at end of file
--- a/doc/api/version1.rst Thu Jul 28 16:26:50 2011 +0100
+++ b/doc/api/version1.rst Thu Jul 28 17:20:40 2011 +0100
@@ -3,6 +3,7 @@
.. include:: title.rst.inc
.. include:: purpose.rst.inc
+.. include:: warning.rst.inc
.. include:: overview.rst.inc
.. include:: interfaces1.rst.inc
.. include:: location.rst.inc
--- a/doc/api/version2.rst Thu Jul 28 16:26:50 2011 +0100
+++ b/doc/api/version2.rst Thu Jul 28 17:20:40 2011 +0100
@@ -97,6 +97,7 @@
.. |site| replace:: CKAN
.. |api| replace:: API
.. |version| replace:: Version 2
+.. |warning| replace:: This is the latest version of the API.
.. |base_location| replace:: ``http://ckan.net/api/2``
.. |main_doc| replace:: :doc:`../api`
.. |usage| replace:: to view and change
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/doc/api/warning.rst.inc Thu Jul 28 17:20:40 2011 +0100
@@ -0,0 +1,1 @@
+.. warning:: This version is now deprecated in favour of :doc:`version2`.
\ No newline at end of file
--- a/doc/authentication.rst Thu Jul 28 16:26:50 2011 +0100
+++ /dev/null Thu Jan 01 00:00:00 1970 +0000
@@ -1,7 +0,0 @@
-CKAN Authentication and Identification
-======================================
-
-CKAN operates a delegated authentication model based on openid. Integration of
-authentication into CKAN is provided by repoze.who and the repoze.who.openid
-plugin.
-
--- a/doc/authorization.rst Thu Jul 28 16:26:50 2011 +0100
+++ b/doc/authorization.rst Thu Jul 28 17:20:40 2011 +0100
@@ -1,325 +1,206 @@
-=====================================
-CKAN Authorization and Access Control
-=====================================
-
-This document gives an overview of CKAN's authorization capabilities and model
-in relation to access control. The authentication/identification aspects of
-access control are dealt with separately in :doc:`authentication`.
-
-
-Overview
-========
+==========================
+Set and Manage Permissions
+==========================
CKAN implements a fine-grained role-based access control system.
- *In a nutshell*: For a particular **package** (or other protected object) a **user** can be assigned a **role** which specifies permitted **actions** (edit, delete, change permissions, etc.).
+This section describes:
-Protected Objects and Authorization
------------------------------------
+* :ref:`permissions-overview`. An overview of the concepts underlying CKAN authorization: objects, actions and roles.
+* :ref:`permissions-default`. The default permissions in CKAN.
+* :ref:`permissions-manage`. Managing and setting permissions.
+* :ref:`permissions-publisher-mode`. Suitable for systems where you want to limit write access to CKAN.
-There are variety of **protected objects** to which access can be controlled,
-for example the ``System``, ``Packages``, Package ``Groups`` and
-``Authorization Groups``. Access control is fine-grained in that it can be
-set for each individual package, group or authorization group instance.
+.. _permissions-overview:
-For each protected object there are a set of relevant **actions** such as
-create', 'admin', 'edit' etc. To facilitate mapping Users and Objects with Actions, Actions are aggregated into a set of **roles** (e.g. an 'editor' role would have 'edit' and 'read' action).
+Overview
+--------
-A special role is taken by the ``System`` object, which serves as an
-authorization object for any assignments which do not relate to a specific
-object. For example, the creation of a package cannot be linked to a
-specific package instance and is therefore a system operation.
+In a nutshell: for a particular **object** (e.g. a package) a CKAN **user** can be assigned a **role** (e.g. editor) which allows permitted **actions** (e.g. read, edit).
-To gain further flexibility, users can be assigned to **authorization
-groups**. Authz groups are both the object of authorization (i.e. one
-can have several roles with regards to an authz group) and the subject
-of authorization (i.e. they can be assigned roles on other objects which
-will apply to their members.
+In more detail, these concepts are as follows:
+
+* There are **objects** to which access can be controlled, such as packages and groups.
+* For each object there are a set of relevant **actions**, such as create and edit, which users can perform on the object.
+* To simplify mapping users to actions and objects, actions are aggregated into a set of **roles**. For example, an editor role would automatically have edit and read actions.
+* Finally, CKAN has registered **users**.
-The assignment of users and authorization groups to roles on a given
-protected object (such as a package) can be done by 'admins' via the
-'authorization' tab of the web interface (or by system admins via that
-interface or the system admin interface). There is also a command-line
-based authorization manager, detailed below.
+Objects
++++++++
-Command-line authorization management
--------------------------------------
+Permissions are controlled per object: access can be controlled for an individual
+package, group or authorization group instance. Current objects include
+**packages**, package **groups**, **authorization groups** and the **system**.
-Although the Admin Extension provides a Web interface for managing authorization, there is a set of more powerful :doc:`paster` commands for fine-grained control.
+* A package is the basic CKAN concept of metadata about a dataset.
+* A group of packages can be set up to specify which users have permission to add or remove packages from the group.
+* Users can be assigned to authorization groups, to increase flexibility. Instead of specifying the privileges of specific users on a package or group, you can also specify a set of users that share the same rights. To do that, an authorization group can be set up and users can be added to it. Authorization groups are both the object of authorization (i.e. one can have several roles with regards to an authorization group, such as being allowed to read or edit it) and the subject of authorization (i.e. they can be assigned roles on other objects which will apply to their members, such as the group having edit rights on a particular group).
+* Finally, the system object is special, serving as an object for assignments that do not relate to a specific object. For example, creating a package cannot be linked to a specific package instance, and is therefore a operation.
-The ``roles`` command will list and modify the assignment of actions to
-roles::
- $ paster --plugin=ckan roles -c my.ini list
- $ paster --plugin=ckan roles -c my.ini deny editor create-package
- $ paster --plugin=ckan roles -c my.ini allow editor create-package
+Actions
++++++++
-This would first list all role action assignments, then remove the
-'create-package' action from the 'editor' role and finally re-assign
-that role.
+**Actions** are defined in the Action enumeration in ``ckan/model/authz.py`` and currently include: **edit**, **change-state**, **read**, **purge**, **edit-permissions**, **create-package**, **create-group**, **create-authorization-group**, **read-site**, **read-user**, **create-user**.
-Similarly, the ``rights`` command will set the authorization roles of
-a specific user on a given object within the system::
+As noted above, some of these (e.g. **read**) have meaning for any type of object, while some (e.g. **create-package**) can not be associated with any particular object, and are therefore only associated with the system object.
- $ paster --plugin=ckan rights -c my.ini list
+The **read-site** action (associated with the system object) allows or denies access to pages not associated with specific objects. These currently include:
+
+ * Package search
+ * Group index
+ * Tags index
+ * Authorization Group index
+ * All requests to the API (on top of any other authorization requirements)
-Will list all assigned rights within the system. It is recommended to then
-grep over the resulting output to search for specific object roles.
+There are also some shortcuts that are provided directly by the authorization
+system (rather than being expressed as subject-object-role tuples):
-Rights assignment follows a similar pattern::
-
- $ paster --plugin=ckan rights -c my.ini make bar admin package:foo
-
-This would assign the user named ``bar`` the ``admin`` role on the package
-foo. Instead of user names and package names, a variety of different
-entities can be the subject or object of a role assignment. Some of those
-include authorization groups, package groups and the system as a whole
-(``system:```)::
-
- # make 'chef' a system-wide admin:
- $ paster --plugin=ckan rights -c my.ini make chef admin system
- # allow all members of authz group 'foo' to edit group 'bar'
- $ paster --plugin=ckan rights -c my.ini make agroup:foo edit \
- group:bar
-
-To revoke one of the roles assigned using ``make``, the ``remove`` command
-is available::
-
- $ paster --plugin=ckan rights -c my.ini remove bar admin package:foo
-
-For more help on either of these commands, also refer to the paster help.
+ * A user given the **admin** right for the **system** object is a 'sysadmin' and can perform any action on any object. (A shortcut for creating a sysadmin is by using the ``paster sysadmin`` command.)
+ * A user given the **admin** right for a particular object can perform any action on that object.
Roles
------
++++++
-Each role has a list of permitted *actions* appropriate for a protected object.
+Each **role** has a list of permitted actions appropriate for a protected object.
-Currently there are three basic roles (although you can add others if these defaults do not suit):
+Currently there are four basic roles:
* **reader**: can read the object
- * **anon_editor**: (anonymous i.e. not logged in) can edit and read the object
+ * **anon_editor**: anonymous users (i.e. not logged in) can edit and read the object
* **editor**: can edit, read and create new objects
* **admin**: admin can do anything including: edit, read, delete,
update-permissions (change authorizations for that object)
-When you install a new CKAN extension or upgrade your version of CKAN then new actions may be created, and permissions may given to these basic roles, according to the broad intention of the name of the roles.
+You can add other roles if these defaults are not sufficient for your system.
-It is suggested that if the broad idea of these basic roles and their actions are not suitable for your CKAN instance then you create new roles and assign them actions of your own choosing, rather than edit the roles. If the definition of the roles drift from their name then it can be confusing for admins and cause problems for CKAN upgrades and new extensions.
+.. warning:: If the broad idea of these basic roles and their actions is not suitable for your CKAN system, we suggest you create new roles, rather than edit the basic roles. If the definition of a role changes but its name does not, it is likely to confuse administrators and cause problems for CKAN upgrades and extensions.
-Actions
--------
+.. note:: When you install a new CKAN extension, or upgrade your version of CKAN, new actions may be created, and permissions given to these basic roles, in line with the broad intention of the roles.
-Actions are defined in the Action enumeration in ckan/model/authz.py and currently include: `edit`, `change-state`, `read`, `purge`, `edit-permissions`, `create-package`, `create-group`, `create-authorization-group`, `read-site`, `read-user`, `create-user`.
+Users
++++++
-Obviously, some of these (e.g. `read`) have meaning for any type of Domain Object, and some (e.g. `create-package`) can not be associated with any particular Domain Object, so the Context for Roles with these Actions is `system`.
+You can manage CKAN users via the command line with the ``paster user`` command - for more information, see :ref:`paster-user`.
-The `read-site` action (with System context) is designed to provide/deny access to pages not associated with Domain Objects. This currently includes:
-
- * Package Search
- * Group index
- * Tags index
- * Authorization Group index
- * all requests to the API (on top of any other authorization requirements)
+There are two special *pseudo-users* in CKAN, **visitor** and **logged-in**. These are used to refer to special sets of users, respectively those who are a) not logged-in ("visitor") and b) logged-in ("logged-in").
-There are also some shortcuts that are provided directly by the authorization
-system (rather than being expressed as subject-object-role tuples):
+The ``default_roles`` config option in the CKAN config file lets you set the default authorization roles (i.e. permissions) for these two types of users. For more information, see :doc:`configuration`.
- * A user given the admin right for the System object is a 'System Admin' and can do any action on any object. (A shortcut for creating a System Admin is by using the ``paster sysadmin`` command.)
- * A user given the admin right for a particular object can do any action to that object.
-Publisher mode setup
+.. _permissions-default:
+
+Default Permissions
+-------------------
+
+CKAN ships with the following default permissions:
+
+* When a new package is created, its creator automatically becomes **admin** for it. This user can then change permissions for other users.
+* By default, any other user (including both visitors and logged-ins) can read and write to this package.
+
+These defaults can be changed in the CKAN config - see ``default_roles`` in :doc:`configuration`.
+
+
+.. _permissions-manage:
+
+Managing Permissions
--------------------
-Although ckan.net is forging ahead with the Wikipedia model of allowing anyone to add and improve metadata, some CKAN instances prefer to operate in 'Publisher mode' which allows edits only from authorized users.
+The assignment of users and authorization groups to roles on a given
+protected object (such as a package) can be done by 'admins' via the
+'authorization' tab of the web interface (or by sysadmins via that
+interface or the system admin interface).
+
+There is also a command-line authorization manager, detailed below.
+
+Command-line authorization management
++++++++++++++++++++++++++++++++++++++
+
+Although the admin extension provides a Web interface for managing authorization,
+there is a set of more powerful ``paster`` commands for fine-grained control
+(see :doc:`paster`).
+
+The ``rights`` command is used to configure the authorization roles of
+a specific user on a given object within the system.
+
+For example, to list all assigned rights in the system (which you can then grep if needed)::
+
+ paster --plugin=ckan rights -c my.ini list
+
+The ``rights make`` command lets you assign specific permissions. For example, to give the user named **bar** the **admin** role on the package foo::
+
+ paster --plugin=ckan rights -c my.ini make bar admin package:foo
+
+As well as users and packages, you can assign rights to other objects. These
+include authorization groups, package groups and the system as a whole.
+
+For example, to make the user 'chef' a system-wide admin::
+
+ paster --plugin=ckan rights -c my.ini make chef admin system
+
+Or to allow all members of authorization group 'foo' to edit group 'bar'::
+
+ paster --plugin=ckan rights -c my.ini make agroup:foo edit \
+ group:bar
+
+To revoke one of the roles assigned using ``rights make``, the ``rights remove`` command
+is available. For example, to remove **bar**'s **admin** role on the foo package::
+
+ paster --plugin=ckan rights -c my.ini remove bar admin package:foo
+
+The ``roles`` command lists and modifies the assignment of actions to
+roles.
+
+To list all role assignments::
+
+ paster --plugin=ckan roles -c my.ini list
+
+To remove the 'create-package' action from the 'editor' role::
+
+ paster --plugin=ckan roles -c my.ini deny editor create-package
+
+And to re-assign 'create-package' to the 'editor' role::
+
+ paster --plugin=ckan roles -c my.ini allow editor create-package
+
+For more help on either of these commands, you can use ``--help`` (as described in :ref:`paster-help`)::
+
+ paster --plugin=ckan roles --help
+ paster --plugin=ckan rights --help
+
+
+.. _permissions-publisher-mode:
+
+Publisher Mode
+--------------
+
+Although the `Data Hub <http://datahub.org>`_ prefers the Wikipedia-style model of allowing anyone to add and improve metadata, some CKAN instances prefer to operate in 'publisher mode'.
+
+This allows edits only from authorized users. It is designed for installations where you wish to limit write access to CKAN and orient the system around specific publishing groups (e.g. government departments or specific institutions).
+
+The key features are:
+
+* Packages are assigned to a specific publishing group.
+* Only users associated to that group are able to create or update packages associated to that group.
To operate in this mode:
- 1. Remove the rights for general public to edit existing packages and create new ones.::
+1. First, remove the general public's rights to create and edit packages::
- paster rights remove visitor anon_editor package:all
- paster rights remove logged_in editor package:all
- paster rights remove visitor anon_editor system
- paster rights remove logged_in editor system
+ paster rights remove visitor anon_editor package:all
+ paster rights remove logged_in editor package:all
+ paster rights remove visitor anon_editor system
+ paster rights remove logged_in editor system
- 2. If logged-in users have already created packages in your system then you may also wish to remove admin rights. e.g.::
+2. If logged-in users have already created packages in your system, you may also wish to remove their admin rights. For example::
- paster rights remove bob admin package:all
+ paster rights remove bob admin package:all
- 3. Change the default rights for newly created packages. Do this by using these values in your config (.ini file)::
+3. Change the default rights for newly created packages. Do this by using these values in your config file (see :doc:`configuration`)::
- ckan.default_roles.Package = {"visitor": ["reader"], "logged_in": ["reader"]}
- ckan.default_roles.Group = {"visitor": ["reader"], "logged_in": ["reader"]}
- ckan.default_roles.System = {"visitor": ["reader"], "logged_in": ["reader"]}
- ckan.default_roles.AuthorizationGroup = {"visitor": ["reader"], "logged_in": ["reader"]}
+ ckan.default_roles.Package = {"visitor": ["reader"], "logged_in": ["reader"]}
+ ckan.default_roles.Group = {"visitor": ["reader"], "logged_in": ["reader"]}
+ ckan.default_roles.System = {"visitor": ["reader"], "logged_in": ["reader"]}
+ ckan.default_roles.AuthorizationGroup = {"visitor": ["reader"], "logged_in": ["reader"]}
- Note there is also the possibility to restrict package edits by a user's authorization group. See http://wiki.ckan.net/Publisher_Mode
-
-
-Examples
---------
-
-Example 1: Package 'paper-industry-stats':
-
- * David Brent is an 'admin'
- * Gareth Keenan is an 'editor'
- * Logged-in is a 'reader' (This is a special user, meaning 'anyone who is
- logged in')
- * Visitor is a 'reader' (Another special user, meaning 'anyone')
-
-That is, Gareth and David can edit this package, but only Gareth can assign
-roles (privileges) to new team members. Anyone can see (read) the package.
-
-
-Example 2: The current default for new packages is:
-
- * the user who creates it is an 'admin'
- * Visitor and Logged-in are both an 'editor' and 'reader'
-
-NB: "Visitor" and "Logged-in" are special "pseudo-users" used as a way of
-concretely referring to the special sets of users, namely those that are a) not
-logged-in ("visitor") and b) logged-in ("Logged-in")
-
-User Notes
-----------
-
-When a new package is created, its creator automatically become admin for
-it. This user can then change permissions for other users.
-
-NB: by default any user (including someone who is not logged-in) will be able
-to read and write. This default can be changed in the CKAN configuration - see ``default_roles`` in :doc:`configuration`.
-
-
-Developer Notes
-===============
-
-We record tuples of the form:
-
-======== ================= ======= ====================
-user authorized_group role object
-======== ================= ======= ====================
-levin editor package::warandpeace
-======== ================= ======= ====================
-
-
-
-
-Requirements and Use Cases
---------------------------
-
- * A user means someone who is logged in.
- * A visitor means someone who is not logged in.
- * An protected object is the subject of a permission (either a user or a
- pseudo-user)
- * There are roles named: Admin, Reader, Writer
-
- 1. A visitor visits a package page and reads the content
- 2. A visitor visits a package page and edits the package
- 3. Ditto 1 for a user
- 4. Ditto 2 for a user
- 5. On package creation if done by a user and not a visitor then user is made
- the 'admin'
- 6. An admin of a package adds a user as an admin
- 7. An admin of a package removes a user as an admin
- 8. Ditto for admin re. editor
- 9. Ditto for admin re. reader
- 10. We wish to be able assign roles to 2 specific entire groups in addition
- to specific users: 'visitor', 'users'. These will be termed pseudo-users
- as we do not have AC 'groups' as such.
- 11. The sysadmin alters the assignment of entities to roles for any package
- 12. A visitor goes to a package where the editor role does not include
- 'visitor' pseudo-user. They are unable to edit the package.
- 13. Ditto for user where users pseudo-user does not have editor role and user
- is not an editor for the package
- 14. Ditto 12 re reader role.
- 15. Ditto 13 re reader role.
- 16. Try to edit over REST interface a package for which 'visitor' has Editor
- role, but no API is supplied. Not allowed.
-
-
-Not Yet Implemented
-+++++++++++++++++++
-
- * Support for access-related groups
- * Support for blacklisting
-
-
-Conceptual Overview
--------------------
-
-**Warning: not all of what is described in this conceptual overview is yet
-fully implemented.**
-
- * There are Users and (User) Authorization Groups
- * There are actions which may be performed on "protected objects" such as
- Package, Group, System
- * Roles aggregate actions
- * UserObjectRole which assign users (or Authorization groups) a role on an
- object (user, role, object). We will often refer to these informally as
- "permissions".
-
-NB: there is no object explicitly named "Permission". This is to avoid
-confusion: a 'normal' "Permission" (as in e.g. repoze.what) would correspond to
-an action-object tuple. This works for the case where protected objects are
-limited e.g. a few core subsystems like email, admin panel etc). However, we
-have many protected objects (e.g. one for each package) and we use roles so
-this 'normal' model does not work well.
-
-Question: do we require for *both* Users and UserAuthorizationGroup to be
-subject of Role or not?
-
-Ans: Yes. Why? Consider, situation where I just want to give an individual user
-permission on a given object (e.g. assigning authz permission for a package)?
-If I just have UserAuthorizationGroups one would need to create a group just
-for that individual. This isn't impossible but consider next how to assign
-permissions to edit the Authorization Groups? One would need create another
-group for this but then we have recursion ad infinitum (unless this were
-especially encompassed in some system level permission or one has some group
-which is uneditable ...)
-
-Thus, one requires both Users and UserAuthorizationGroups to be subject of
-"permissions". To summarize the approximate structure we have is::
-
- class SubjectOfAuthorization
- class User
- class UserAuthorizationGroup
-
- class ObjectOfAuthorization
- class Package
- class Group
- class UserAuthorizationGroup
- ...
-
- class SubjectRoleObject
- subject_of_authorization
- object_of_authorization
- role
-
-
-Determining permissions
------------------------
-
-See ckan.authz.Authorizer.is_authorized
-
-.. automethod:: ckan.authz.Authorizer.is_authorized
-
-
-Comparison with other frameworks and approaches
-===============================================
-
-repoze.what
------------
-
-Demo example model::
-
- User
- Group
- Permission
-
- * Users are assigned to groups
- * Groups are assigned permissions
-
-Capabilities
-------------
-
-Each possible action-object tuple receive an identifier which we term the
-"capability". We would then list tuples (capability_subject, capability).
+Note you can also restrict package edits by a user's authorization group.
--- a/doc/buildbot.rst Thu Jul 28 16:26:50 2011 +0100
+++ b/doc/buildbot.rst Thu Jul 28 17:20:40 2011 +0100
@@ -1,13 +1,15 @@
-Setting up buildbot
-+++++++++++++++++++
+================
+Install Buildbot
+================
-These directions provide some rough information for setting up a build bot on a Lucid machine.
+This section provides information for CKAN core developers setting up buildbot on an Ubuntu Lucid machine.
+If you simply want to check the status of the latest CKAN builds, visit http://buildbot.okfn.org/.
Apt Installs
============
-Install CKAN core dependencies from Lucid distribution (as per :doc:`README`)::
+Install CKAN core dependencies from Lucid distribution::
sudo apt-get install build-essential libxml2-dev libxslt-dev
sudo apt-get install wget mercurial postgresql libpq-dev git-core
@@ -22,7 +24,7 @@
sudo apt-get install buildbot
-Deb building software (as per :doc:`deb`)::
+Deb building software::
sudo apt-get install -y dh-make devscripts fakeroot cdbs
@@ -36,7 +38,7 @@
sudo dpkg-reconfigure locales
-Postgres setup
+Postgres Setup
==============
If installation before failed to create a cluster, do this after fixing errors::
@@ -51,7 +53,7 @@
sudo -u postgres createdb -O buildslave ckanext
-Buildslave setup
+Buildslave Setup
================
Rough commands::
@@ -74,7 +76,7 @@
pip -E pyenv-tools install buildkit
-Buildmaster setup
+Buildmaster Setup
=================
Rough commands::
@@ -113,23 +115,23 @@
sudo /etc/init.d/buildbot start
-Now check you can view buildbot at: http://localhost:8010/
+Now check you can view buildbot at http://localhost:8010/
-Connect ports
+Connect Ports
=============
It's preferable to view the buildbot site at port 80 rather than 8010.
-If there is no other web service on this machine, you might connect up the addresses using iptables::
+If there is no other web service on this machine, you might connect up the addresses using ``iptables``::
sudo iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 8010
-Otherwise it is best to do a reverse proxy. Using apache, edit this file::
+Otherwise it is best to set up a reverse proxy. Using Apache, edit this file::
sudo vim /etc/apache2/sites-available/buildbot.okfn.org
-to be like this::
+to look like this::
<VirtualHost *:80>
ServerName buildbot.okfn.org
--- a/doc/ckan-features.svg Thu Jul 28 16:26:50 2011 +0100
+++ /dev/null Thu Jan 01 00:00:00 1970 +0000
@@ -1,323 +0,0 @@
-<?xml version="1.0" encoding="UTF-8" standalone="no"?>
-<!-- Created with Inkscape (http://www.inkscape.org/) -->
-<svg
- xmlns:dc="http://purl.org/dc/elements/1.1/"
- xmlns:cc="http://creativecommons.org/ns#"
- xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
- xmlns:svg="http://www.w3.org/2000/svg"
- xmlns="http://www.w3.org/2000/svg"
- xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
- xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
- width="744.09448819"
- height="1052.3622047"
- id="svg2"
- sodipodi:version="0.32"
- inkscape:version="0.46"
- sodipodi:docname="ckan-features.svg"
- inkscape:output_extension="org.inkscape.output.svg.inkscape">
- <defs
- id="defs4">
- <linearGradient
- id="linearGradient3557">
- <stop
- style="stop-color:#f52971;stop-opacity:0.71794873;"
- offset="1"
- id="stop3559" />
- </linearGradient>
- <linearGradient
- id="linearGradient3497">
- <stop
- style="stop-color:#f50b00;stop-opacity:0.54700857;"
- offset="1"
- id="stop3499" />
- </linearGradient>
- <linearGradient
- id="linearGradient3435">
- <stop
- style="stop-color:#f56c00;stop-opacity:0.64957267;"
- offset="0"
- id="stop3543" />
- <stop
- id="stop3549"
- offset="1"
- style="stop-color:#f56c00;stop-opacity:0.58039216;" />
- </linearGradient>
- <linearGradient
- id="linearGradient3155">
- <stop
- id="stop3457"
- offset="0"
- style="stop-color:#eff529;stop-opacity:0.67521369;" />
- <stop
- style="stop-color:#eff529;stop-opacity:0.65811968;"
- offset="1"
- id="stop3515" />
- <stop
- id="stop3455"
- offset="1"
- style="stop-color:#eff529;stop-opacity:1;" />
- </linearGradient>
- <inkscape:perspective
- sodipodi:type="inkscape:persp3d"
- inkscape:vp_x="0 : 526.18109 : 1"
- inkscape:vp_y="0 : 1000 : 0"
- inkscape:vp_z="744.09448 : 526.18109 : 1"
- inkscape:persp3d-origin="372.04724 : 350.78739 : 1"
- id="perspective10" />
- </defs>
- <sodipodi:namedview
- id="base"
- pagecolor="#ffffff"
- bordercolor="#666666"
- borderopacity="1.0"
- gridtolerance="10000"
- guidetolerance="10"
- objecttolerance="10"
- inkscape:pageopacity="0.0"
- inkscape:pageshadow="2"
- inkscape:zoom="1"
- inkscape:cx="380.55961"
- inkscape:cy="767.44485"
- inkscape:document-units="px"
- inkscape:current-layer="layer1"
- showgrid="true"
- inkscape:snap-global="false"
- inkscape:window-width="1014"
- inkscape:window-height="726"
- inkscape:window-x="-5"
- inkscape:window-y="25">
- <inkscape:grid
- type="xygrid"
- id="grid3377" />
- </sodipodi:namedview>
- <metadata
- id="metadata7">
- <rdf:RDF>
- <cc:Work
- rdf:about="">
- <dc:format>image/svg+xml</dc:format>
- <dc:type
- rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
- </cc:Work>
- </rdf:RDF>
- </metadata>
- <g
- inkscape:label="Layer 1"
- inkscape:groupmode="layer"
- id="layer1">
- <path
- sodipodi:type="arc"
- style="opacity:0.6;fill:#a3bcce;fill-opacity:1;stroke:none;stroke-width:4.65620232;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
- id="path2383"
- sodipodi:cx="252.85715"
- sodipodi:cy="389.50504"
- sodipodi:rx="144.28572"
- sodipodi:ry="137.14285"
- d="M 397.14287,389.50504 A 144.28572,137.14285 0 1 1 108.57143,389.50504 A 144.28572,137.14285 0 1 1 397.14287,389.50504 z"
- transform="matrix(0.5964695,0,0,0.5964695,105.76357,68.787717)"
- inkscape:export-filename="/home/rgrp/hgroot/ckan/doc/ckan-features.png"
- inkscape:export-xdpi="90"
- inkscape:export-ydpi="90" />
- <text
- xml:space="preserve"
- style="font-size:14.81217575px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:center;line-height:125%;writing-mode:lr-tb;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;stroke-width:3;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;font-family:Sans;-inkscape-font-specification:Sans"
- x="234.04362"
- y="312.77817"
- id="text3367"
- sodipodi:linespacing="125%"
- inkscape:export-filename="/home/rgrp/hgroot/ckan/doc/ckan-features.png"
- inkscape:export-xdpi="90"
- inkscape:export-ydpi="90"><tspan
- sodipodi:role="line"
- x="236.40141"
- y="312.77817"
- id="tspan3371">Wiki / </tspan><tspan
- sodipodi:role="line"
- x="234.04362"
- y="331.29337"
- id="tspan3375">Versioned DB</tspan></text>
- <path
- sodipodi:type="arc"
- style="opacity:0.6;fill:#a0cfa1;fill-opacity:1;stroke:none;stroke-width:4.65620232;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
- id="path3405"
- sodipodi:cx="252.85715"
- sodipodi:cy="389.50504"
- sodipodi:rx="144.28572"
- sodipodi:ry="137.14285"
- d="M 397.14287,389.50504 A 144.28572,137.14285 0 1 1 108.57143,389.50504 A 144.28572,137.14285 0 1 1 397.14287,389.50504 z"
- transform="matrix(0.5964695,0,0,0.5964695,170.8961,-0.1971008)"
- inkscape:export-filename="/home/rgrp/hgroot/ckan/doc/ckan-features.png"
- inkscape:export-xdpi="90"
- inkscape:export-ydpi="90" />
- <text
- xml:space="preserve"
- style="font-size:14.81217575px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:center;line-height:125%;writing-mode:lr-tb;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;font-family:Sans;-inkscape-font-specification:Sans"
- x="323.72488"
- y="187.99576"
- id="text3407"
- sodipodi:linespacing="125%"
- inkscape:export-filename="/home/rgrp/hgroot/ckan/doc/ckan-features.png"
- inkscape:export-xdpi="90"
- inkscape:export-ydpi="90"><tspan
- sodipodi:role="line"
- x="323.72488"
- y="187.99576"
- id="tspan3461">Package Index</tspan><tspan
- sodipodi:role="line"
- x="323.72488"
- y="206.51097"
- id="tspan3511">(APT/PyPI/CPAN)</tspan></text>
- <path
- transform="matrix(0.5964695,0,0,0.5964695,240.3075,69.096304)"
- d="M 397.14287,389.50504 A 144.28572,137.14285 0 1 1 108.57143,389.50504 A 144.28572,137.14285 0 1 1 397.14287,389.50504 z"
- sodipodi:ry="137.14285"
- sodipodi:rx="144.28572"
- sodipodi:cy="389.50504"
- sodipodi:cx="252.85715"
- id="path3415"
- style="opacity:0.6;fill:#c7d9a0;fill-opacity:1;stroke:none;stroke-width:4.65620232;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
- sodipodi:type="arc"
- inkscape:export-filename="/home/rgrp/hgroot/ckan/doc/ckan-features.png"
- inkscape:export-xdpi="90"
- inkscape:export-ydpi="90" />
- <text
- sodipodi:linespacing="125%"
- id="text3417"
- y="311.95526"
- x="423.10461"
- style="font-size:14.81217575px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:center;line-height:125%;writing-mode:lr-tb;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;stroke-width:3;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;font-family:Sans;-inkscape-font-specification:Sans"
- xml:space="preserve"
- inkscape:export-filename="/home/rgrp/hgroot/ckan/doc/ckan-features.png"
- inkscape:export-xdpi="90"
- inkscape:export-ydpi="90"><tspan
- y="311.95526"
- x="423.10461"
- sodipodi:role="line"
- id="tspan3447">Listing</tspan><tspan
- y="330.47049"
- x="423.10461"
- sodipodi:role="line"
- id="tspan3179">(Freshmeat)</tspan></text>
- <text
- xml:space="preserve"
- style="font-size:18.10377121px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:center;line-height:125%;writing-mode:lr-tb;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;stroke-width:3;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;font-family:Sans;-inkscape-font-specification:Sans"
- x="323.94531"
- y="289.737"
- id="text3183"
- sodipodi:linespacing="125%"
- inkscape:export-filename="/home/rgrp/hgroot/ckan/doc/ckan-features.png"
- inkscape:export-xdpi="90"
- inkscape:export-ydpi="90"><tspan
- sodipodi:role="line"
- x="323.94531"
- y="289.737"
- id="tspan3185">CKAN</tspan></text>
- <path
- transform="matrix(0.7248396,0,0,0.7248396,91.826254,471.23864)"
- d="M 397.14287,389.50504 A 144.28572,137.14285 0 1 1 108.57143,389.50504 A 144.28572,137.14285 0 1 1 397.14287,389.50504 z"
- sodipodi:ry="137.14285"
- sodipodi:rx="144.28572"
- sodipodi:cy="389.50504"
- sodipodi:cx="252.85715"
- id="path3178"
- style="opacity:0.6;fill:#a3bcce;fill-opacity:1;stroke:none;stroke-width:4.65620232;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
- sodipodi:type="arc" />
- <text
- sodipodi:linespacing="125%"
- id="text3180"
- y="732.73987"
- x="224.71428"
- style="font-size:15px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:center;line-height:125%;writing-mode:lr-tb;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;stroke-width:3;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;font-family:Sans;-inkscape-font-specification:Sans"
- xml:space="preserve"><tspan
- id="tspan3182"
- y="732.73987"
- x="227.10197"
- sodipodi:role="line">Wiki / </tspan><tspan
- id="tspan3184"
- y="751.48987"
- x="224.71428"
- sodipodi:role="line">Versioned DB</tspan></text>
- <path
- transform="matrix(0.7248396,0,0,0.7248396,170.97637,387.40715)"
- d="M 397.14287,389.50504 A 144.28572,137.14285 0 1 1 108.57143,389.50504 A 144.28572,137.14285 0 1 1 397.14287,389.50504 z"
- sodipodi:ry="137.14285"
- sodipodi:rx="144.28572"
- sodipodi:cy="389.50504"
- sodipodi:cx="252.85715"
- id="path3186"
- style="opacity:0.6;fill:#a0cfa1;fill-opacity:1;stroke:none;stroke-width:4.65620232;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
- sodipodi:type="arc" />
- <text
- sodipodi:linespacing="125%"
- id="text3188"
- y="608.10217"
- x="356.69644"
- style="font-size:15px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:center;line-height:125%;writing-mode:lr-tb;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;font-family:Sans;-inkscape-font-specification:Sans"
- xml:space="preserve"><tspan
- id="tspan3190"
- y="608.10217"
- x="356.69644"
- sodipodi:role="line">Package Index</tspan><tspan
- id="tspan3192"
- y="626.85217"
- x="356.69644"
- sodipodi:role="line">(APT/PyPI/CPAN)</tspan></text>
- <path
- sodipodi:type="arc"
- style="opacity:0.6;fill:#c7d9a0;fill-opacity:1;stroke:none;stroke-width:4.65620232;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
- id="path3194"
- sodipodi:cx="252.85715"
- sodipodi:cy="389.50504"
- sodipodi:rx="144.28572"
- sodipodi:ry="137.14285"
- d="M 397.14287,389.50504 A 144.28572,137.14285 0 1 1 108.57143,389.50504 A 144.28572,137.14285 0 1 1 397.14287,389.50504 z"
- transform="matrix(0.7248396,0,0,0.7248396,255.32625,471.61364)" />
- <text
- xml:space="preserve"
- style="font-size:15px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:center;line-height:125%;writing-mode:lr-tb;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;stroke-width:3;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;font-family:Sans;-inkscape-font-specification:Sans"
- x="494.46429"
- y="732.73987"
- id="text3196"
- sodipodi:linespacing="125%"><tspan
- id="tspan3198"
- sodipodi:role="line"
- x="494.46429"
- y="732.73987">Listing</tspan><tspan
- id="tspan3200"
- sodipodi:role="line"
- x="494.46429"
- y="751.48987">(Freshmeat)</tspan></text>
- <path
- transform="matrix(0.7248396,0,0,0.7248396,170.97637,545.68123)"
- d="M 397.14287,389.50504 A 144.28572,137.14285 0 1 1 108.57143,389.50504 A 144.28572,137.14285 0 1 1 397.14287,389.50504 z"
- sodipodi:ry="137.14285"
- sodipodi:rx="144.28572"
- sodipodi:cy="389.50504"
- sodipodi:cx="252.85715"
- id="path3202"
- style="opacity:0.6;fill:#b2a3c4;fill-opacity:1;stroke:none;stroke-width:4.65620232;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
- sodipodi:type="arc" />
- <text
- xml:space="preserve"
- style="font-size:15px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:center;line-height:125%;writing-mode:lr-tb;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;stroke-width:3;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;font-family:Sans;-inkscape-font-specification:Sans"
- x="354.71429"
- y="886.48987"
- id="text3204"
- sodipodi:linespacing="125%"><tspan
- sodipodi:role="line"
- x="354.71429"
- y="886.48987"
- id="tspan3206">Forum</tspan></text>
- <text
- sodipodi:linespacing="125%"
- id="text3208"
- y="753.73987"
- x="356.96429"
- style="font-size:18px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:center;line-height:125%;writing-mode:lr-tb;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;stroke-width:3;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;font-family:Sans;-inkscape-font-specification:Sans"
- xml:space="preserve"><tspan
- id="tspan3210"
- y="753.73987"
- x="356.96429"
- sodipodi:role="line">CKAN</tspan></text>
- </g>
-</svg>
--- a/doc/ckan-vision.dia Thu Jul 28 16:26:50 2011 +0100
+++ /dev/null Thu Jan 01 00:00:00 1970 +0000
@@ -1,825 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<dia:diagram xmlns:dia="http://www.lysator.liu.se/~alla/dia/">
- <dia:diagramdata>
- <dia:attribute name="background">
- <dia:color val="#ffffff"/>
- </dia:attribute>
- <dia:attribute name="pagebreak">
- <dia:color val="#000099"/>
- </dia:attribute>
- <dia:attribute name="paper">
- <dia:composite type="paper">
- <dia:attribute name="name">
- <dia:string>#A4#</dia:string>
- </dia:attribute>
- <dia:attribute name="tmargin">
- <dia:real val="2.8222000598907471"/>
- </dia:attribute>
- <dia:attribute name="bmargin">
- <dia:real val="2.8222000598907471"/>
- </dia:attribute>
- <dia:attribute name="lmargin">
- <dia:real val="2.8222000598907471"/>
- </dia:attribute>
- <dia:attribute name="rmargin">
- <dia:real val="2.8222000598907471"/>
- </dia:attribute>
- <dia:attribute name="is_portrait">
- <dia:boolean val="true"/>
- </dia:attribute>
- <dia:attribute name="scaling">
- <dia:real val="1"/>
- </dia:attribute>
- <dia:attribute name="fitto">
- <dia:boolean val="false"/>
- </dia:attribute>
- </dia:composite>
- </dia:attribute>
- <dia:attribute name="grid">
- <dia:composite type="grid">
- <dia:attribute name="width_x">
- <dia:real val="1"/>
- </dia:attribute>
- <dia:attribute name="width_y">
- <dia:real val="1"/>
- </dia:attribute>
- <dia:attribute name="visible_x">
- <dia:int val="1"/>
- </dia:attribute>
- <dia:attribute name="visible_y">
- <dia:int val="1"/>
- </dia:attribute>
- <dia:composite type="color"/>
- </dia:composite>
- </dia:attribute>
- <dia:attribute name="color">
- <dia:color val="#d8e5e5"/>
- </dia:attribute>
- <dia:attribute name="guides">
- <dia:composite type="guides">
- <dia:attribute name="hguides"/>
- <dia:attribute name="vguides"/>
- </dia:composite>
- </dia:attribute>
- </dia:diagramdata>
- <dia:layer name="Background" visible="true">
- <dia:object type="Standard - Box" version="0" id="O0">
- <dia:attribute name="obj_pos">
- <dia:point val="7.5,18.5"/>
- </dia:attribute>
- <dia:attribute name="obj_bb">
- <dia:rectangle val="7.45,18.45;14.55,25.55"/>
- </dia:attribute>
- <dia:attribute name="elem_corner">
- <dia:point val="7.5,18.5"/>
- </dia:attribute>
- <dia:attribute name="elem_width">
- <dia:real val="7"/>
- </dia:attribute>
- <dia:attribute name="elem_height">
- <dia:real val="7"/>
- </dia:attribute>
- <dia:attribute name="show_background">
- <dia:boolean val="false"/>
- </dia:attribute>
- <dia:attribute name="line_style">
- <dia:enum val="1"/>
- </dia:attribute>
- <dia:attribute name="corner_radius">
- <dia:real val="1.1754943508222875e-38"/>
- </dia:attribute>
- </dia:object>
- <dia:object type="Standard - Box" version="0" id="O1">
- <dia:attribute name="obj_pos">
- <dia:point val="17.5,13"/>
- </dia:attribute>
- <dia:attribute name="obj_bb">
- <dia:rectangle val="17.45,12.95;30.55,32.05"/>
- </dia:attribute>
- <dia:attribute name="elem_corner">
- <dia:point val="17.5,13"/>
- </dia:attribute>
- <dia:attribute name="elem_width">
- <dia:real val="13"/>
- </dia:attribute>
- <dia:attribute name="elem_height">
- <dia:real val="19"/>
- </dia:attribute>
- <dia:attribute name="show_background">
- <dia:boolean val="false"/>
- </dia:attribute>
- <dia:attribute name="line_style">
- <dia:enum val="1"/>
- </dia:attribute>
- <dia:attribute name="corner_radius">
- <dia:real val="1.1754943508222875e-38"/>
- </dia:attribute>
- </dia:object>
- <dia:object type="Flowchart - Box" version="0" id="O2">
- <dia:attribute name="obj_pos">
- <dia:point val="19,16"/>
- </dia:attribute>
- <dia:attribute name="obj_bb">
- <dia:rectangle val="18.95,15.95;29.045,21.05"/>
- </dia:attribute>
- <dia:attribute name="elem_corner">
- <dia:point val="19,16"/>
- </dia:attribute>
- <dia:attribute name="elem_width">
- <dia:real val="9.9949999999999992"/>
- </dia:attribute>
- <dia:attribute name="elem_height">
- <dia:real val="5"/>
- </dia:attribute>
- <dia:attribute name="show_background">
- <dia:boolean val="true"/>
- </dia:attribute>
- <dia:attribute name="padding">
- <dia:real val="0.5"/>
- </dia:attribute>
- <dia:attribute name="text">
- <dia:composite type="text">
- <dia:attribute name="string">
- <dia:string>#CKAN
-(The Registry)
-+Web User Interface (WUI)
-+Machine Web API (WAPI)#</dia:string>
- </dia:attribute>
- <dia:attribute name="font">
- <dia:font family="sans" style="0" name="Helvetica"/>
- </dia:attribute>
- <dia:attribute name="height">
- <dia:real val="0.80000000000000004"/>
- </dia:attribute>
- <dia:attribute name="pos">
- <dia:point val="23.9975,17.4425"/>
- </dia:attribute>
- <dia:attribute name="color">
- <dia:color val="#000000"/>
- </dia:attribute>
- <dia:attribute name="alignment">
- <dia:enum val="1"/>
- </dia:attribute>
- </dia:composite>
- </dia:attribute>
- </dia:object>
- <dia:object type="Flowchart - Box" version="0" id="O3">
- <dia:attribute name="obj_pos">
- <dia:point val="8.065,21"/>
- </dia:attribute>
- <dia:attribute name="obj_bb">
- <dia:rectangle val="8.015,20.95;13.985,24.05"/>
- </dia:attribute>
- <dia:attribute name="elem_corner">
- <dia:point val="8.065,21"/>
- </dia:attribute>
- <dia:attribute name="elem_width">
- <dia:real val="5.8699999999999992"/>
- </dia:attribute>
- <dia:attribute name="elem_height">
- <dia:real val="3"/>
- </dia:attribute>
- <dia:attribute name="show_background">
- <dia:boolean val="true"/>
- </dia:attribute>
- <dia:attribute name="padding">
- <dia:real val="0.5"/>
- </dia:attribute>
- <dia:attribute name="text">
- <dia:composite type="text">
- <dia:attribute name="string">
- <dia:string>#Is It Open
-isitopendata.org#</dia:string>
- </dia:attribute>
- <dia:attribute name="font">
- <dia:font family="sans" style="0" name="Helvetica"/>
- </dia:attribute>
- <dia:attribute name="height">
- <dia:real val="0.80000000000000004"/>
- </dia:attribute>
- <dia:attribute name="pos">
- <dia:point val="11,22.2425"/>
- </dia:attribute>
- <dia:attribute name="color">
- <dia:color val="#000000"/>
- </dia:attribute>
- <dia:attribute name="alignment">
- <dia:enum val="1"/>
- </dia:attribute>
- </dia:composite>
- </dia:attribute>
- </dia:object>
- <dia:object type="Flowchart - Box" version="0" id="O4">
- <dia:attribute name="obj_pos">
- <dia:point val="21,28"/>
- </dia:attribute>
- <dia:attribute name="obj_bb">
- <dia:rectangle val="20.95,27.95;27.05,31.05"/>
- </dia:attribute>
- <dia:attribute name="elem_corner">
- <dia:point val="21,28"/>
- </dia:attribute>
- <dia:attribute name="elem_width">
- <dia:real val="6"/>
- </dia:attribute>
- <dia:attribute name="elem_height">
- <dia:real val="3"/>
- </dia:attribute>
- <dia:attribute name="show_background">
- <dia:boolean val="true"/>
- </dia:attribute>
- <dia:attribute name="padding">
- <dia:real val="0.5"/>
- </dia:attribute>
- <dia:attribute name="text">
- <dia:composite type="text">
- <dia:attribute name="string">
- <dia:string>#CKAN CLIENT#</dia:string>
- </dia:attribute>
- <dia:attribute name="font">
- <dia:font family="sans" style="0" name="Helvetica"/>
- </dia:attribute>
- <dia:attribute name="height">
- <dia:real val="0.80000000000000004"/>
- </dia:attribute>
- <dia:attribute name="pos">
- <dia:point val="24,29.6425"/>
- </dia:attribute>
- <dia:attribute name="color">
- <dia:color val="#000000"/>
- </dia:attribute>
- <dia:attribute name="alignment">
- <dia:enum val="1"/>
- </dia:attribute>
- </dia:composite>
- </dia:attribute>
- </dia:object>
- <dia:object type="Flowchart - Box" version="0" id="O5">
- <dia:attribute name="obj_pos">
- <dia:point val="33,28"/>
- </dia:attribute>
- <dia:attribute name="obj_bb">
- <dia:rectangle val="32.95,27.95;42.05,31.05"/>
- </dia:attribute>
- <dia:attribute name="elem_corner">
- <dia:point val="33,28"/>
- </dia:attribute>
- <dia:attribute name="elem_width">
- <dia:real val="9"/>
- </dia:attribute>
- <dia:attribute name="elem_height">
- <dia:real val="3"/>
- </dia:attribute>
- <dia:attribute name="show_background">
- <dia:boolean val="true"/>
- </dia:attribute>
- <dia:attribute name="padding">
- <dia:real val="0.5"/>
- </dia:attribute>
- <dia:attribute name="text">
- <dia:composite type="text">
- <dia:attribute name="string">
- <dia:string>#DATAPKG
-(like apt-get,gem,pip)#</dia:string>
- </dia:attribute>
- <dia:attribute name="font">
- <dia:font family="sans" style="0" name="Helvetica"/>
- </dia:attribute>
- <dia:attribute name="height">
- <dia:real val="0.80000000000000004"/>
- </dia:attribute>
- <dia:attribute name="pos">
- <dia:point val="37.5,29.2425"/>
- </dia:attribute>
- <dia:attribute name="color">
- <dia:color val="#000000"/>
- </dia:attribute>
- <dia:attribute name="alignment">
- <dia:enum val="1"/>
- </dia:attribute>
- </dia:composite>
- </dia:attribute>
- </dia:object>
- <dia:object type="Flowchart - Box" version="0" id="O6">
- <dia:attribute name="obj_pos">
- <dia:point val="32,16"/>
- </dia:attribute>
- <dia:attribute name="obj_bb">
- <dia:rectangle val="31.95,15.95;43.05,21.05"/>
- </dia:attribute>
- <dia:attribute name="elem_corner">
- <dia:point val="32,16"/>
- </dia:attribute>
- <dia:attribute name="elem_width">
- <dia:real val="11"/>
- </dia:attribute>
- <dia:attribute name="elem_height">
- <dia:real val="5"/>
- </dia:attribute>
- <dia:attribute name="show_background">
- <dia:boolean val="true"/>
- </dia:attribute>
- <dia:attribute name="padding">
- <dia:real val="0.5"/>
- </dia:attribute>
- <dia:attribute name="text">
- <dia:composite type="text">
- <dia:attribute name="string">
- <dia:string>#STORAGE
-(The Payload)
-archive.org / knowledgeforge.net
-repositories (svn/hg/git/...)#</dia:string>
- </dia:attribute>
- <dia:attribute name="font">
- <dia:font family="sans" style="0" name="Helvetica"/>
- </dia:attribute>
- <dia:attribute name="height">
- <dia:real val="0.80000000000000004"/>
- </dia:attribute>
- <dia:attribute name="pos">
- <dia:point val="37.5,17.4425"/>
- </dia:attribute>
- <dia:attribute name="color">
- <dia:color val="#000000"/>
- </dia:attribute>
- <dia:attribute name="alignment">
- <dia:enum val="1"/>
- </dia:attribute>
- </dia:composite>
- </dia:attribute>
- </dia:object>
- <dia:object type="Flowchart - Ellipse" version="0" id="O7">
- <dia:attribute name="obj_pos">
- <dia:point val="22,22.5"/>
- </dia:attribute>
- <dia:attribute name="obj_bb">
- <dia:rectangle val="21.95,22.45;26.05,26.2338"/>
- </dia:attribute>
- <dia:attribute name="elem_corner">
- <dia:point val="22,22.5"/>
- </dia:attribute>
- <dia:attribute name="elem_width">
- <dia:real val="4.0000000000000009"/>
- </dia:attribute>
- <dia:attribute name="elem_height">
- <dia:real val="3.6838104413780557"/>
- </dia:attribute>
- <dia:attribute name="show_background">
- <dia:boolean val="true"/>
- </dia:attribute>
- <dia:attribute name="padding">
- <dia:real val="0.35355339059327379"/>
- </dia:attribute>
- <dia:attribute name="text">
- <dia:composite type="text">
- <dia:attribute name="string">
- <dia:string>#API
-(JSON)#</dia:string>
- </dia:attribute>
- <dia:attribute name="font">
- <dia:font family="sans" style="0" name="Helvetica"/>
- </dia:attribute>
- <dia:attribute name="height">
- <dia:real val="0.80000000000000004"/>
- </dia:attribute>
- <dia:attribute name="pos">
- <dia:point val="24,24.0844"/>
- </dia:attribute>
- <dia:attribute name="color">
- <dia:color val="#000000"/>
- </dia:attribute>
- <dia:attribute name="alignment">
- <dia:enum val="1"/>
- </dia:attribute>
- </dia:composite>
- </dia:attribute>
- </dia:object>
- <dia:object type="Standard - ZigZagLine" version="1" id="O8">
- <dia:attribute name="obj_pos">
- <dia:point val="24,27.9496"/>
- </dia:attribute>
- <dia:attribute name="obj_bb">
- <dia:rectangle val="23.95,26.1636;24.05,27.9496"/>
- </dia:attribute>
- <dia:attribute name="orth_points">
- <dia:point val="24,27.9496"/>
- <dia:point val="24,27.9496"/>
- <dia:point val="24,26.2343"/>
- <dia:point val="24,26.2343"/>
- </dia:attribute>
- <dia:attribute name="orth_orient">
- <dia:enum val="0"/>
- <dia:enum val="1"/>
- <dia:enum val="0"/>
- </dia:attribute>
- <dia:attribute name="autorouting">
- <dia:boolean val="true"/>
- </dia:attribute>
- <dia:attribute name="end_arrow">
- <dia:enum val="22"/>
- </dia:attribute>
- <dia:attribute name="end_arrow_length">
- <dia:real val="0.5"/>
- </dia:attribute>
- <dia:attribute name="end_arrow_width">
- <dia:real val="0.5"/>
- </dia:attribute>
- <dia:connections>
- <dia:connection handle="0" to="O4" connection="16"/>
- <dia:connection handle="1" to="O7" connection="16"/>
- </dia:connections>
- </dia:object>
- <dia:object type="Standard - ZigZagLine" version="1" id="O9">
- <dia:attribute name="obj_pos">
- <dia:point val="32.9497,29.5"/>
- </dia:attribute>
- <dia:attribute name="obj_bb">
- <dia:rectangle val="26.9797,29.45;32.9497,29.55"/>
- </dia:attribute>
- <dia:attribute name="orth_points">
- <dia:point val="32.9497,29.5"/>
- <dia:point val="32.9497,29.5"/>
- <dia:point val="27.0504,29.5"/>
- <dia:point val="27.0504,29.5"/>
- </dia:attribute>
- <dia:attribute name="orth_orient">
- <dia:enum val="0"/>
- <dia:enum val="1"/>
- <dia:enum val="0"/>
- </dia:attribute>
- <dia:attribute name="autorouting">
- <dia:boolean val="true"/>
- </dia:attribute>
- <dia:attribute name="end_arrow">
- <dia:enum val="22"/>
- </dia:attribute>
- <dia:attribute name="end_arrow_length">
- <dia:real val="0.5"/>
- </dia:attribute>
- <dia:attribute name="end_arrow_width">
- <dia:real val="0.5"/>
- </dia:attribute>
- <dia:connections>
- <dia:connection handle="0" to="O5" connection="16"/>
- <dia:connection handle="1" to="O4" connection="16"/>
- </dia:connections>
- </dia:object>
- <dia:object type="Standard - ZigZagLine" version="1" id="O10">
- <dia:attribute name="obj_pos">
- <dia:point val="29.0453,18.5"/>
- </dia:attribute>
- <dia:attribute name="obj_bb">
- <dia:rectangle val="29.0453,18.45;32.0204,18.55"/>
- </dia:attribute>
- <dia:attribute name="orth_points">
- <dia:point val="29.0453,18.5"/>
- <dia:point val="29.0453,18.5"/>
- <dia:point val="31.9497,18.5"/>
- <dia:point val="31.9497,18.5"/>
- </dia:attribute>
- <dia:attribute name="orth_orient">
- <dia:enum val="0"/>
- <dia:enum val="1"/>
- <dia:enum val="0"/>
- </dia:attribute>
- <dia:attribute name="autorouting">
- <dia:boolean val="true"/>
- </dia:attribute>
- <dia:attribute name="start_arrow">
- <dia:enum val="22"/>
- </dia:attribute>
- <dia:attribute name="start_arrow_length">
- <dia:real val="0.5"/>
- </dia:attribute>
- <dia:attribute name="start_arrow_width">
- <dia:real val="0.5"/>
- </dia:attribute>
- <dia:attribute name="end_arrow">
- <dia:enum val="22"/>
- </dia:attribute>
- <dia:attribute name="end_arrow_length">
- <dia:real val="0.5"/>
- </dia:attribute>
- <dia:attribute name="end_arrow_width">
- <dia:real val="0.5"/>
- </dia:attribute>
- <dia:connections>
- <dia:connection handle="0" to="O2" connection="16"/>
- <dia:connection handle="1" to="O6" connection="16"/>
- </dia:connections>
- </dia:object>
- <dia:object type="Standard - ZigZagLine" version="1" id="O11">
- <dia:attribute name="obj_pos">
- <dia:point val="37.5,27.9504"/>
- </dia:attribute>
- <dia:attribute name="obj_bb">
- <dia:rectangle val="37,20.9994;38,28.0004"/>
- </dia:attribute>
- <dia:attribute name="orth_points">
- <dia:point val="37.5,27.9504"/>
- <dia:point val="37.5,25"/>
- <dia:point val="37.5,25"/>
- <dia:point val="37.5,21.0494"/>
- </dia:attribute>
- <dia:attribute name="orth_orient">
- <dia:enum val="1"/>
- <dia:enum val="0"/>
- <dia:enum val="1"/>
- </dia:attribute>
- <dia:attribute name="autorouting">
- <dia:boolean val="false"/>
- </dia:attribute>
- <dia:attribute name="end_arrow">
- <dia:enum val="22"/>
- </dia:attribute>
- <dia:attribute name="end_arrow_length">
- <dia:real val="0.5"/>
- </dia:attribute>
- <dia:attribute name="end_arrow_width">
- <dia:real val="0.5"/>
- </dia:attribute>
- <dia:connections>
- <dia:connection handle="0" to="O5" connection="16"/>
- <dia:connection handle="1" to="O6" connection="16"/>
- </dia:connections>
- </dia:object>
- <dia:object type="Standard - Text" version="1" id="O12">
- <dia:attribute name="obj_pos">
- <dia:point val="25,11"/>
- </dia:attribute>
- <dia:attribute name="obj_bb">
- <dia:rectangle val="15.0375,9.21016;34.9966,12.7898"/>
- </dia:attribute>
- <dia:attribute name="text">
- <dia:composite type="text">
- <dia:attribute name="string">
- <dia:string>#The "Debian" of Data
-Automating use and reuse of data#</dia:string>
- </dia:attribute>
- <dia:attribute name="font">
- <dia:font family="sans" style="0" name="Helvetica"/>
- </dia:attribute>
- <dia:attribute name="height">
- <dia:real val="1.6000000000000001"/>
- </dia:attribute>
- <dia:attribute name="pos">
- <dia:point val="25,10.3292"/>
- </dia:attribute>
- <dia:attribute name="color">
- <dia:color val="#000000"/>
- </dia:attribute>
- <dia:attribute name="alignment">
- <dia:enum val="1"/>
- </dia:attribute>
- </dia:composite>
- </dia:attribute>
- <dia:attribute name="valign">
- <dia:enum val="2"/>
- </dia:attribute>
- </dia:object>
- <dia:object type="Standard - ZigZagLine" version="1" id="O13">
- <dia:attribute name="obj_pos">
- <dia:point val="24,22.4495"/>
- </dia:attribute>
- <dia:attribute name="obj_bb">
- <dia:rectangle val="23.4975,21.0003;24.4975,22.4995"/>
- </dia:attribute>
- <dia:attribute name="orth_points">
- <dia:point val="24,22.4495"/>
- <dia:point val="24,21.7499"/>
- <dia:point val="23.9975,21.7499"/>
- <dia:point val="23.9975,21.0503"/>
- </dia:attribute>
- <dia:attribute name="orth_orient">
- <dia:enum val="1"/>
- <dia:enum val="0"/>
- <dia:enum val="1"/>
- </dia:attribute>
- <dia:attribute name="autorouting">
- <dia:boolean val="true"/>
- </dia:attribute>
- <dia:attribute name="end_arrow">
- <dia:enum val="22"/>
- </dia:attribute>
- <dia:attribute name="end_arrow_length">
- <dia:real val="0.5"/>
- </dia:attribute>
- <dia:attribute name="end_arrow_width">
- <dia:real val="0.5"/>
- </dia:attribute>
- <dia:connections>
- <dia:connection handle="0" to="O7" connection="16"/>
- <dia:connection handle="1" to="O2" connection="16"/>
- </dia:connections>
- </dia:object>
- <dia:object type="Standard - Text" version="1" id="O14">
- <dia:attribute name="obj_pos">
- <dia:point val="24,22.5"/>
- </dia:attribute>
- <dia:attribute name="obj_bb">
- <dia:rectangle val="24,22.1;24,23.3"/>
- </dia:attribute>
- <dia:attribute name="text">
- <dia:composite type="text">
- <dia:attribute name="string">
- <dia:string>##</dia:string>
- </dia:attribute>
- <dia:attribute name="font">
- <dia:font family="sans" style="0" name="Helvetica"/>
- </dia:attribute>
- <dia:attribute name="height">
- <dia:real val="0.80000000000000004"/>
- </dia:attribute>
- <dia:attribute name="pos">
- <dia:point val="24,22.5"/>
- </dia:attribute>
- <dia:attribute name="color">
- <dia:color val="#000000"/>
- </dia:attribute>
- <dia:attribute name="alignment">
- <dia:enum val="0"/>
- </dia:attribute>
- </dia:composite>
- </dia:attribute>
- <dia:attribute name="valign">
- <dia:enum val="3"/>
- </dia:attribute>
- <dia:connections>
- <dia:connection handle="0" to="O1" connection="8"/>
- </dia:connections>
- </dia:object>
- <dia:object type="Standard - Text" version="1" id="O15">
- <dia:attribute name="obj_pos">
- <dia:point val="24,22.5"/>
- </dia:attribute>
- <dia:attribute name="obj_bb">
- <dia:rectangle val="24,22.1;24,23.3"/>
- </dia:attribute>
- <dia:attribute name="text">
- <dia:composite type="text">
- <dia:attribute name="string">
- <dia:string>##</dia:string>
- </dia:attribute>
- <dia:attribute name="font">
- <dia:font family="sans" style="0" name="Helvetica"/>
- </dia:attribute>
- <dia:attribute name="height">
- <dia:real val="0.80000000000000004"/>
- </dia:attribute>
- <dia:attribute name="pos">
- <dia:point val="24,22.5"/>
- </dia:attribute>
- <dia:attribute name="color">
- <dia:color val="#000000"/>
- </dia:attribute>
- <dia:attribute name="alignment">
- <dia:enum val="0"/>
- </dia:attribute>
- </dia:composite>
- </dia:attribute>
- <dia:attribute name="valign">
- <dia:enum val="3"/>
- </dia:attribute>
- <dia:connections>
- <dia:connection handle="0" to="O1" connection="8"/>
- </dia:connections>
- </dia:object>
- <dia:object type="Standard - Text" version="1" id="O16">
- <dia:attribute name="obj_pos">
- <dia:point val="20,15"/>
- </dia:attribute>
- <dia:attribute name="obj_bb">
- <dia:rectangle val="20,14.1875;27.865,15.595"/>
- </dia:attribute>
- <dia:attribute name="text">
- <dia:composite type="text">
- <dia:attribute name="string">
- <dia:string>#The CKAN System#</dia:string>
- </dia:attribute>
- <dia:attribute name="font">
- <dia:font family="sans" style="0" name="Helvetica"/>
- </dia:attribute>
- <dia:attribute name="height">
- <dia:real val="1.2"/>
- </dia:attribute>
- <dia:attribute name="pos">
- <dia:point val="20,15"/>
- </dia:attribute>
- <dia:attribute name="color">
- <dia:color val="#000000"/>
- </dia:attribute>
- <dia:attribute name="alignment">
- <dia:enum val="0"/>
- </dia:attribute>
- </dia:composite>
- </dia:attribute>
- <dia:attribute name="valign">
- <dia:enum val="3"/>
- </dia:attribute>
- </dia:object>
- <dia:object type="Standard - Text" version="1" id="O17">
- <dia:attribute name="obj_pos">
- <dia:point val="11,22"/>
- </dia:attribute>
- <dia:attribute name="obj_bb">
- <dia:rectangle val="11,21.6;11,22.8"/>
- </dia:attribute>
- <dia:attribute name="text">
- <dia:composite type="text">
- <dia:attribute name="string">
- <dia:string>##</dia:string>
- </dia:attribute>
- <dia:attribute name="font">
- <dia:font family="sans" style="0" name="Helvetica"/>
- </dia:attribute>
- <dia:attribute name="height">
- <dia:real val="0.80000000000000004"/>
- </dia:attribute>
- <dia:attribute name="pos">
- <dia:point val="11,22"/>
- </dia:attribute>
- <dia:attribute name="color">
- <dia:color val="#000000"/>
- </dia:attribute>
- <dia:attribute name="alignment">
- <dia:enum val="0"/>
- </dia:attribute>
- </dia:composite>
- </dia:attribute>
- <dia:attribute name="valign">
- <dia:enum val="3"/>
- </dia:attribute>
- <dia:connections>
- <dia:connection handle="0" to="O0" connection="8"/>
- </dia:connections>
- </dia:object>
- <dia:object type="Standard - Text" version="1" id="O18">
- <dia:attribute name="obj_pos">
- <dia:point val="8,20"/>
- </dia:attribute>
- <dia:attribute name="obj_bb">
- <dia:rectangle val="8,19.3013;14.0938,20.5375"/>
- </dia:attribute>
- <dia:attribute name="text">
- <dia:composite type="text">
- <dia:attribute name="string">
- <dia:string>#Related Services#</dia:string>
- </dia:attribute>
- <dia:attribute name="font">
- <dia:font family="sans" style="0" name="Helvetica"/>
- </dia:attribute>
- <dia:attribute name="height">
- <dia:real val="1"/>
- </dia:attribute>
- <dia:attribute name="pos">
- <dia:point val="8,20"/>
- </dia:attribute>
- <dia:attribute name="color">
- <dia:color val="#000000"/>
- </dia:attribute>
- <dia:attribute name="alignment">
- <dia:enum val="0"/>
- </dia:attribute>
- </dia:composite>
- </dia:attribute>
- <dia:attribute name="valign">
- <dia:enum val="3"/>
- </dia:attribute>
- </dia:object>
- <dia:object type="Standard - ZigZagLine" version="1" id="O19">
- <dia:attribute name="obj_pos">
- <dia:point val="13.9854,22.5"/>
- </dia:attribute>
- <dia:attribute name="obj_bb">
- <dia:rectangle val="13.9354,18;19.05,22.55"/>
- </dia:attribute>
- <dia:attribute name="orth_points">
- <dia:point val="13.9854,22.5"/>
- <dia:point val="16.4927,22.5"/>
- <dia:point val="16.4927,18.5"/>
- <dia:point val="19,18.5"/>
- </dia:attribute>
- <dia:attribute name="orth_orient">
- <dia:enum val="0"/>
- <dia:enum val="1"/>
- <dia:enum val="0"/>
- </dia:attribute>
- <dia:attribute name="autorouting">
- <dia:boolean val="true"/>
- </dia:attribute>
- <dia:attribute name="end_arrow">
- <dia:enum val="22"/>
- </dia:attribute>
- <dia:attribute name="end_arrow_length">
- <dia:real val="0.5"/>
- </dia:attribute>
- <dia:attribute name="end_arrow_width">
- <dia:real val="0.5"/>
- </dia:attribute>
- <dia:connections>
- <dia:connection handle="0" to="O3" connection="16"/>
- <dia:connection handle="1" to="O2" connection="7"/>
- </dia:connections>
- </dia:object>
- </dia:layer>
-</dia:diagram>
--- a/doc/conf.py Thu Jul 28 16:26:50 2011 +0100
+++ b/doc/conf.py Thu Jul 28 17:20:40 2011 +0100
@@ -29,7 +29,7 @@
extensions = ['sphinx.ext.autodoc']
# Add any paths that contain templates here, relative to this directory.
-templates_path = ['.templates']
+templates_path = ['_templates']
# The suffix of source filenames.
source_suffix = '.rst'
@@ -41,8 +41,10 @@
master_doc = 'index'
# General information about the project.
-project = u'Comprehensive Knowledge Archive Network (CKAN)'
+project = u'CKAN (Comprehensive Knowledge Archive Network)'
+project_short_name = u'CKAN'
copyright = u'2009, Open Knowledge Foundation'
+html_show_sphinx = False
# The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the
@@ -92,31 +94,40 @@
# Options for HTML output
# -----------------------
+html_theme = 'default'
+html_theme_options = {
+"relbarbgcolor": "#777",
+'sidebarbgcolor': '#F2F2F2',
+'sidebartextcolor': 'black',
+'sidebarlinkcolor': '#355F7C',
+'headfont': 'Trebuchet MS'
+}
+
# The style sheet to use for HTML and HTML Help pages. A file of that name
# must exist either in Sphinx' static/ path, or in one of the custom paths
# given in html_static_path.
-html_style = 'default.css'
+#html_style = 'default.css'
# The name for this set of Sphinx documents. If None, it defaults to
# "<project> v<release> documentation".
-#html_title = None
+html_title = "%s v%s Administration Guide" % (project, release)
# A shorter title for the navigation bar. Default is the same as html_title.
-#html_short_title = None
+html_short_title = "%s Admin Guide" % (project_short_name)
# The name of an image file (relative to this directory) to place at the top
# of the sidebar.
-#html_logo = None
+html_logo = 'images/ckan_logo_box.png'
# The name of an image file (within the static path) to use as favicon of the
# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32
# pixels large.
-#html_favicon = None
+html_favicon = 'images/favicon.ico'
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
-html_static_path = ['.static']
+#html_static_path = ['.static']
# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
# using the given strftime format.
--- a/doc/configuration.rst Thu Jul 28 16:26:50 2011 +0100
+++ b/doc/configuration.rst Thu Jul 28 17:20:40 2011 +0100
@@ -1,23 +1,26 @@
-CKAN Configuration
-==================
+.. index::
+ single: config file
-A CKAN instance is configured with the .ini file in the root ckan directory. The most important parameter to set is the `sqlalchemy.url` giving the database connection. But there are also several additional options to change the way the CKAN site operates.
+=====================================
+Reference: CKAN Configuration Options
+=====================================
-On a production machine the file will be probably be named after the site name. e.g. `ca.ckan.net.ini` On a development machine it is probably `development.ini`
+You can change many important CKAN settings in the CKAN config file. This is the file called ``std.ini`` that you first encountered in :ref:`create-admin-user`. It is usually located at ``/etc/ckan/std/std.ini``.
-On a new installation of ckan, you have to create a new config file from the template, something like this::
+The file is well-documented, but we recommend reading this section in full to learn about the CKAN config options available to you.
- paster --plugin ckan make-config ckan ca.ckan.net.ini
+.. note:: After editing this file, you will need to restart Apache for the changes to take effect.
-This creates ca.ckan.net.ini based on the template for this file in ckan/config/deployment.ini_tmpl
+.. note:: The CKAN config file also includes general Pylons options. All CKAN-specific settings are in the `[app:main]` section.
-There are several general Pylons options, and all the CKAN-specific ones are in the `[app:main]` section.
+Database Settings
+-----------------
-Once the config file is changed, Apache needs to be restarted to read in the new changes.
-
+.. index::
+ single: sqlalchemy.url
sqlalchemy.url
---------------
+^^^^^^^^^^^^^^
Example::
@@ -28,20 +31,47 @@
sqlalchemy.url = postgres://USERNAME:PASSWORD@HOST/DBNAME
-package_form
-------------
+Front-End Settings
+------------------
+
+
+.. index::
+ single: site_description
+
+site_description
+^^^^^^^^^^^^^^^^
Example::
- package_form = ca
+ ckan.site_description=
-Default value: ``standard``
+Default value: (none)
-This sets the name of the form to use when editing a package. This can be a form defined in the core CKAN code or in another setuputils-managed python module. The only requirement is that the setup.py has an entrypoint for the form defined in the `ckan.forms` section. See :doc:`forms`
+This is for a description, or tag line for the site, as displayed in the header of the CKAN web interface.
+.. index::
+ single: site_logo
+
+site_logo
+^^^^^^^^^
+
+Example::
+
+ ckan.site_logo=/images/ckan_logo_fullname_long.png
+
+Default value: (none)
+
+This sets the logo used in the title bar.
+
+.. index::
+ single: site_url
+
+
+.. index::
+ single: package_hide_extras
package_hide_extras
--------------------
+^^^^^^^^^^^^^^^^^^^
Example::
@@ -49,12 +79,15 @@
Default value: (empty)
-This sets a space-seperated list of extra field key values which will not be shown on the package read page. While this is useful to create internal notes etc., it is not a security measure in any way. The keys will
-still be available via the API and in revision diffs.
+This sets a space-separated list of extra field key values which will not be shown on the package read page.
+.. warning:: While this is useful to e.g. create internal notes, it is not a security measure. The keys will still be available via the API and in revision diffs.
+
+.. index::
+ single: rdf_packages
rdf_packages
-------------
+^^^^^^^^^^^^
Example::
@@ -68,9 +101,30 @@
3. A visible RDF link on the page. e.g. `<a href="http://semantic.ckan.net/record/b410e678-8a96-40cf-8e46-e8bd4bf02684.rdf">`
+.. index::
+ single: dumps_url, dumps_format
+
+dumps_url & dumps_format
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+Example::
+
+ ckan.dumps_url = http://ckan.net/dump/
+ ckan.dumps_format = CSV/JSON
+
+If there is a page which allows you to download a dump of the entire catalogue then specify the URL and the format here, so that it can be advertised in the web interface. ``dumps_format`` is just a string for display.
+
+For more information on using dumpfiles, see :doc:`database_dumps`.
+
+
+Cache Settings
+--------------
+
+.. index::
+ single: cache_validation_enabled
cache_validation_enabled
-------------------------
+^^^^^^^^^^^^^^^^^^^^^^^^
Example::
@@ -78,13 +132,15 @@
Default value: ``True``
-This option determines whether browsers (or other caching services running between the browser and CKAN) are helped to cache particular CKAN pages, by validating when the page content hasn't changed. This is achieved using ETags headers provided by CKAN, which is a hash that changes when the content has changed.
+This option determines whether browsers (or other caching services running between the browser and CKAN) are helped to cache particular CKAN pages, by validating when the page content hasn't changed. This is achieved using ETag headers provided by CKAN, which is a hash that changes when the content has changed.
-Developers editing the templates should set this to False, since Etags hashes don't look for template changes.
+Developers editing the templates should set this to False, since ETag hashes don't look for template changes.
+.. index::
+ single: cache_enabled
cache_enabled
--------------
+^^^^^^^^^^^^^
Example::
@@ -92,13 +148,13 @@
Default value: ``False``
-Setting this option to True turns on several server-side caches. When the caching is on, caching can be further configured as follows. (The key has been renamed from ``cache_enabled``, which is deprecated, but still works ``ckan.cache_enabled`` for now.)
+Setting this option to True turns on several server-side caches. When caching is on, caching can be further configured as follows.
To set the type of Beaker storage::
beaker.cache.type = file
-To set the expiry times (in seconds) for specific controllers (which use the proxy_cache) specifiy the methods like this::
+To set the expiry times (in seconds) for specific controllers (which use the proxy_cache) specify the methods like this::
ckan.controllers.package.list.expires = 600
ckan.controllers.tag.read.expires = 600
@@ -107,13 +163,19 @@
ckan.controllers.apiv2.package.list.expires = 600
ckan.controllers.apiv2.package.show.expires = 600
-There is also en option to set the max-age value of static files delivered by
-paster::
+There is also an option to set the max-age value of static files delivered by paster::
ckan.static_max_age = 3600
+
+Authentication Settings
+-----------------------
+
+.. index::
+ single: openid_enabled
+
openid_enabled
---------------
+^^^^^^^^^^^^^^
Example::
@@ -121,33 +183,21 @@
Default value: ``True``
-Setting this option to Fase turns off openid for login.
+CKAN operates a delegated authentication model based on `OpenID <http://openid.net/>`_.
+Setting this option to False turns off OpenID for login.
-licenses_group_url
-------------------
-A url pointing to a JSON file containing a list of license objects. This list
-determines the licenses offered by the system to users, for example when
-creating or editing a package.
+.. _config-i18n:
-This is entirely optional -- by default the system will use the ckan list of
-licenses available in the Licenses package.
+Internationalisation Settings
+-----------------------------
-.. _licenses python package: http://pypi.python.org/pypi/licenses
-
-More details about the license objects including the license format and some
-example license lists can be found on the open license service at
-http://licenses.opendefinition.org/.
-
-Examples::
-
- licenses_group_url = file:///path/to/my/local/json-list-of-licenses.js
- licenses_group_url = http://licenses.opendefinition.org/2.0/ckan_original
-
+.. index::
+ single: lang
lang
-----
+^^^^
Example::
@@ -155,58 +205,108 @@
Default value: ``en`` (English)
-Use this to specify the language of the text displayed in the CKAN web UI. This requires a suitable `mo` file installed for the language. For more information on internationalization, see: http://wiki.okfn.org/ckan/i18n#DeployingaTranslation
+Use this to specify the language of the text displayed in the CKAN web UI. This requires a suitable `mo` file installed for the language. For more information on internationalization, see :doc:`i18n`.
+Theming Settings
+----------------
+
+.. index::
+ single: extra_template_paths
extra_template_paths
---------------------
+^^^^^^^^^^^^^^^^^^^^
Example::
extra_template_paths=/home/okfn/brazil_ckan_config/templates
-To customise the display of CKAN you can supply replacements for the Genshi template files. Use this option to specify where CKAN should look for them, before reverting to the 'ckan/templates' folder. You can supply more than one folder, separating the paths with a comma (,).
+To customise the display of CKAN you can supply replacements for the Genshi template files. Use this option to specify where CKAN should look for additional templates, before reverting to the ``ckan/templates`` folder. You can supply more than one folder, separating the paths with a comma (,).
-The example value for the extra_template_paths option could, for example, be used to override CKAN templates with these ones:
+For more information on theming, see :doc:`theming`.
- * /home/okfn/brazil_ckan_config/templates/layout.html
- * /home/okfn/brazil_ckan_config/templates/package/edit.html
-
-More details about this feature are found at: http://wiki.okfn.org/ckan/doc/theme
-
+.. index::
+ single: extra_public_paths
extra_public_paths
-------------------
+^^^^^^^^^^^^^^^^^^
Example::
extra_public_paths = /home/okfn/brazil_ckan_config/public
-To customise the display of CKAN you can supply replacements for staticly served files such as HTML, CSS, script and PNG files. Use this option to specify where CKAN should look for them, before reverting to the 'ckan/public' folder. You can supply more than one folder, separating the paths with a comma (,).
+To customise the display of CKAN you can supply replacements for static files such as HTML, CSS, script and PNG files. Use this option to specify where CKAN should look for additional files, before reverting to the ``ckan/public`` folder. You can supply more than one folder, separating the paths with a comma (,).
-The example value for the extra_public_paths option could, for example, be used to provide an image and stylesheet:
+For more information on theming, see :doc:`theming`.
- * /home/okfn/brazil_ckan_config/public/images/brazil.png
- * /home/okfn/brazil_ckan_config/public/css/extra.css
-More details about this feature are found at: http://wiki.okfn.org/ckan/doc/theme
+Form Settings
+-------------
+.. index::
+ single: package_form
+
+package_form
+^^^^^^^^^^^^
+
+Example::
+
+ package_form = ca
+
+Default value: ``standard``
+
+This sets the name of the form to use when editing a package. This can be a form defined in the core CKAN code or in another setuputils-managed python module. The only requirement is that the ``setup.py`` file has an entry point for the form defined in the ``ckan.forms`` section.
+
+For more information on forms, see :doc:`forms`.
+
+.. _config-package-urls:
+
+.. index::
+ single: package_new_return_url, package_edit_return_url
package_new_return_url & package_edit_return_url
-------------------------------------------------
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Example::
package_new_return_url = http://datadotgc.ca/new_dataset_complete?name=<NAME>
package_edit_return_url = http://datadotgc.ca/dataset/<NAME>
-To allow the Edit Package and New Package forms to be integrated into a third party interface, setting these options allows you to set a the return address. So when the user has completed the form and presses 'commit', the user is redirected to the URL specified.
+If integrating the Edit Package and New Package forms into a third-party interface, setting these options allows you to set the return address. When the user has completed the form and presses 'commit', the user is redirected to the URL specified.
-The '<NAME>' string is replaced with the name of the package edited. Full details of this process are given in :doc:`form-integration`.
+The ``<NAME>`` string is replaced with the name of the package edited. Full details of this process are given in :doc:`form-integration`.
+.. index::
+ single: licenses_group_url
+
+licenses_group_url
+^^^^^^^^^^^^^^^^^^
+
+A url pointing to a JSON file containing a list of licence objects. This list
+determines the licences offered by the system to users, for example when
+creating or editing a package.
+
+This is entirely optional - by default, the system will use the CKAN list of
+licences available in the `Python licenses package <http://pypi.python.org/pypi/licenses>`_.
+
+More details about the CKAN license objects - including the licence format and some
+example licence lists - can be found at the `Open Licenses Service
+<http://licenses.opendefinition.org/>`_.
+
+Examples::
+
+ licenses_group_url = file:///path/to/my/local/json-list-of-licenses.js
+ licenses_group_url = http://licenses.opendefinition.org/2.0/ckan_original
+
+
+Messaging Settings
+------------------
+
+.. index::
+ single: carrot_messaging_library
+
carrot_messaging_library
-------------------------
+^^^^^^^^^^^^^^^^^^^^^^^^
Example::
@@ -222,11 +322,13 @@
* ``queue`` - native Python Queue (default) - NB this doesn't work inter-process
-See `carrot documentation <http://packages.python.org/carrot/index.html>`_ for details.
+See the `Carrot documentation <http://packages.python.org/carrot/index.html>`_ for details.
+.. index::
+ single: amqp_hostname, amqp_port, amqp_user_id, amqp_password
amqp_hostname, amqp_port, amqp_user_id, amqp_password
------------------------------------------------------
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Example::
@@ -235,11 +337,16 @@
amqp_user_id=guest
amqp_password=guest
-These are the setup parameters for AMQP messaging. These only apply if the messageing library has been set to use AMQP (see `carrot_messaging_library`_). The values given in the example are the default values.
+These are the setup parameters for AMQP messaging. These only apply if the messaging library has been set to use AMQP (see `carrot_messaging_library`_). The values given above are the default values.
+Search Settings
+---------------
+
+.. index::
+ single: build_search_index_synchronously
build_search_index_synchronously
---------------------------------
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Example::
@@ -248,11 +355,15 @@
Default (if you don't define it)::
indexing is on
-This controls the operation of the CKAN Postgres full text search indexing. If you don't define this option then indexing is on. You will want to turn this off if you want to use a different search engine for CKAN (e.g. SOLR). In this case you need to define the option equal to blank (as in the given example).
+This controls the operation of the CKAN Postgres full text search indexing. If you don't define this option then indexing is on. You will want to turn this off if you want to use a different search engine for CKAN (e.g. Solr). In this case you need to define the option equal to blank (as in the example).
+.. _config-search-backend:
search_backend
---------------
+^^^^^^^^^^^^^^
+
+.. index::
+ single: search_backend
Example::
@@ -260,24 +371,31 @@
Default value: ``sql``
-This controls the type of search backend. Currently valid values are ``sql`` (meaning Postgres full text search) and ``solr``. If you specify ``sql`` then ensure indexing is on (`build_search_index_synchronously`_ is not defined). If you specify ``solr`` then ensure you specify a `solr_url`_.
+This controls the type of search backend. Currently valid values are ``sql`` (meaning Postgres full text search) and ``solr`` (meaning Solr). If you specify ``sql`` then ensure indexing is on (`build_search_index_synchronously`_ is not defined). If you specify ``solr`` then ensure you specify a `solr_url`_.
+.. index::
+ single: solr_url
solr_url
---------
+^^^^^^^^
Example::
solr_url = http://solr.okfn.org/solr/test.ckan.net
-This configures SOLR search, (if selected with 'search_backend'_). Running solr will require a schema.xml file, such as the one
-in `the ckan-solr-index repository <http://bitbucket.org/pudo/ckan-solr-index>`_.
+This configures Solr search (if selected with `search_backend`_). Running Solr will require a schema.xml file, such as the one in `the ckanext-solr repository <https://bitbucket.org/okfn/ckanext-solr/src>`_.
-Optionally, ``solr_user`` and ``solr_password`` can also be passed along to specify HTTP Basic authentication details for all solr requests.
+Optionally, ``solr_user`` and ``solr_password`` can also be passed along to specify HTTP Basic authentication details for all Solr requests.
+Site Settings
+-------------
+
+.. index::
+ single: site_title
+
site_title
-----------
+^^^^^^^^^^
Example::
@@ -287,33 +405,11 @@
This sets the name of the site, as displayed in the CKAN web interface.
-
-site_description
-----------------
-
-Example::
-
- ckan.site_description=
-
-Default value: (none)
-
-This is for a description, or tag line for the site, as displayed in the header of the CKAN web interface.
-
-
-site_logo
----------
-
-Example::
-
- ckan.site_logo=/images/ckan_logo_fullname_long.png
-
-Default value: (none)
-
-This sets the logo used in the title bar.
-
+.. index::
+ single: site_url
site_url
---------
+^^^^^^^^
Example::
@@ -321,13 +417,13 @@
Default value: (none)
-The primary URL used by this site. Uses::
+The primary URL used by this site. Used in the API to provide packages with links to themselves in the web UI.
- * in the API to provide packages with links to themselves in the web UI.
-
+.. index::
+ single: api_url
api_url
---------
+^^^^^^^
Example::
@@ -335,71 +431,91 @@
Default value: ``/api``
-The URL which resolves to the CKAN API part of the site. This is useful if the
-API is hosted on a different domain, for example when a third party site uses
+The URL that resolves to the CKAN API part of the site. This is useful if the
+API is hosted on a different domain, for example when a third-party site uses
the forms API.
+Authorization Settings
+----------------------
+
+.. index::
+ single: default_roles
+
default_roles
--------------
+^^^^^^^^^^^^^
-This allows you to set the default authorization roles (i.e. permissions) for new objects. Currently this extends to new packages, groups, authorization groups and the 'system' object. For full details of these, see :doc:`authorization`.
+This allows you to set the default authorization roles (i.e. permissions) for new objects. Currently this extends to new packages, groups, authorization groups and the ``system`` object. For full details of these, see :doc:`authorization`.
-The value is a strict JSON dictionary of user names "visitor" and "logged_in" with lists of their roles.
+The value is a strict JSON dictionary of user names ``visitor`` (any user who is not logged in) and ``logged_in`` (any user who is logged in) with lists of their roles.
Example::
ckan.default_roles.Package = {"visitor": ["editor"], "logged_in": ["editor"]}
ckan.default_roles.Group = {"visitor": ["reader"], "logged_in": ["reader"]}
-With this example setting, visitors (any user who is not logged in) and logged in users can only read packages that get created (only sysadmins can edit).
+With this example setting, visitors and logged-in users can only read packages that get created.
-Defaults: see in ckan/model/authz.py for: ``default_default_user_roles``
+Defaults: see in ``ckan/model/authz.py`` for: ``default_default_user_roles``
+Plugin Settings
+---------------
+
+.. index::
+ single: plugins
+
plugins
--------
+^^^^^^^
Example::
ckan.plugins = disqus synchronous_search datapreview googleanalytics stats storage admin follower
-Specify which CKAN extensions are to be enabled. If you specify an extension but have not installed the code then CKAN will not start. Format in a space separated list of the extension names. The extension name is the key in the [ckan.plugins] section of the extension's setup.py.
+Specify which CKAN extensions are to be enabled.
+.. warning:: If you specify an extension but have not installed the code, CKAN will not start.
-dumps_url & dumps_format
-------------------------
+Format as a space-separated list of the extension names. The extension name is the key in the [ckan.plugins] section of the extension's ``setup.py``. For more information on extensions, see :doc:`extensions`.
-Example::
- ckan.dumps_url = http://ckan.net/dump/
- ckan.dumps_format = CSV/JSON
-If there is a page which allows you to download a dump of the entire catalogue then specify the URL and the format here, so that it can be advertised in the web interface. The dumps_format is just a string for display.
+Directory Settings
+------------------
+.. index::
+ single: log_dir
log_dir
--------
+^^^^^^^
Example::
ckan.log_dir = /var/log/ckan/
-This is a directory where CKAN cron scripts (if there are any installed) should write log files to. Note: this setting is nothing to do with the main CKAN log file, whose filepath is set in the [handler_file] args.
+This is the directory to which CKAN cron scripts (if there are any installed) should write log files.
+.. note:: This setting is nothing to do with the main CKAN log file, whose filepath is set in the ``[handler_file]`` args.
+
+.. index::
+ single: dump_dir
dump_dir
---------
+^^^^^^^^
Example::
ckan.dump_dir = /var/lib/ckan/dump/
-This is a directory where JSON or CSV dumps of the database are to be written, assuming a script has been installed to do this. Note it is usual to setup the apache config to serve this directory.
+This is the directory to which JSON or CSV dumps of the database are to be written, assuming a script has been installed to do this.
+.. note:: It is usual to set up the Apache config to serve this directory.
+
+.. index::
+ single: backup_dir
backup_dir
-----------
+^^^^^^^^^^
Example::
--- a/doc/database_dumps.rst Thu Jul 28 16:26:50 2011 +0100
+++ b/doc/database_dumps.rst Thu Jul 28 17:20:40 2011 +0100
@@ -1,24 +1,47 @@
-Database dumps
+Database Dumps
==============
-It's often useful to allow users to download the complete CKAN database in a dumpfile. For example, you can download ckan.net daily dump at: http://ckan.net/dump/ in JSON format: ckan.net-daily.json.gz
+It's often useful to allow users to download a complete CKAN database in a dumpfile.
+For example, you can download ckan.net's daily dump at: http://ckan.net/dump/ in JSON format. The file is called ``ckan.net-daily.json.gz``.
-Creating a dump
+Creating a Dump
-----------------
-Your dump script needs to enable the python environment and run the paster command. For example you could create `/home/okfn/var/srvc/ckan.net/dump.sh`::
+We provide two ``paster`` methods to create dumpfiles.
+
+* ``db simple-dump-json`` - A simple dumpfile, useful to create a public listing of the packages with no user information. All packages are dumped, including deleted packages and ones with strict authorization.
+* ``db dump`` - A more complicated dumpfile, useful for backups. Replicates the database completely, including users, their personal info and API keys, and hence should be kept private.
+
+For more information on paster, see :doc:`paster`.
+
+Using db simple-dump-json
++++++++++++++++++++++++++
+
+If you are using a Python environment, as part of a development installation, first enable the environment::
. /home/okfn/var/srvc/ckan.net/pyenv/bin/activate || exit 1
- paster --plugin=ckan db simple-dump-json /home/okfn/var/srvc/ckan.net/dumps/ckan.net-daily.json --config=/home/okfn/var/srvc/ckan.net/ckan.net.ini
- gzip /home/okfn/var/srvc/ckan.net/dumps/ckan.net-daily.json
-You could change simple-dump-json to simple-dump-csv if you want CSV format instead of JSON. Or you could run both!
+Then create and zip the dumpfile::
-These dump functions dump the entire database as it is stored in CKAN, omitting user account details.
+ paster --plugin=ckan db simple-dump-json /var/srvc/ckan/dumps/ckan.net-daily.json --config=/etc/ckan/std/std.ini
+ gzip /var/srvc/ckan/dumps/ckan.net-daily.json
+Change ``simple-dump-json`` to ``simple-dump-csv`` if you want CSV format instead of JSON.
-Daily dumps
+Using db dump
++++++++++++++
+
+If you are using a Python environment, as part of a development installation, first enable the environment::
+
+ . /var/srvc/ckan/pyenv/bin/activate || exit 1
+
+Then create and zip the dumpfile::
+
+ paster --plugin=ckan db dump /var/srvc/ckan/dumps/ckan.net-daily --config=/etc/ckan/std/std.ini
+ gzip /var/srvc/ckan/dumps/ckan.net-daily
+
+Daily Dumps
-----------
You can set the dump to be created daily with a cron job.
@@ -31,13 +54,12 @@
0 21 * * * /home/okfn/var/srvc/ckan.net/dump.sh
-
-Serving the files
+Serving the Files
-----------------
Some simple additions to the Apache config can serve the files to users in a directory listing.
-To do this, add these lines to the virtual host config (e.g. `/etc/apache2/sites-enabled/ckan.net`)::
+To do this, add these lines to your virtual host config (e.g. ``/etc/apache2/sites-enabled/ckan.net``)::
Alias /dump/ /home/okfn/var/srvc/ckan.net/dumps/
@@ -46,4 +68,3 @@
SetHandler None
Options +Indexes
</Location>
-
--- a/doc/deb.rst Thu Jul 28 16:26:50 2011 +0100
+++ /dev/null Thu Jan 01 00:00:00 1970 +0000
@@ -1,902 +0,0 @@
-CKAN's Approach to Dependencies
-+++++++++++++++++++++++++++++++
-
-.. WARNING::
- This document is still under development, use only if you are a member
- of the CKAN team who wishes to be an early adopter and are interested in
- experimenting with virtual machines.
-
-.. contents ::
-
-Abstract
-========
-
-A typical CKAN install can have many dependencies, in the form of required
-system software such as PostgreSQL, Python libraries such as SQLAlchemy and
-other CKAN extensions such as ``ckanext-qa`` or ``ckanext-harvest``.
-
-As such we have to deal with lots of interdependencies which are often
-different depending on the combinations of features a particular CKAN install
-requires.
-
-There are three audiences we are primarily targetting that our dependency
-approach is designed to support:
-
-* Interested parties who just want to get a CKAN instance installed quickly
- and easily to test it or to begin contributing
-* CKAN developers who want to have editable versions of all of the dependant
- libraries so that they can improve them
-* System administrators who want to be able to deploy and upgrade CKAN
- instances quickly, easily and reliably
-* Deployment and test managers who want to be confident that the live system
- is running off exactly the libraries they have tested and configured in
- exactly the same way
-
-In order to support these three groups, we allow installation of CKAN in two ways:
-
-* Development install via ``pip`` as described in the `README.rst <../README.html>`_ file
-* Production install on Ubuntu Lucid 10.04 LTS via ``apt-get install``
-
-The old instructions for a `production deployment <../deployment.html>`_ can
-also be followed but these will be deprecated over time as the ``.deb``
-packaging approach becomes better documented and understood.
-
-Virutally all CKAN instances currently run on Ubuntu Lucid 10.04 LTS so we have
-decided that this is the only officially supported production deployment
-platform. Of course CKAN will also run on any modern Mac or Linux distribution
-but if you go down this route you'll need to do a development install.
-
-In this document I'll explain in detail how and why we package CKAN the way we
-do.
-
-
-Overview of CKAN's Structure from a Packaging Point of View
-===========================================================
-
-There are conceptually two main parts to any CKAN installation:
-
-* Python libraries such as CKAN itself, SQLAlchemy, owslib and all the CKAN
- extensions libraries that a particular CKAN or customized CKAN rely on
-* The configuration and deployment scripts that lead to a running CKAN
- application (which may be a vanilla CKAN or a client-specific version (eg
- CKAN, CKAN-DGU, CKAN-DATANL etc)
-* Third party servers that CKAN relies on (eg Apache, PostgreSQL etc)
-
-Luckily all third party servers that CKAN relies on are all ready packaged in
-Ubuntu Lucid for us so we don't need to worry about packaging them. We do need
-to worry about configuring them though. Let's look more at the other two.
-
-Python Libraries
- The Python libraries CKAN relies on all have their own interdependencies.
- If the libraries have already been packaged as ``.deb`` files we don't need to worry about
- them because their dependencies will already be specified in the package.
- Other libraries usually have their dependencies specified as the ``install_requires`` line in their
- ``setup.py`` files. For the sorts of libraries that are already available
-
-
-Configuration and Deployment
-
-
-Understanding The Dependency Difficulty
----------------------------------------
-
-In the past
-
-
-
-The Three Different Requires Files for CKAN
-===========================================
-
-In order to support both a development install and a package-based install it
-is important that we package the same versions of libraries that we develop
-against. There are three categories of dependant Python libraries:
-
-``present``
- Those that already exist as packages in Ubuntu Lucid
-
-``missing``
- Those that don't exist as packages in Ubuntu Lucid
-
-``conflict``
- Those that have a version which is different from the version in Ubuntu
- Lucid
-
-For each of these categories we have a file in the ``ckan`` source tree's
-``requires`` directory which you can view `here
-<https://bitbucket.org/okfn/ckan/src/default/requires/>`_.
-
-
-Understanding the ``lucid_present.txt`` File
---------------------------------------------
-
-The Python dependencies listed in the ``lucid_present.txt`` file are ``pip``
-installable links to the source tree holding the exact versions of the Python
-dependencies that Ubuntu uses. By running the command below you get development
-copies of the same software that Ubuntu has packaged:
-
-::
-
- pip install --ignore-installed -r lucid_present.txt
-
-We never need to package software in the ``lucid_present.txt`` file because it
-already exists so most of the time you would just install it directly rather
-than running the command above to get source versions. You can see the packages
-you would need to install by looking at the comment at the top of the file. At
-the time of writing it reads:
-
-::
-
- # The CKAN dependencies are already in Lucid and should be installed via
- # apt-get if you are on that platform. If you are using a different platform
- # you can install these dependencies via pip instead.
- #
- # sudo apt-get install python-psycopg2 python-lxml python-sphinx
- # sudo apt-get install python-pylons python-formalchemy python-repoze.who
- # sudo apt-get install python-repoze.who-plugins python-tempita python-zope.interface
-
-Packaging Dependencies Listed in ``lucid_missing.txt``
-------------------------------------------------------
-
-.. note ::
-
- These are already packaged, so you don't need to package them yourself, this
- section just describes how you *could* do if you wanted to.
-
-Python dependencies listed in the ``lucid_missing.txt`` file are ``pip``
-installable links to the source tree holding the exact versions of the Python
-dependencies that CKAN requries. We have an automatic build process which can
-take these entries and automatically generate Ubuntu packages for them. The
-resulting packages are then published to our CKAN apt repository so that they
-can be automatically installed in production environments.
-
-To follow the automatic build process to build the missing packages you can do this:
-
-
-::
-
- sudo apt-get install -y python wget dh-make devscripts build-essential fakeroot cdbs mercurial git-core subversion python-virtualenv
- virtualenv missing
- bin/pip install --ignore-installed -r lucid_missing.txt
- bin/pip install Buildkit
-
-BuildKit script will build and place Debian packages in your ``missing``
-directory. Make sure there is nothing in there that shouldn't be overwritten by
-this script.
-
-Now run the BuildKit command like this:
-
-::
-
- cd missing
- bin/python -m buildkit.update_all .
-
-For each package you'll be loaded into ``vim`` to edit the changelog. Save and
-quit when you are done. Names, version numbers and dependencies are
-automatically generated.
-
-.. caution ::
-
- Most of the time you will never use the automatic process above for lazy
- batch packaging. You'll more likely generate a single package with explicit
- version numbers using the ``buildkit.deb`` command or build your package
- manually. Both approaches are described later.
-
-Packaging Conflicting Python dependencies from ``lucid_conflicts.txt``
-----------------------------------------------------------------------
-
-.. note ::
-
- These are already packaged, so you don't need to package them yourself, this
- section just describes how you *could* do if you wanted to.
-
-Python packages where CKAN depends on a version that is different from the one
-in the Ubuntu Lucid repositories are handled slightly differently. If we were
-to simply package them up and make them available the same way we do with
-missing packages there is a slim chance that any existing software which used
-the other version of the library would stop working. To avoid the risk of
-interfering with other software on the system we take the following approach:
-
-* Create a ``python-ckan-deps`` package with copies of all the libraries we need
-* Change the ``python-ckan`` library to automatically try to import
- ``ckan_deps`` if it can and then adjust the Python's ``sys.path`` just for
- this instance to use the versions of the libraries in ``python-ckan-deps`` in
- preference to any other versions installed.
-
-In this way we can use any arbitrary versions, without introducing conflicts.
-
-.. caution ::
-
- The ``repoze.who`` sets of libraries are nigh-on impossible to package in
- this way so we don't actually package ``repoze.who.openid`` at all, even
- though we need a slightly more recent version. This is such an edge case
- though that you should just install it manually into the system Python
- and not worry too much for the time being.
-
-To actually build the ``python-ckan-deps`` package we follow the semi-manual
-Python packaging approach described next. (The example in the next section is
-actually for a CKAN Python extension called ``python-ckanext-qa`` but the same
-process applies).
-
-hg clone ckan-deps
-
-::
-
- python -m buildkit.deb /path/to/ckan-deps/.. python-ckan-deps 1.3~01+lucid http://ckan.org
-
-
-Semi-Manual Python Packaging
-============================
-
-The easiest way to package a Python library is with a tool called BuildKit I
-wrote specfically for the purpose. This section describes how to use it, but
-even if you don't want to use BuildKit and prefer to understand the
-complexities yourself by reading the `Understanding .deb files`_ section,
-please read this section too so you at least understand the naming conventions
-we are using.
-
-::
-
- pip install buildkit
-
-For each Python package you wish to build a ``.deb`` file for you run the
-``buildkit.deb`` command. Here's an example:
-
-::
-
- python -m buildkit.deb /path/to/virtualenv ckanext-qa 1.3~01+lucid http://ckan.org python-owslib python-ckanext-csw
-
-Let's break this down.
-
-``python -m buildkit.deb``
- This is just how you invoke the command from the command line
-
-``/path/to/virtualenv``
- I think this can just be the path to the directory containing the
- installed source code directory you wish to package, it doesn't
- have to be a virtualenv does it?
-
-``ckanext-qa``
- The lowercase Python package name of the ``.deb`` file to be created.
-
-
-``1.3~01+lucid``
- The version number of the package. There are three parts to this:
-x
- ``1.3``
- This should always match exactly the version number specified in the
- ``setup.py`` file for the library being packaged.
-
- ``~01``
- This is an incrementing number (starting at 01 each time the version
- number above changes) which you change every time you re-package the
- same version of the code to force apt to recognise your new package
- as being more recent than the old one, even if the underlying code
- hasn't changed.
-
- ``+lucid``
- This is a string representing the Debian/Ubuntu distribution that the
- package targets. The apt repository doesn't assign any meaning to it,
- it is just that in order to eventually support more than one flavour
- of Debian or Ubuntu, the packages for each must have different
- filenames *in addition* to being in a separate part of the apt repo
- so we begin this convention now.
-
-``http://ckan.org``
- The homepage for the package, usually ckan.org for ckan extensions.
-
-``python-owslib python-ckanext-csw ... etc``
-
- Any extra arguments are treated as the Debian names of dependencies. These
- always begin ``python-`` for Python libraries and would usually follow
- ``ckanext-`` for all CKAN extensions.
-
- .. tip ::
-
- You can also specify any other Debian
- packages here that are dependcies of the software you are packaging but as
- you'll see later it is usually best to add such dependencies to the
- *packaged application*. See "Packaging CKAN Extensions" for more information.
-
-When you run the command you will get your ``.deb`` file created.
-
-To release an upgrade of a package it must have a higher version number. There
-is a chance you may want to release a more recent version of a package despite
-the fact the underlying version number hasn't changed. For this reason, we
-always add a ``~`` character followed by a two digit number to the end of the
-actual version number as specified in ``setup.py`` for the package.
-
-For example, if the version number for the ``ckanext-qa`` package in the
-example above is ``1.3~01``, a package named
-``python-ckanext-qa_1.3~01_amd64.deb`` would be produced by the command we've
-looked at.
-
-.. note ::
-
- All packages that CKAN itself depends on are already packaged according to
- the settings in the three ``requires`` files that from part of the ``ckan``
- source distribution so you shouldn't need to use the approach above to
- package any of them, you should only need to do this for your own extensions
- or libraries they rely on which aren't already CKAN dependencies. See
- "The Three Different Requires Files" for more information on how packaging
- of the core CKAN dependencies is managed.
-
-Understanding ``.deb`` files
-============================
-
-Broad Structure
----------------
-
-Naming Conventions
-------------------
-
-The base naming conventions we use for packages are as follows:
-
-``ckan``
- Uninstalls CKAN, PostgreSQL, Apache etc. It adds the ``ckan-instance-create`` command which is then the only thing you need to create a new instance.
-
-``python-ckan``
- The CKAN Python library packaged from code at http://bitbucket.org/okfn/ckan
-
-``python-ckanext-*``
- Any CKAN extensions (can be application extensions or library extensions)
-
-``ckan-*``
- Installs a client specific CKAN application
-
-
-
-The ``postinst`` and ``postrm`` files
--------------------------------------
-
-The ``control`` file
---------------------
-
-Extra scripts and permissions
------------------------------
-
-Packaging Python libraries
---------------------------
-
-
-
-Packaging CKAN Extensions
-=========================
-
-There are two types of CKAN extension:
-
-* Client Applications (eg ``ckanext-dgu``, ``ckanext-datanl`` etc)
-* Helpful libraries (eg ``ckanext-qa``, ``ckanext-harvest``, ``ckanext-queue`` etc)
-
-All CKAN extensions (whether client applications or helpful libraries) are
-Python libraries and therefore need packaging. Their ``.deb`` filenames are the
-same as the Python package names but are always prefixed with ``python-`` so
-that ``ckanext-dgu`` becomes ``python-ckanext-dgu`` when packaged as a ``.deb``
-and ``ckanext-harvest`` becomes ``python-ckanext-harvest`` etc.
-
-CKAN extensions which are also client applications generally need to be
-deployed and therefore need require Apache and PostgreSQL to be installed and
-configured correctly too. In addition to the *python* package we therefore also
-create an *application* package for the extension which is named ``ckan-``
-followed by the last part of the extension name. So for ``ckanext-dgu`` two
-packages are created named ``python-ckanext-dgu`` and ``ckan-dgu``. This naming
-may sound slightly inconsistent but it allows a user who wishes to install a
-DGU CKAN instance to just type the command below:
-
-::
-
- sudo apt-get install ckan-dgu
-
-Usually the ``ckan`` package will be a dependency of the your client
-application CKAN extension. When the ``ckan`` package is installed it installs
-``python-ckan`` as a dependency as well as a series of scripts in ``/usr/bin``
-such as:
-
-``ckan-create-instance``
- create a new CKAN instance
-
-``ckan-maintenance-mode``
- put a CKAN intance into or out of maintenence mode (prevent POSTs from
- the web user interface)
-
-In the simple cases, these scripts can then be used in your client application
-CKAN extension's ``postinst`` script to set up the custom instance. In more
-complex cases you may write a ``postinst`` script from scratch. The
-``postinst`` script then forms part of the package and is run by the apt system
-as part of the package installation or upgrade process to configure your CKAN
-instance.
-
-
-
-
-
-
-
-
-
-
-
-
-Before we look at how to actually create an apt repository for your packages
-and how to publish your packages to it, let's understand what a user of your
-package will do to install it.
-
-Understanding How a User Installs from an apt repository
-========================================================
-
-A user will follow the following process:
-
-First create the file ``/etc/apt/sources.list.d/okfn.list`` using this command, replacing ``ubuntu_ckan_dev`` with the correct repo you want to use:
-
-::
-
- echo "deb http://apt.okfn.org/ubuntu_ckan_dev lucid universe" | sudo tee /etc/apt/sources.list.d/okfn.list
-
-Then add the package key to say you trust packages from this repository:
-
-::
-
- sudo apt-get install wget
- wget -qO- http://apt.okfn.org/packages.okfn.key | sudo apt-key add -
- sudo apt-get update
-
-Now you can not install a CKAN extension application, just like any other Debian package:
-
-::
-
- sudo apt-get install ckan-std
-
-At this point you should have a running instance. You may need to copy across
-an existing database if you need your instance pre-populated with data.
-
-
-Setting up a CKAN Apt Repository
-================================
-
-Now you've seen what a user expects to be able to do, let's set up the
-infrastructure to make to make it happen.
-
-
-From Scratch
-------------
-
-Our set up is based on `Joseph Ruscio's set up
-<http://joseph.ruscio.org/blog/2010/08/19/setting-up-an-apt-repository/>`_ and
-will allow us to support multiple operating systems if we want to as well as
-multiple architectures. At the moment we only support Ubuntu Lucid amd64.
-
-To help with repository management we use the ``reprepro`` tool. Despite the fact that the repositories could be set up for different OSs and versions (eg ``lenny``, ``lucid`` etc) we need to make sure that the package names are still unique. This means that we always add the distribution to the version number when we package.
-
-
-The most important detail that AFAIK isn’t covered in any of the tutorials had to do with package naming conventions. The naive assumption (at least on my part) is that you’ll have a different build of your package for each distro/arch combination, and import them into your repository as such. In other words reprepro should track the distro/arch of each import. In actuality, each build’s <PACKAGE>_<VERSION>_<ARCH> must be unique, even though you specify the distro during the includedeb operation.
-
-
-
-
-The Easy Way
-------------
-
-Log into the existing CKAN apt repository server and copy an existing directory
-that already contains the packages you need. For example, to create a
-repository for a new ``ckanext-dgu`` instance you might do:
-
-::
-
- cd /var/packages/
- cp -pr lucid dgu-new
-
-At this point you have a brand new repo, you can add new packages to it like this:
-
-::
-
- cd dgu-new
- sudo reprepro includedeb lucid ~/*.deb
-
-You can remove them like this from the same directory:
-
-::
-
- sudo reprepro remove lucid python-ckan
-
-Any time a change is made you will need to enter the passphrase for the key.
-
-
-Automatic Packaging
-===================
-
-The BuildKit script will build and place Debian packages in your ``missing``
-directory. Make sure there is nothing in there that shouldn't be overwritten by
-this script.
-
-
-Adding a Package to a Repository
-================================
-
-
-Packaging CKAN Itself
-=====================
-
-
-
-
-Why We use ``pip`` rather than ``install_requires``
-===================================================
-
-
-Packaging CKAN and its dependencies for a production install
-============================================================
-
-Installing a Packaged CKAN-based Site
-=====================================
-
-Testing Your Packaging in a VM
-==============================
-
-The Release Process
-===================
-
-
-
-
-
-Creating the CKAN Command
-=========================
-
-Create a directory named ``ckan``. Then within it create a ``DEBIAN`` directory with three files:
-
-``control``:
-
- ::
-
- Package: ckan
- Version: 1.3.2~09
- Architecture: amd64
- Maintainer: James Gardner <james.gardner at okfn.org>
- Installed-Size: 0
- Depends: python-ckan
- Recommends: postgresql, curl
- Section: main/web
- Priority: extra
- Homepage: http://ckan.org
- Description: ckan
- The Data Hub
-
-``postinst``:
-
- ::
-
- #!/bin/sh
- set -e
- # Any commands that happen after install or upgrade go here
-
-``postrm``
-
- ::
-
- #!/bin/sh
- set -e
- # Any commands that happen after removal or before upgrade go here
-
-Then in the ``ckan`` directory you add any files you want copied. In this case
-we want a ``/usr/bin/ckan-create-instance`` script so we create the ``usr``
-directory in the ``ckan`` directory at the same level as the ``DEBIAN``
-directory, then create the ``bin`` directory within it and add the script in
-there.
-
-Finally we want to package up the ``.deb`` file. From within the ``ckan``
-directory run this:
-
-::
-
- dpkg-deb -b . ..
-
-This will create the ``../ckan_1.3.2~09_amd64.deb`` package ready for you to
-upload to the repo.
-
-The ``ckan`` package is already created so in reality you will usually be
-packaging ``ckan-<instance>``. If you make sure your package depends on
-``ckan`` and ``python-ckanext-<instance>`` you can then call the ``ckan``
-package's ``ckan-create-instance`` command in your ``ckan-<instance>``'s
-``postinst`` command to set up Apache and PostgreSQL for the instance
-automatically.
-
-
-Setting up the Repositories
-===========================
-
-Build individual dependencies like this:
-
-::
-
- python -m buildkit.deb . ckanext-importlib 0.1~02 http://ckan.org python-ckan
- python -m buildkit.deb . owslib 0.3.2beta~03 http://ckan.org python-lxml
-
- python -m buildkit.deb . ckanext-inspire 0.1~03 http://ckan.org python-ckan
- python -m buildkit.deb . ckanext-spatial 0.1~04 http://ckan.org python-ckan
- python -m buildkit.deb . ckanext-harvest 0.1~15 http://ckan.org python-ckan python-ckanext-spatial python-carrot
- python -m buildkit.deb . ckanext-csw 0.3~10 http://ckan.org python-ckanext-harvest python-owslib python-ckan
- python -m buildkit.deb . ckanext-dgu 0.2~11 http://ckan.org python-ckan python-ckanext-importlib python-ckanext-dgu python-ckanext-csw python-ckan python-ckanext-spatial python-ckanext-inspire
- python -m buildkit.deb . ckanext-qa 0.1~19 http://ckan.org python-ckan
- python -m buildkit.deb . ckan 1.4~01 http://ckan.org python-routes python-vdm python-pylons python-genshi python-sqlalchemy python-repoze.who python-repoze.who-plugins python-pyutilib.component.core python-migrate python-formalchemy python-sphinx python-markupsafe python-setuptools python-psycopg2 python-licenses python-ckan-deps
-
-There's a dependency on postfix. Choose internet site and the default hostname unless you know better.
-
-Once you have packages you'll want to put them in a repo. You can do that as described here:
-
-* http://joseph.ruscio.org/blog/2010/08/19/setting-up-an-apt-repository/
-
-Then add them like this:
-
-::
-
- cd /var/packages/lucid/
- sudo reprepro includedeb lucid ~/*.deb
-
-You can remove them like this from the same directory:
-
-::
-
- sudo reprepro remove lucid python-ckan
-
-Automatic Packaging
-===================
-
-The BuildKit script will build and place Debian packages in your ``missing``
-directory. Make sure there is nothing in there that shouldn't be overwritten by
-this script.
-
-To package everything automatically, run it like this:
-
-::
-
- cd missing
- bin/python -m buildkit.update_all .
-
-For each package you'll be loaded into ``vim`` to edit the changelog. Save and
-quit when you are done. Names, version numbers and dependencies are
-automatically generated.
-
-You should find all your packages nicely created now.
-
-
-Next Steps
-==========
-
-* Delayed updates
-
-
-Proposed Changes to CKAN
-========================
-
-* Change the config file to support file based logging by default
-* Move who.ini into the config
-* Add a ckan/wsgi.py for standard DGU deployment
-* Modify __init__.py to change
-
-
-* No __init__.py in test directory
-
-
-
-ubuntu at ip-10-226-226-132:/var/packages/dgu-uat$ sudo reprepro includedeb lucid /home/ubuntu/release/2011-04-18_01/*.deb
-/home/ubuntu/release/2011-04-18_01/ckan_1.3.2~10_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan' '1.3.2~10' in 'lucid|universe|amd64', as it has already '1.3.4~03'.
-/home/ubuntu/release/2011-04-18_01/ckan_1.3.2~11_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan' '1.3.2~11' in 'lucid|universe|amd64', as it has already '1.3.4~03'.
-/home/ubuntu/release/2011-04-18_01/ckan_1.3.4~01_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan' '1.3.4~01' in 'lucid|universe|amd64', as it has already '1.3.4~03'.
-/home/ubuntu/release/2011-04-18_01/ckan_1.3.4~02_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan' '1.3.4~02' in 'lucid|universe|amd64', as it has already '1.3.4~03'.
-/home/ubuntu/release/2011-04-18_01/ckan_1.3.4~03_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan' '1.3.4~03' in 'lucid|universe|amd64', as it has already '1.3.4~03'.
-/home/ubuntu/release/2011-04-18_01/ckan-dgu_0.2~06_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan-dgu' '0.2~06' in 'lucid|universe|amd64', as it has already '0.2~15'.
-/home/ubuntu/release/2011-04-18_01/ckan-dgu_0.2~07_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan-dgu' '0.2~07' in 'lucid|universe|amd64', as it has already '0.2~15'.
-/home/ubuntu/release/2011-04-18_01/ckan-dgu_0.2~08_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan-dgu' '0.2~08' in 'lucid|universe|amd64', as it has already '0.2~15'.
-/home/ubuntu/release/2011-04-18_01/ckan-dgu_0.2~09_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan-dgu' '0.2~09' in 'lucid|universe|amd64', as it has already '0.2~15'.
-/home/ubuntu/release/2011-04-18_01/ckan-dgu_0.2~11_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan-dgu' '0.2~11' in 'lucid|universe|amd64', as it has already '0.2~15'.
-/home/ubuntu/release/2011-04-18_01/ckan-dgu_0.2~12_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan-dgu' '0.2~12' in 'lucid|universe|amd64', as it has already '0.2~15'.
-/home/ubuntu/release/2011-04-18_01/ckan-dgu_0.2~13_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan-dgu' '0.2~13' in 'lucid|universe|amd64', as it has already '0.2~15'.
-/home/ubuntu/release/2011-04-18_01/ckan-dgu_0.2~14_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan-dgu' '0.2~14' in 'lucid|universe|amd64', as it has already '0.2~15'.
-/home/ubuntu/release/2011-04-18_01/ckan-dgu_0.2~15_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan-dgu' '0.2~15' in 'lucid|universe|amd64', as it has already '0.2~15'.
-/home/ubuntu/release/2011-04-18_01/python-ckan_1.3.4~01-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckan' '1.3.4~01-1' in 'lucid|universe|amd64', as it has already '1.3.4~02-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckan_1.3.4~02-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckan' '1.3.4~02-1' in 'lucid|universe|amd64', as it has already '1.3.4~02-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-csw_0.3~10-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-csw' '0.3~10-1' in 'lucid|universe|amd64', as it has already '0.3~10-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-dgu_0.2~08-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-dgu' '0.2~08-1' in 'lucid|universe|amd64', as it has already '0.2~11-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-dgu_0.2~09-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-dgu' '0.2~09-1' in 'lucid|universe|amd64', as it has already '0.2~11-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-dgu_0.2~10-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-dgu' '0.2~10-1' in 'lucid|universe|amd64', as it has already '0.2~11-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-dgu_0.2~11-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-dgu' '0.2~11-1' in 'lucid|universe|amd64', as it has already '0.2~11-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-harvest_0.1~13-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-harvest' '0.1~13-1' in 'lucid|universe|amd64', as it has already '0.1~15-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-harvest_0.1~14-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-harvest' '0.1~14-1' in 'lucid|universe|amd64', as it has already '0.1~15-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-harvest_0.1~15-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-harvest' '0.1~15-1' in 'lucid|universe|amd64', as it has already '0.1~15-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-inspire_0.1~01-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-inspire' '0.1~01-1' in 'lucid|universe|amd64', as it has already '0.1~03-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-inspire_0.1~02-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-inspire' '0.1~02-1' in 'lucid|universe|amd64', as it has already '0.1~03-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-inspire_0.1~03-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-inspire' '0.1~03-1' in 'lucid|universe|amd64', as it has already '0.1~03-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-qa_0.1~19-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-qa' '0.1~19-1' in 'lucid|universe|amd64', as it has already '0.1~19-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-spatial_0.1~01-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-spatial' '0.1~01-1' in 'lucid|universe|amd64', as it has already '0.1~04-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-spatial_0.1~03-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-spatial' '0.1~03-1' in 'lucid|universe|amd64', as it has already '0.1~04-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-spatial_0.1~04-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-spatial' '0.1~04-1' in 'lucid|universe|amd64', as it has already '0.1~04-1'.
-/home/ubuntu/release/2011-04-18_01/python-owslib_0.3.2beta~03-1_amd64.deb: component guessed as 'universe'
-ERROR: '/home/ubuntu/release/2011-04-18_01/python-owslib_0.3.2beta~03-1_amd64.deb' cannot be included as 'pool/universe/p/python-owslib/python-owslib_0.3.2beta~03-1_amd64.deb'.
-Already existing files can only be included again, if they are the same, but:
-md5 expected: 3f38d2e844c8d6ec15da6ba51910f3e2, got: ee48427eb11f8152f50f6dc93aeb70d4
-sha1 expected: 87cd7724d8d8f0aaeaa24633abd86e02297771d7, got: 8476b1b0e022892ceb8a35f1848818c31d7441bf
-sha256 expected: 4c9937c78be05dfa5b9dfc85f3a26a51ca4ec0a2d44e8bca530a0c85f12ef400, got: ad3f7458d069a9dd268d144577a7932735643056e45d0a30b7460c38e64057d7
-size expected: 57658, got: 57656
-/home/ubuntu/release/2011-04-18_01/python-owslib_0.3.2beta~04-1_amd64.deb: component guessed as 'universe'
-Exporting indices...
-19A05DDEB16777A2 James Gardner (thejimmyg) <james.gardner at okfn.org> needs a passphrase
-Please enter passphrase:
-Deleting files just added to the pool but not used (to avoid use --keepunusednewfiles next time)
-deleting and forgetting pool/universe/c/ckan/ckan_1.3.2~10_amd64.deb
-deleting and forgetting pool/universe/c/ckan/ckan_1.3.2~11_amd64.deb
-deleting and forgetting pool/universe/c/ckan/ckan_1.3.4~01_amd64.deb
-deleting and forgetting pool/universe/c/ckan/ckan_1.3.4~02_amd64.deb
-deleting and forgetting pool/universe/c/ckan-dgu/ckan-dgu_0.2~06_amd64.deb
-deleting and forgetting pool/universe/c/ckan-dgu/ckan-dgu_0.2~07_amd64.deb
-deleting and forgetting pool/universe/c/ckan-dgu/ckan-dgu_0.2~08_amd64.deb
-deleting and forgetting pool/universe/c/ckan-dgu/ckan-dgu_0.2~09_amd64.deb
-deleting and forgetting pool/universe/c/ckan-dgu/ckan-dgu_0.2~11_amd64.deb
-deleting and forgetting pool/universe/c/ckan-dgu/ckan-dgu_0.2~12_amd64.deb
-deleting and forgetting pool/universe/c/ckan-dgu/ckan-dgu_0.2~13_amd64.deb
-deleting and forgetting pool/universe/c/ckan-dgu/ckan-dgu_0.2~14_amd64.deb
-deleting and forgetting pool/universe/p/python-ckan/python-ckan_1.3.4~01-1_amd64.deb
-deleting and forgetting pool/universe/p/python-ckanext-dgu/python-ckanext-dgu_0.2~08-1_amd64.deb
-deleting and forgetting pool/universe/p/python-ckanext-dgu/python-ckanext-dgu_0.2~09-1_amd64.deb
-deleting and forgetting pool/universe/p/python-ckanext-dgu/python-ckanext-dgu_0.2~10-1_amd64.deb
-deleting and forgetting pool/universe/p/python-ckanext-harvest/python-ckanext-harvest_0.1~13-1_amd64.deb
-deleting and forgetting pool/universe/p/python-ckanext-harvest/python-ckanext-harvest_0.1~14-1_amd64.deb
-deleting and forgetting pool/universe/p/python-ckanext-inspire/python-ckanext-inspire_0.1~01-1_amd64.deb
-deleting and forgetting pool/universe/p/python-ckanext-inspire/python-ckanext-inspire_0.1~02-1_amd64.deb
-deleting and forgetting pool/universe/p/python-ckanext-spatial/python-ckanext-spatial_0.1~01-1_amd64.deb
-deleting and forgetting pool/universe/p/python-ckanext-spatial/python-ckanext-spatial_0.1~03-1_amd64.deb
-Not deleting possibly left over files due to previous errors.
-(To keep the files in the still existing index files from vanishing)
-Use dumpunreferenced/deleteunreferenced to show/delete files without references.
-1 files lost their last reference.
-(dumpunreferenced lists such files, use deleteunreferenced to delete them.)
-There have been errors!
-(reverse-i-search)`delete': cat src/pip-^Clete-this-directory.txt
-ubuntu at ip-10-226-226-132:/var/packages/dgu-uat$ sudo reprepro deleteunreferenced --help
-Error: Too many arguments for command 'deleteunreferenced'!
-Syntax: reprepro deleteunreferenced
-There have been errors!
-ubuntu at ip-10-226-226-132:/var/packages/dgu-uat$ sudo reprepro deleteunreferenced
-deleting and forgetting pool/universe/p/python-owslib/python-owslib_0.3.2beta~03-1_amd64.deb
-ubuntu at ip-10-226-226-132:/var/packages/dgu-uat$
-ubuntu at ip-10-226-226-132:/var/packages/dgu-uat$ sudo reprepro includedeb lucid /home/ubuntu/release/2011-04-18_01/*.deb
-/home/ubuntu/release/2011-04-18_01/ckan_1.3.2~10_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan' '1.3.2~10' in 'lucid|universe|amd64', as it has already '1.3.4~03'.
-/home/ubuntu/release/2011-04-18_01/ckan_1.3.2~11_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan' '1.3.2~11' in 'lucid|universe|amd64', as it has already '1.3.4~03'.
-/home/ubuntu/release/2011-04-18_01/ckan_1.3.4~01_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan' '1.3.4~01' in 'lucid|universe|amd64', as it has already '1.3.4~03'.
-/home/ubuntu/release/2011-04-18_01/ckan_1.3.4~02_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan' '1.3.4~02' in 'lucid|universe|amd64', as it has already '1.3.4~03'.
-/home/ubuntu/release/2011-04-18_01/ckan_1.3.4~03_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan' '1.3.4~03' in 'lucid|universe|amd64', as it has already '1.3.4~03'.
-/home/ubuntu/release/2011-04-18_01/ckan-dgu_0.2~06_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan-dgu' '0.2~06' in 'lucid|universe|amd64', as it has already '0.2~15'.
-/home/ubuntu/release/2011-04-18_01/ckan-dgu_0.2~07_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan-dgu' '0.2~07' in 'lucid|universe|amd64', as it has already '0.2~15'.
-/home/ubuntu/release/2011-04-18_01/ckan-dgu_0.2~08_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan-dgu' '0.2~08' in 'lucid|universe|amd64', as it has already '0.2~15'.
-/home/ubuntu/release/2011-04-18_01/ckan-dgu_0.2~09_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan-dgu' '0.2~09' in 'lucid|universe|amd64', as it has already '0.2~15'.
-/home/ubuntu/release/2011-04-18_01/ckan-dgu_0.2~11_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan-dgu' '0.2~11' in 'lucid|universe|amd64', as it has already '0.2~15'.
-/home/ubuntu/release/2011-04-18_01/ckan-dgu_0.2~12_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan-dgu' '0.2~12' in 'lucid|universe|amd64', as it has already '0.2~15'.
-/home/ubuntu/release/2011-04-18_01/ckan-dgu_0.2~13_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan-dgu' '0.2~13' in 'lucid|universe|amd64', as it has already '0.2~15'.
-/home/ubuntu/release/2011-04-18_01/ckan-dgu_0.2~14_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan-dgu' '0.2~14' in 'lucid|universe|amd64', as it has already '0.2~15'.
-/home/ubuntu/release/2011-04-18_01/ckan-dgu_0.2~15_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'ckan-dgu' '0.2~15' in 'lucid|universe|amd64', as it has already '0.2~15'.
-/home/ubuntu/release/2011-04-18_01/python-ckan_1.3.4~01-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckan' '1.3.4~01-1' in 'lucid|universe|amd64', as it has already '1.3.4~02-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckan_1.3.4~02-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckan' '1.3.4~02-1' in 'lucid|universe|amd64', as it has already '1.3.4~02-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-csw_0.3~10-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-csw' '0.3~10-1' in 'lucid|universe|amd64', as it has already '0.3~10-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-dgu_0.2~08-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-dgu' '0.2~08-1' in 'lucid|universe|amd64', as it has already '0.2~11-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-dgu_0.2~09-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-dgu' '0.2~09-1' in 'lucid|universe|amd64', as it has already '0.2~11-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-dgu_0.2~10-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-dgu' '0.2~10-1' in 'lucid|universe|amd64', as it has already '0.2~11-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-dgu_0.2~11-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-dgu' '0.2~11-1' in 'lucid|universe|amd64', as it has already '0.2~11-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-harvest_0.1~13-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-harvest' '0.1~13-1' in 'lucid|universe|amd64', as it has already '0.1~15-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-harvest_0.1~14-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-harvest' '0.1~14-1' in 'lucid|universe|amd64', as it has already '0.1~15-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-harvest_0.1~15-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-harvest' '0.1~15-1' in 'lucid|universe|amd64', as it has already '0.1~15-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-inspire_0.1~01-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-inspire' '0.1~01-1' in 'lucid|universe|amd64', as it has already '0.1~03-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-inspire_0.1~02-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-inspire' '0.1~02-1' in 'lucid|universe|amd64', as it has already '0.1~03-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-inspire_0.1~03-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-inspire' '0.1~03-1' in 'lucid|universe|amd64', as it has already '0.1~03-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-qa_0.1~19-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-qa' '0.1~19-1' in 'lucid|universe|amd64', as it has already '0.1~19-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-spatial_0.1~01-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-spatial' '0.1~01-1' in 'lucid|universe|amd64', as it has already '0.1~04-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-spatial_0.1~03-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-spatial' '0.1~03-1' in 'lucid|universe|amd64', as it has already '0.1~04-1'.
-/home/ubuntu/release/2011-04-18_01/python-ckanext-spatial_0.1~04-1_amd64.deb: component guessed as 'universe'
-Skipping inclusion of 'python-ckanext-spatial' '0.1~04-1' in 'lucid|universe|amd64', as it has already '0.1~04-1'.
-/home/ubuntu/release/2011-04-18_01/python-owslib_0.3.2beta~03-1_amd64.deb: component guessed as 'universe'
-ERROR: '/home/ubuntu/release/2011-04-18_01/python-owslib_0.3.2beta~03-1_amd64.deb' cannot be included as 'pool/universe/p/python-owslib/python-owslib_0.3.2beta~03-1_amd64.deb'.
-Already existing files can only be included again, if they are the same, but:
-md5 expected: 3f38d2e844c8d6ec15da6ba51910f3e2, got: ee48427eb11f8152f50f6dc93aeb70d4
-sha1 expected: 87cd7724d8d8f0aaeaa24633abd86e02297771d7, got: 8476b1b0e022892ceb8a35f1848818c31d7441bf
-sha256 expected: 4c9937c78be05dfa5b9dfc85f3a26a51ca4ec0a2d44e8bca530a0c85f12ef400, got: ad3f7458d069a9dd268d144577a7932735643056e45d0a30b7460c38e64057d7
-size expected: 57658, got: 57656
-/home/ubuntu/release/2011-04-18_01/python-owslib_0.3.2beta~04-1_amd64.deb: component guessed as 'universe'
-Exporting indices...
-19A05DDEB16777A2 James Gardner (thejimmyg) <james.gardner at okfn.org> needs a passphrase
-Please enter passphrase:
-Deleting files just added to the pool but not used (to avoid use --keepunusednewfiles next time)
-deleting and forgetting pool/universe/c/ckan/ckan_1.3.2~10_amd64.deb
-deleting and forgetting pool/universe/c/ckan/ckan_1.3.2~11_amd64.deb
-deleting and forgetting pool/universe/c/ckan/ckan_1.3.4~01_amd64.deb
-deleting and forgetting pool/universe/c/ckan/ckan_1.3.4~02_amd64.deb
-deleting and forgetting pool/universe/c/ckan-dgu/ckan-dgu_0.2~06_amd64.deb
-deleting and forgetting pool/universe/c/ckan-dgu/ckan-dgu_0.2~07_amd64.deb
-deleting and forgetting pool/universe/c/ckan-dgu/ckan-dgu_0.2~08_amd64.deb
-deleting and forgetting pool/universe/c/ckan-dgu/ckan-dgu_0.2~09_amd64.deb
-deleting and forgetting pool/universe/c/ckan-dgu/ckan-dgu_0.2~11_amd64.deb
-deleting and forgetting pool/universe/c/ckan-dgu/ckan-dgu_0.2~12_amd64.deb
-deleting and forgetting pool/universe/c/ckan-dgu/ckan-dgu_0.2~13_amd64.deb
-deleting and forgetting pool/universe/c/ckan-dgu/ckan-dgu_0.2~14_amd64.deb
-deleting and forgetting pool/universe/p/python-ckan/python-ckan_1.3.4~01-1_amd64.deb
-deleting and forgetting pool/universe/p/python-ckanext-dgu/python-ckanext-dgu_0.2~08-1_amd64.deb
-deleting and forgetting pool/universe/p/python-ckanext-dgu/python-ckanext-dgu_0.2~09-1_amd64.deb
-deleting and forgetting pool/universe/p/python-ckanext-dgu/python-ckanext-dgu_0.2~10-1_amd64.deb
-deleting and forgetting pool/universe/p/python-ckanext-harvest/python-ckanext-harvest_0.1~13-1_amd64.deb
-deleting and forgetting pool/universe/p/python-ckanext-harvest/python-ckanext-harvest_0.1~14-1_amd64.deb
-deleting and forgetting pool/universe/p/python-ckanext-inspire/python-ckanext-inspire_0.1~01-1_amd64.deb
-deleting and forgetting pool/universe/p/python-ckanext-inspire/python-ckanext-inspire_0.1~02-1_amd64.deb
-deleting and forgetting pool/universe/p/python-ckanext-spatial/python-ckanext-spatial_0.1~01-1_amd64.deb
-deleting and forgetting pool/universe/p/python-ckanext-spatial/python-ckanext-spatial_0.1~03-1_amd64.deb
-Not deleting possibly left over files due to previous errors.
-(To keep the files in the still existing index files from vanishing)
-Use dumpunreferenced/deleteunreferenced to show/delete files without references.
-1 files lost their last reference.
-(dumpunreferenced lists such files, use deleteunreferenced to delete them.)
-There have been errors!
-(reverse-i-search)`delete': cat src/pip-^Clete-this-directory.txt
-ubuntu at ip-10-226-226-132:/var/packages/dgu-uat$ sudo reprepro deleteunreferenced --help
-Error: Too many arguments for command 'deleteunreferenced'!
-Syntax: reprepro deleteunreferenced
-There have been errors!
-ubuntu at ip-10-226-226-132:/var/packages/dgu-uat$ sudo reprepro deleteunreferenced
-deleting and forgetting pool/universe/p/python-owslib/python-owslib_0.3.2beta~03-1_amd64.deb
-ubuntu at ip-10-226-226-132:/var/packages/dgu-uat$
-
--- a/doc/deployment.rst Thu Jul 28 16:26:50 2011 +0100
+++ /dev/null Thu Jan 01 00:00:00 1970 +0000
@@ -1,280 +0,0 @@
-Production Deployment
-=====================
-
-Here's an example for deploying CKAN to http://demo.ckan.net/ via Apache.
-
-1. Ideally setup the server with Ubuntu.
-
-
-2. Ensure these packages are installed:
- (e.g. sudo apt-get install <package-name>)
-
- ===================== ============================================
- Package Notes
- ===================== ============================================
- mercurial Source control
- python-dev Python interpreter v2.5 - v2.7 and dev headers
- apache2 Web server
- libapache2-mod-python Apache module for python
- libapache2-mod-wsgi Apache module for WSGI
- postgresql PostgreSQL database
- libpq-dev PostgreSQL library
- python-psycopg2 PostgreSQL python module
- python-setuptools Python package management
- python-libxml2 Python XML library
- python-libxslt1 Python XSLT library
- libxml2-dev XML library development files
- libxslt1-dev XSLT library development files
- git-core Git source control (for getting MarkupSafe src)
- subversion Subversion source control (for pyutilib)
- ===================== ============================================
-
- Now use easy_install (which comes with python-setuptools) to install
- these python packages:
- (e.g. sudo easy_install <package-name>)
-
- ===================== ============================================
- Python Package Notes
- ===================== ============================================
- virtualenv Python virtual environment sandboxing
- pip Python installer
- ===================== ============================================
-
- Check that you received:
-
- * virtualenv v1.3 or later
- * pip v0.4 or later
-
-
-NB: Instead of using these manual instructions, steps 3 to 10 can be achieved
-automatically on a remote server by running the fabric deploy script on
-your local machine. You need fabric and python-dev modules installed locally.
-If you don't have the ckan repo checked out locally then download the
-fabfile.py using::
-
- $ wget https://bitbucket.org/okfn/ckan/raw/default/fabfile.py
-
-Now you can then do the deployment with something like::
-
- $ fab config_0:demo.ckan.net,hosts_str=someserver.net,db_pass=my_password deploy
-
-
-3. Setup a PostgreSQL database
-
- List existing databases::
-
- $ sudo -u postgres psql -l
-
- It is advisable to ensure that the encoding of databases is 'UTF8', or
- internationalisation may be a problem. Since changing the encoding of Postgres
- may mean deleting existing databases, it is suggested that this is fixed before
- continuing with the CKAN install.
-
- Create a database user if one doesn't already exist::
-
- $ sudo -u postgres createuser -S -D -R -P <user>
-
- Replace <user> with the unix username whose home directory has the ckan install.
- It should prompt you for a new password for the CKAN data in the database.
-
- Now create the database::
-
- $ sudo -u postgres createdb -O <user> ckandemo
-
-
-4. Create a python virtual environment
-
- In a general user's home directory::
-
- $ mkdir -p ~/var/srvc/demo.ckan.net
- $ cd ~/var/srvc/demo.ckan.net
- $ virtualenv pyenv
- $ . pyenv/bin/activate
-
-
-5. Create the Pylons WSGI script
-
- Create a file ~/var/srvc/demo.ckan.net/pyenv/bin/demo.ckan.net.py as follows (editing the first couple of variables as necessary)::
-
- import os
- instance_dir = '/home/USER/var/srvc/demo.ckan.net'
- config_file = 'demo.ckan.net.ini'
- pyenv_bin_dir = os.path.join(instance_dir, 'pyenv', 'bin')
- activate_this = os.path.join(pyenv_bin_dir, 'activate_this.py')
- execfile(activate_this, dict(__file__=activate_this))
- from paste.deploy import loadapp
- config_filepath = os.path.join(instance_dir, config_file)
- from paste.script.util.logging_config import fileConfig
- fileConfig(config_filepath)
- application = loadapp('config:%s' % config_filepath)
-
-
-6. Install code and dependent packages into the environment
-
- Decide which release of CKAN you want to install. The CHANGELOG.txt has details on the releases. You'll need the exact tag name, and these are listed on the bitbucket page: https://bitbucket.org/okfn/ckan/src and hover over tags to see the options, e.g. ``ckan-1.4``. ::
-
- $ wget https://bitbucket.org/okfn/ckan/raw/ckan-1.4/pip-requirements.txt
-
- Or for the bleeding edge use::
-
- $ wget https://bitbucket.org/okfn/ckan/raw/default/pip-requirements.txt
-
- And now install::
-
- $ pip -E pyenv install -r pip-requirements.txt
-
- If everything goes correctly then you'll finally see: ``Successfully installed``.
-
-
-7. Create CKAN config file
-
- ::
-
- $ paster make-config ckan demo.ckan.net.ini
-
-
-8. Configure CKAN
-
- Edit 'demo.ckan.net.ini' and change the default values as follows:
-
- 8.1. sqlalchemy.url
-
- Set the sqlalchemy.url database connection information using values from step 3.
-
- 8.2. licenses_group_url
-
- Set the licenses_group_url to point to a licenses service. Options
- include: ::
-
- http://licenses.opendefinition.org/2.0/ckan_original
- http://licenses.opendefinition.org/2.0/all_alphabetical
-
- For information about creating your own licenses services, please refer to
- the Python package called 'licenses' (http://pypi.python.org/pypi/licenses).
-
- 8.3. loggers
-
- CKAN can make a log file if you change the ``[loggers]`` section to this::
-
- [loggers]
- keys = root, ckan
-
- [handlers]
- keys = file
-
- [formatters]
- keys = generic
-
- [logger_root]
- level = INFO
- handlers = file
-
- [logger_ckan]
- level = DEBUG
- handlers = file
- qualname = ckan
-
- [handler_file]
- class = handlers.RotatingFileHandler
- formatter = generic
- level = NOTSET
- args = ('/var/log/ckan/demo.ckan.log', 'a', 2048, 3)
-
- [formatter_generic]
- format = %(asctime)s %(levelname)-5.5s [%(name)s] %(message)s
-
-
-9. Initialise database
-
- ::
-
- $ . pyenv/bin/activate
- $ paster --plugin ckan db init --config demo.ckan.net.ini
-
-
-10. Set some permissions for Pylons
-
- Whilst still in the ~/var/srvc/demo.ckan.net directory::
-
- $ mkdir data sstore
- $ chmod g+w -R data sstore
- $ sudo chgrp -R www-data data sstore
- $ ln -s pyenv/src/ckan/who.ini ./
-
- Also edit the who.ini configuration file to set a secret for the auth_tkt plugin.
-
-
-11. Setup Apache with Ckan
-
- Create file /etc/apache2/sites-available/demo.ckan.net as follows::
-
- <VirtualHost *:80>
- ServerName demo.ckan.net
- ServerAlias demo.ckan.net
-
- WSGIScriptAlias / /home/USER/var/srvc/demo.ckan.net/pyenv/bin/demo.ckan.net.py
- # pass authorization info on (needed for rest api)
- WSGIPassAuthorization On
-
- ErrorLog /var/log/apache2/demo.ckan.net.error.log
- CustomLog /var/log/apache2/demo.ckan.net.custom.log combined
- </VirtualHost>
-
-
-12. Enable site in Apache
-
- ::
-
- $ sudo a2ensite demo.ckan.net
-
-
-13. Restart Apache
-
- ::
-
- $ sudo /etc/init.d/apache2 restart
-
-
-14. Browse CKAN website at http://demo.ckan.net/ (assuming you have the DNS setup for this server). Should you have problems, take a look at the log files specified in your apache config and ckan oconfig. e.g. ``/var/log/apache2/demo.ckan.net.error.log`` and ``/var/log/ckan/demo.ckan.log``.
-
-
-Upgrade
--------
-
-Ideally production deployments are upgraded with fabric, but here are the manual instructions.
-
-1. Activate the virtual environment for your install::
-
- $ cd ~/var/srvc/demo.ckan.net
- $ . pyenv/bin/activate
-
-2. It's probably wise to backup your database::
-
- $ paster --plugin=ckan db dump demo_ckan_backup.pg_dump --config=demo.ckan.net.ini
-
- If you get a message about the command being 'mothballed' then you have a particularly old ckan! In this case, use pg_dump directly, specifying the database details from your config file.
-
- $ grep -i sqlalchemy.url demo.ckan.net.ini
- sqlalchemy.url = postgres://okfn:testpassword@psql.okfn.org/demo.okfn.org
- $ pg_dump -U okfn -h psql.okfn.org >demo_ckan_backup.pg_dump
-
-3. Get a version of pip-requirements.txt for the new version you want to install (see info on finding a suitable tag name above)::
-
- $ wget https://bitbucket.org/okfn/ckan/raw/ckan-1.4/pip-requirements.txt
-
-4. Update all the modules::
-
- $ pip -E pyenv install -r pip-requirements.txt
-
-5. Upgrade the database::
-
- $ paster --plugin ckan db upgrade --config {config.ini}
-
-6. Restart apache (so modpython has the latest code)::
-
- $ sudo /etc/init.d/apache2 restart
-
-7. You could manually try CKAN in a browser, or better still run the smoke tests found in ckanext/blackbox. To do this, install ckanext and run ckanext from another machine - see ckanext README.txt for instructions: https://bitbucket.org/okfn/ckanext and then run::
-
- $ python blackbox/smoke.py blackbox/ckan.net.profile.json
-
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/doc/extensions.rst Thu Jul 28 17:20:40 2011 +0100
@@ -0,0 +1,66 @@
+==============
+Add Extensions
+==============
+
+This is where it gets interesting! The CKAN software can be customised with 'extensions'. These are a simple way to extend core CKAN functions.
+
+Extensions allow you to customise CKAN for your own requirements, without interfering with the basic CKAN system.
+
+.. warning:: This is an advanced topic. At the moment, you need to have prepared your system to work with extensions, as described in :doc:`prepare-extensions`. We are working to make the most popular extensions more easily available as Debian packages.
+
+Finding Extensions
+------------------
+
+Many CKAN extensions are listed on the CKAN wiki's `List of Extensions <http://wiki.ckan.net/List_of_Extensions>`_. All CKAN extensions can be found at `OKFN's bitbucket page <https://bitbucket.org/okfn/>`_, prefaced with ``ckanext-``.
+
+Some popular extensions include:
+
+* `ckanext-admin <https://bitbucket.org/okfn/ckanext-admin>`_: Admin web interface for CKAN.
+* `ckanext-apps <https://bitbucket.org/okfn/ckanext-apps>`_: Apps and ideas catalogue extension for CKAN.
+* `ckanext-deliverance <https://bitbucket.org/okfn/ckanext-deliverance>`_: Extends CKAN to use the Deliverance HTTP proxy, which can request and render web pages from * an external site (e.g. a CMS like Drupal or Wordpress).
+* `ckanext-disqus <https://bitbucket.org/okfn/ckanext-disqus>`_: Allows users to comment on package pages with Disqus.
+* `ckanext-follower <https://bitbucket.org/okfn/ckanext-follower>`_: Allow users to follow packages.
+* `ckanext-googleanalytics <https://bitbucket.org/okfn/ckanext-googleanalytics>`_: Integrates Google Analytics data into CKAN. Gives download stats on package pages, list * of most popular packages, etc.
+* `ckanext-qa <https://bitbucket.org/okfn/ckanext-qa>`_: Provides link checker, 5 stars of openness and other Quality Assurance features.
+* `ckanext-rdf <https://bitbucket.org/okfn/ckanext-rdf>`_: Consolidated handling of RDF export and import for CKAN.
+* `ckanext-stats <https://bitbucket.org/okfn/ckanext-stats>`_: Statistics (and visuals) about the datasets in a CKAN instance.
+* `ckanext-wordpresser <https://bitbucket.org/okfn/ckanext-wordpresser>`_: CKAN plugin / WSGI middleware for combining CKAN with a Wordpress site.
+
+Installing an Extension
+-----------------------
+
+You can install an extension on a CKAN instance as follows.
+
+1. First, ensure you are working within your virtualenv (see :doc:`prepare-extensions` if you are not sure what this means)::
+
+ . /home/ubuntu/pyenv/bin/activate
+
+2. Install the extension package code using ``pip``.
+
+ For example, to install the Disqus extension, which allows users to comment on datasets::
+
+ pip install -E ~/var/srvc/ckan.net/pyenv hg+http://bitbucket.org/okfn/ckanext-disqus
+
+ The ``-E`` parameter is for your CKAN Python environment (e.g. ``~/var/srvc/ckan.net/pyenv``).
+
+ Prefix the source URL with the repo type (``hg+`` for Mercurial, ``git+`` for Git).
+
+ The dependency you've installed will appear in the ``src/`` directory under your Python environment.
+
+3. Add the names of any plugin implementations the extension uses to the CKAN
+config file. You can find these in the plugin's ``setup.py`` file under ``[ckan.plugins]``.
+
+ The config plugins variable is in the '[app:main]' section under 'ckan.plugins'. e.g.::
+
+ [app:main]
+ ckan.plugins = disqus
+
+ If your extension implements multiple different plugin interfaces, separate them with spaces::
+
+ ckan.plugins = disqus amqp myplugin
+
+4. If necessary, restart WSGI, which usually means restarting Apache::
+
+ sudo /etc/init.d/apache2 restart
+
+Your extension should now be installed.
--- a/doc/feeds.rst Thu Jul 28 16:26:50 2011 +0100
+++ /dev/null Thu Jan 01 00:00:00 1970 +0000
@@ -1,111 +0,0 @@
-==========
-CKAN Feeds
-==========
-
-Introduction
-============
-
-Access to the CKAN metadata is provided in the RESTful API, but to help users in touch with changes to the data, CKAN also provides feeds. The main feed provides a list of recent changes to all packages, and there are also individual feeds for each package.
-
-Requests
-========
-
-The main feed is at::
-
-/revision/list?format=atom&days=1
-
-For example, to see all the changes made to packages on ckan.net in the past day, browse address: `<http://ckan.net/revision/list?format=atom&days=1>`_
-
-In addition, each package has a feed at::
-
-/package/history/PACKAGENAME?format=atom&days=7
-
-You might want to keep track of developments of the Ordnance Survey open data seen here `<http://ckan.net/package/ordnance_survey>`_, for example. Here is the corresponding URL you'd add to your RSS/Atom Reader: `<http://ckan.net/package/history/ordnance_survey?format=atom&days=7>`_
-
-Format
-======
-
-The feeds are in Atom format.
-
-Each 'entry' in the feed corresponds to a Revision, which is a set of changes.
-
-The Package feed is a subset of the Main Feed, with only the Revisions that reflect a change in that particular package.
-
-The details given for each Revision are:
-
-+-----------+-------------------------------------+
-| Field | Description |
-+===========+=====================================+
-| updated / | Date & time of the change, given |
-| published | in ISO format. |
-+-----------+-------------------------------------+
-| author | 'name' is login name or IP address. |
-+-----------+-------------------------------------+
-| link | Link to Revision in the web |
-| | interface. |
-+-----------+-------------------------------------+
-| summary | Log message for the revision. Also, |
-| | for the main feed, details of |
-| | packages changes, created and |
-| | deleted. See below. |
-+-----------+-------------------------------------+
-
-Example main feed::
-
- <?xml version="1.0" encoding="utf-8"?>
- <feed xmlns="http://www.w3.org/2005/Atom" xml:lang="None">
- <title>CKAN Package Revision History</title>
- <link href="/revision/list/" rel="alternate"></link>
- <id>/revision/list/</id>
- <updated>2010-04-18T21:17:57Z</updated>
- <entry>
- <title>rdbc6b54d-1fd1-4b13-b250-340a01646909 [pgcanada:]</title>
- <link href="/revision/read/dbc6b54d-1fd1-4b13-b250-340a01646909" rel="alternate"></link>
- <updated>2010-04-18T21:17:57Z</updated>
- <author><name>208.65.246.156</name></author>
- <id>tag:,2010-04-18:/revision/read/dbc6b54d-1fd1-4b13-b250-340a01646909</id>
- <summary type="html">Packages affected: [pgcanada:created].</summary>
- </entry>
- <!-- ...other entries... -->
- </feed>
-
-Example package feed for package 'water_voles'::
-
- <?xml version="1.0" encoding="utf-8"?>
- <feed xmlns="http://www.w3.org/2005/Atom" xml:lang="None">
- <title>CKAN Package Revision History</title>
- <link href="/revision/read/water_voles" rel="alternate"></link>
- <id>/revision/read/water_voles</id>
- <updated>2010-04-19T15:32:28Z</updated>
- <entry>
- <title>Creating test data.</title>
- <link href="/revision/read/2bad6982-2394-4187-b289-2ed77ad25d65" rel="alternate"></link>
- <updated>2010-04-19T15:30:00Z</updated>
- <published>2010-04-19T15:30:00Z</published>
- <author><name>http://someone.somecompany.com</name></author>
- <id>tag:,2010-04-19:/revision/read/2bad6982-2394-4187-b289-2ed77ad25d65</id>
- <summary type="html">Log message: Added a test package.</summary>
- </entry>
- <!-- ...other entries... -->
- </feed>
-
-Main feed summary
-=================
-
-The main field provides a little more detail to the edits made to packages. In each entry summary, there is a list of packages affected, and the change to each is described as 'created', 'updated' or 'deleted'. For example::
-
- <summary type="html>Packages affected: [hospital_performance:created].</summary>
-
-For package updates, some further information is provided if there are changes to the package's resources, or the 'date_updated' extra field. For example::
-
- Packages affected: [water_voles:updated:resources].
-
-or::
-
- Packages affected: [water_voles:updated:date_updated].
-
-(The date_updated field is highlighted for a particular use of CKAN and this feature may be generalised in the future.)
-
-A revision may have changes to several packages, particularly if it was created using the API. In this case, the package list is space separated::
-
- Packages affected: [hospital_performance:created water_voles:updated bus_stops:updated:resources bus_stats:updated:resources:date_updated].</summary>
--- a/doc/form-integration.rst Thu Jul 28 16:26:50 2011 +0100
+++ b/doc/form-integration.rst Thu Jul 28 17:20:40 2011 +0100
@@ -1,30 +1,30 @@
================
-Form integration
+Form Integration
================
-CKAN provides facility for integrating the package editing forms into another front end
+CKAN allows you to integrate its Edit Package and New Package forms forms into an external front-end. To that end, CKAN also provides a simple way to redirect these forms back to the external front-end upon submission.
-Form completion redirect
-========================
+Redirecting CKAN Forms
+======================
-It is simple enough for the external front end to link to CKAN's package creation or edit pages, but once the form is submitted the user needs to be redirected back to the external front end, instead of CKAN's package read page. This is achieved with a parameter to the CKAN URL.
+It is obviously simple enough for an external front-end to link to CKAN's Edit Package and New Package forms, but once the forms are submitted, it would be desirable to redirect the user back to the external front-end, rather than CKAN's package read page.
-The 'return URL' can be specified in two places:
+This is achieved with a parameter to the CKAN URL. The 'return URL' can be specified in two places:
- 1. passed as a URL encoded value with the parameter "return_to" in the link to CKAN's form page.
+ 1. Passed as a URL-encoded value with the parameter ``return_to`` in the link to CKAN's form page.
- 2. specified in the CKAN config key 'package_new_return_url' and 'package_edit_return_url'.
+ 2. Specified in the CKAN config keys ``package_new_return_url`` and ``package_edit_return_url`` (see :ref:`config-package-urls`).
-(If the 'return URL' is given in both ways then the first takes precedence.)
+(If the 'return URL' is supplied in both places, then the first takes precedence.)
-Since the 'return URL' may need to include the package name, which could be set in the form, CKAN replaces a known placeholder "<NAME>" with this value on redirect.
+Since the 'return URL' may need to include the package name, which could be changed by the user, CKAN replaces a known placeholder ``<NAME>`` with this value on redirect.
-Note that the downside of specifying the 'return URL' in the CKAN config is that the CKAN web interface is less usable on its own, since the user is hampered by the redirects to the external interface.
+.. note:: Note that the downside of specifying the 'return URL' in the CKAN config is that the CKAN web interface becomes less usable on its own, since the user is hampered by the redirects to the external interface.
Example
-------
-An external front end displays a package 'ontariolandcoverv100' here::
+An external front-end displays a package 'ontariolandcoverv100' here::
http://datadotgc.ca/dataset/ontariolandcoverv100
@@ -32,27 +32,23 @@
http://ca.ckan.net/package/edit/ontariolandoverv100
-On first thought, the return link is::
-
- http://datadotgc.ca/dataset/ontariolandcoverv100
-
-But when the user edit's this package, the name may change. So the return link needs to be::
+At first, it may seem that the return link should be ``http://datadotgc.ca/dataset/ontariolandcoverv100``. But when the user edits this package, the name may change. So the return link needs to be::
http://datadotgc.ca/dataset/<NAME>
-This is URL encoded to be::
+And this is URL-encoded to become::
http%3A%2F%2Fdatadotgc.ca%2Fdataset%2F%3CNAME%3E
-So the edit link becomes::
+So, in summary, the edit link becomes::
http://ca.ckan.net/package/edit/ontariolandoverv100?return_to=http%3A%2F%2Fdatadotgc.ca%2Fdataset%2F%3CNAME%3E
-During editing the package, the user changes the name to `canadalandcover`, presses 'preview' and finally 'commit'. The user is now redirected back to the external front end at::
+During editing the package, the user changes the package name to `canadalandcover`, presses 'preview' and finally 'commit'. The user is now redirected back to the external front-end at::
http://datadotgc.ca/dataset/canadalandcover
-This same functionality could be achieved by this line in the config (ca.ckan.net.ini)::
+The same functionality could be achieved by this line in the config file (``ca.ckan.net.ini``)::
...
--- a/doc/forms.rst Thu Jul 28 16:26:50 2011 +0100
+++ b/doc/forms.rst Thu Jul 28 17:20:40 2011 +0100
@@ -1,56 +1,55 @@
-Forms Using Templates
-=====================
+=================
+Customizing Forms
+=================
-The forms used to edit Packages and Groups in CKAN can be customised. This makes it easier to help users input data, helping them choose from sensible options or to use different data formats. This document sets out to show how this is achieved, without getting embroilled in the main CKAN code.
+The forms used to edit packages and groups in CKAN can be customized. This lets you tailor them to your needs, helping your users choose from sensible options or use different data formats.
-Note that this section deals with the form used to *edit* packages and groups, not the way they are displayed. Display is done in the CKAN templates.
+This document explains how to customize the package and group forms you offer to your users, without getting embroiled in the core CKAN code.
-Location of related code
-------------------------
-
- * ckan/controllers/package.py
- * ckan/templates/package/new_package_form.html
+.. note:: This section deals with the form used to *edit* packages and groups, not the way they are displayed. For information on customizing the display of forms, see :doc:`theming`.
-Building a package form
+.. warning:: This is an advanced topic. Ensure you are familiar with :doc:`extensions` before attempting to customize forms.
+
+Building a Package Form
-----------------------
-Basics
-^^^^^^
+The Best Way: Extensions
+^^^^^^^^^^^^^^^^^^^^^^^^
-You will firstly need to make a new controller in your extension. This should subclass PackageController like::
+The best way to build a package form is by using a CKAN extension.
+
+You will firstly need to make a new controller in your extension. This should subclass PackageController as follows::
from ckan.controllers.package import PackageController
class PackageNew(PackageController):
package_form = 'custom_package_form.html'
-The package_form variable in the subclass will be used as the new form template.
+The ``package_form`` variable in the subclass will be used as the new form template.
-It is recommended that you copy the package form (named new_package_form.html) and make modifications to it. However, it is possible to start from scratch.
+It is recommended that you copy the package form (``new_package_form.html``) and make modifications to it. However, it is possible to start from scratch.
-In order for you to get this new controller pointed at correctly, your extension should look like the following::
+To point at this new controller correctly, your extension should look like the following::
class CustomForm(SingletonPlugin):
-
implements(IRoutes)
implements(IConfigurer)
-
def before_map(self, map):
map.connect('/package/new', controller='ckanext.extension_name.controllers.PackageNewController:PackageNew', action='new')
map.connect('/package/edit/{id}', controller='ckanext.extension_name.controllers.PackageNewController:PackageNew', action='edit')
return map
-
def after_map(self, map):
- return map
-
+ return map
def update_config(self, config):
configure_template_directory(config, 'templates')
-Replace extension_name with the name of your extension. This also assumes that custom_package_form.html is located in templates subdirectory of your extension i.e ckanext/extension_name/templates/custom_package_form.html.
+Replace ``extension_name`` with the name of your extension.
+
+This also assumes that ``custom_package_form.html`` is located in the ``templates`` subdirectory of your extension i.e ``ckanext/extension_name/templates/custom_package_form.html``.
Advanced Use
^^^^^^^^^^^^
-The PackageController has a more hooks to customize how and what data is displayed. These functions can be overridden subclass of PackageController::
+The PackageController has more hooks to customize the displayed data. These functions can be overridden in a subclass of PackageController::
_setup_template_variables(self, context)
@@ -64,185 +63,4 @@
This defines a navl schema to customize conversion from the database to the form.
-A complicated example of the use of these hooks can be found in extension ckanext-dgu.
-
-
-Forms using FormAlchemy (depreciated)
-=====================================
-
-This is now depreciated in order to get this to work your extention must implement IRoutes and have a before_map like the following::
-
- class Plugin(SingletonPlugin):
- implements(IRoutes)
-
- def before_map(self, map):
- map.connect('/package/new', controller='package_formalchemy', action='new')
- map.connect('/package/edit/{id}', controller='package_formalchemy', action='edit')
- return map
-
-Location of related code
-------------------------
-
-In the CKAN code, the majority of forms code is in ckan/forms.
-
- * ckan/forms/common.py - A common place for fields used by standard forms
- * ckan/forms/builder.py - The FormBuilder class, which provides an easy way to define a form. It creates a FormAlchemy fieldset used in the CKAN controller code.
- * ckan/forms/package.py - This contains the 'standard' form, which is a good example and is useful to derive custom forms from.
- * ckan/forms/package_gov.py - Contains the a government form, which serves as an example of using lots of extra fields for dates, geographical coverage, etc.
-
-
-Building a package form
------------------------
-
-Basics
-^^^^^^
-
-The *PackageFormBuilder* class initialises with the basic package form which we can then configure::
-
- builder = PackageFormBuilder()
-
-All the basic package fields are added to the form automatically - this currently includes: name, title, version, url, author, author_email, maintainer, maintainer_email, notes and license_id. In case this changes in future, consult the fields for table 'Package' in ckan/model/core.py.
-
-To provide editing of other fields beyond the basic ones, you need to use *add_field* and select either an existing *ConfiguredField* in common.py, or define your own. For example, the autocompleting tag entry field is defined as a field in common.py and it is added to the standard form like this::
-
- builder.add_field(common.TagField('tags'))
-
-The basic fields (name, title, etc) and a few more (license, tags, resources) are defined for all packages. Additional information can be stored on each package in the 'extra' fields. Often we want to provide a nicer interface to these 'extra' fields to help keep consistency in format between the packages. For example, in the government form (package_gov.py) we have added a field for the release date. This is stored as a Package 'extra' with key 'date_released' and by using the DateExtraField, in the form the user is asked for a date.::
-
- builder.add_field(common.DateExtraField('date_released'))
-
-You can configure existing fields using the usual `FormAlchemy Field options <http://docs.formalchemy.org/fields.html#fields>`_. For example, here we add a validator to a standard field::
-
- builder.set_field_option('name', 'validate', package_name_validator)
-
-Options are given keyword parameters by passing a dictionary. For example, this is how we set the notes field's size::
-
- builder.set_field_option('notes', 'textarea', {'size':'60x15'})
-
-Fields in package forms are grouped together. You should specify which fields are displayed in which groups and in which order like this::
-
- from sqlalchemy.util import OrderedDict
- builder.set_displayed_fields_in_groups(OrderedDict([
- ('Basic information', ['name', 'title', 'version', 'url']),
- ('Resources', ['resources']),
- ('Detail', ['author', 'author_email'])]))
-
-To complete the form design you need to return the fieldset object. Ensure this is executed once - when your python form file is imported::
-
- my_fieldset = builder.get_fieldset()
-
-
-Field labels
-^^^^^^^^^^^^
-
-The field labels are derived from the model key using a 'prettify' function. The default munge capitalises the first letter and changes underscores to spaces. You can write a more advanced function depending on your needs. Here is the template for a prettify function::
-
- def prettify(field_name):
- return field_name.replace('_', ' ').capitalize())
-
-If you write a new one, you tell the builder about it like this::
-
- builder.set_label_prettifier(prettify)
-
-
-Templates
-^^^^^^^^^
-
-Package forms by default use the Genshi template *ckan/package/form.html*. If you want to use a modified one then specify it for example like this::
-
- builder.set_form_template('package/my_form')
-
-
-Hidden labels
-^^^^^^^^^^^^^
-
-A couple of common fields (ResourceField and ExtrasField currently) are designed to go in their own field group (see below) and without the usual field label. To hide the label, add these fields like this::
-
- builder.add_field(common.ResourcesField('resources', hidden_label=True))
-
-Instead of starting with just the basic fields, many people will want to edit the standard form, which already contains the resources, extra fields and customise that further. To achieve that you import the builder object like this::
-
- import ckan.forms.package as package
- builder = package.build_package_form()
-
-
-Defining custom fields
-----------------------
-
-If you want to define a completely new field then here is a useful template::
-
- class MyField(common.ConfiguredField):
- def get_configured(self):
- return self.MyField(self.name).with_renderer(self.MyRenderer).validate(self.my_validator)
-
- class MyField(formalchemy.Field):
- def sync(self):
- # edit self.model with using value self._deserialize()
-
- class MyRenderer(formalchemy.fields.FieldRenderer):
- def render(self, **kwargs):
- # return html of field editor based on self._value
-
- def _serialized_value(self):
- # take self._params and serialize them ready for rendering
- # or self.deserialize() into python value that can be saved
- # on a sync.
-
- def my_validator(self, val, field):
- if not ...:
- raise formalchemy.ValidationError('Invalid value')
-
-More examples are in common.py and further information can be obtained from the `FormAlchemy documentation <http://docs.formalchemy.org/>`_.
-
-
-Using a custom form
--------------------
-
-To register your new form with CKAN you need to do three things.
-
-1. In your form you need a function that returns your new form's field set.
-
- For example you might add below your form code::
-
- my_fieldset = builder.get_fieldset()
-
- def get_fieldset(is_admin=False):
- return my_fieldset
-
- (The *is_admin* parameter can be considered if you wish to return a different fieldset for administrator users.)
-
-2. You need to provide an 'entry point' into your code package so that CKAN can access your new form.
-
- It is anticipated that your form code will live in a python package outside the CKAN main code package, managed by setuptools. The entry points are listed in the python package's setup.py and you just need to add a category [ckan.forms] and list the function that returns::
-
- from setuptools import setup, find_packages
- setup(
- ...
-
- entry_points="""
- [ckan.forms]
- my_form = my_module.forms.my_form:get_fieldset
- """,
- )
-
- For this change to have an effect, you need to recreate the egg information, so run::
-
- $ python setup.py egg_info
-
-3. Change an option in your CKAN pylons config file to switch to using the new form.
-
- For example, your pylons config file will probably be 'development.ini' during development, when you 'paster serve' your CKAN app for testing.
-
- You need to change the 'package_form' setting in the '[app:main]' section to the name defined int he entry point. For example::
-
- [app:main]
- ...
- package_form = my_form
- group_form = my_group_form
- package_group_form = my_package_group_form
-
- For this to have an effect you may need to restart the pylons (either by restarting the 'serve' command or the Apache host). Now go and edit a package and try out the new form!
-
- You can also override the config file setting with a URL parameter in your browser. For example you might browse:
-
- http://eco.ckan.net/package/edit/water-voles?package_form=my_form
+A complex example of the use of these hooks can be found in the ``ckanext-dgu`` extension.
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/doc/i18n.rst Thu Jul 28 17:20:40 2011 +0100
@@ -0,0 +1,143 @@
+=====================
+Internationalize CKAN
+=====================
+
+CKAN is used in many countries, and adding a new language is a simple process.
+
+Supported Languages
+===================
+
+CKAN already supports numerous languages. To check whether your language is supported, look in the source at ``ckan/i18n`` for translation files. Languages are named using two-letter ISO language codes (e.g. ``es``, ``de``).
+
+If your language is present, you can switch language simply by setting the ``lang`` option in your CKAN config file, as described in :ref:`config-i18n`. For example, to switch to German::
+
+ lang=de
+
+If your language is not supported yet, the remainder of this section section provides instructions on how to prepare a translation file and add it to CKAN.
+
+Adding a New Language
+=====================
+
+If you want to add an entirely new language to CKAN, you have two options.
+
+* :ref:`i18n-manual`. Creating translation files manually.
+* :ref:`i18n-transifex`. Creating translation files using Transifex, the open source translation software.
+
+
+.. _i18n-manual:
+
+Manual Setup
+------------
+
+If you prefer not to use Transifex, you can create translation files manually.
+
+All the English strings in CKAN are extracted into the ``ckan.pot`` file, which can be found in ``ckan/i18n``.
+
+.. note:: For information, the pot file was created with the ``babel`` command ``python setup.py extract_messages``.
+
+0. Install Babel
+++++++++++++++++
+
+You need Python's ``babel`` library (Debian package ``python-pybabel``). Install it as follows with pip::
+
+ pip -E pyenv install babel
+
+1. Create a 'po' File for Your Language
++++++++++++++++++++++++++++++++++++++++
+
+First, grab the CKAN i18n repository::
+
+ hg clone http://bitbucket.org/bboissin/ckan-i18n/
+
+Then create a translation file for your language (a po file) using the pot file::
+
+ python setup.py init_catalog --locale YOUR_LANGUAGE
+
+Replace ``YOUR_LANGUAGE`` with the two-letter ISO language code (e.g. ``es``, ``de``).
+
+In future, when the pot file is updated, you can update the strings in your po file, while preserving your po edits, by doing::
+
+ python setup.py update_catalog --locale YOUR-LANGUAGE
+
+2. Do the Translation
++++++++++++++++++++++
+
+Edit the po file and translate the strings. For more information on how to do this, see `the Pylons book <http://pylonsbook.com/en/1.1/internationalization-and-localization.html>`_.
+
+We recommend using a translation tool, such as `poedit <http://www.poedit.net/>`_, to check the syntax is correct. There are also extensions for editors such as emacs.
+
+3. Commit the Translation
+++++++++++++++++++++++++++
+
+When the po is complete, commit it to the CKAN i18n repo::
+
+ hg add ckan/i18n/YOUR_LANGUAGE/LC_MESSAGES/ckan.po
+ hg ci -m '[i18n]: New language po added: YOUR_LANGUAGE' ckan/i18n/YOUR_LANGUAGE/LC_MESSAGES/ckan.po
+ hg push
+
+.. note:: You need to be given credentials to do this - to request these, contact the `ckan-discuss <http://lists.okfn.org/mailman/listinfo/ckan-discuss>`_ list.
+
+4. Compile a Translation
+++++++++++++++++++++++++
+
+Once you have created a translation (either with Transifex or manually) you can build the po file into a ``mo`` file, ready for deployment.
+
+With either method of creating the po file, it should be found in the CKAN i18n repository: ``ckan/i18n/YOUR_LANGUAGE/LC_MESSAGES/ckan.po``
+
+In this repo, compile the po file like this::
+
+ python setup.py compile_catalog --locale YOUR_LANGUAGE
+
+As before, replace ``YOUR_LANGUAGE`` with your language short code, e.g. ``es``, ``de``.
+
+This will result in a binary 'mo' file of your translation at ``ckan/i18n/YOUR_LANGUAGE/LC_MESSAGES/ckan.mo``.
+
+5. (optional) Deploy the Translation
+++++++++++++++++++++++++++++++++++++
+
+This section explains how to deploy your translation automatically to your host, if you are using a remote host.
+
+It assumes a standard layout on the server (you may want to check before you upload!) and that you are deploying to ``hu.ckan.net`` for language ``hu``.
+
+Once you have a compiled translation file, for automated deployment to your host do::
+
+ fab config_0:hu.ckan.net upload_i18n:hu
+
+See the ``config_0`` options if more configuration is needed e.g. of host or location.
+
+Alternatively, if you do not want to use fab, you can just scp::
+
+ scp ckan.mo /home/okfn/var/srvc/hu.ckan.net/pyenv/src/ckan/ckan/i18n/hu/LC_MESSAGES/ckan.mo
+
+6. Configure the Language
++++++++++++++++++++++++++
+
+Finally, once the mo file is in place, you can switch between the installed languages using the ``lang`` option in the CKAN config file, as described in :ref:`config-i18n`.
+
+
+.. _i18n-transifex:
+
+Transifex Setup
+---------------
+
+Transifxes, the open translation platform, provides a simple web interface for writing translations and is widely used for CKAN internationalization.
+
+Using Transifex makes it easier to handle collaboration, with an online editor that makes the process more accessible.
+
+Existing CKAN translation projects can be found at: https://www.transifex.net/projects/p/ckan/teams/
+
+Updated translations are automatically pushed to https://bitbucket.org/bboissin/ckan-i18n and these can be compiled and placed on CKAN servers by the server administrators.
+
+Transifex Administration
+++++++++++++++++++++++++
+
+The Transifex workflow is as follows:
+
+* Install transifex command-line utilities
+* ``tx init`` in CKAN to connect to Transifex
+* Run ``python setup.py extract_messages`` on the CKAN source
+* Upload the local .pot file via command-line ``tx push``
+* Get people to complete translations on Transifex
+* Pull locale .po files via ``tx pull``
+* ``python setup.py compile_catalog``
+* Commit and push po and mo files
--- a/doc/importer.rst Thu Jul 28 16:26:50 2011 +0100
+++ /dev/null Thu Jan 01 00:00:00 1970 +0000
@@ -1,98 +0,0 @@
-================
-Package Importer
-================
-
-
-Introduction
-============
-
-This part of the CKAN web interface provides an easy way to import packages
-into CKAN in bulk via a spreadsheet. It fills the gap between entry via a
-simple form (/package/new) and the RESTful API (/api/rest).
-
-** NB: This feature is not currently available. **
-
-
-Details
-=======
-
-Importing a package with the same name as one that exists in the CKAN database results in the new package overwriting the existing one. There is a warning for this.
-
-To perform an import, the user must be logged in. To add a package to a group, the user must have priviledges to edit the particular group.
-
-Format
-======
-
-The details of the packages should be stored in an Excel spreadsheet or CSV file. In Excel format, the package details should be the first (or only) sheet of a workbook.
-
-The importer looks for a header row (which must contain 'name' or 'title') and below that all the rows are the package details. The header row can contain any or all of the field names, but must include 'name' or 'title'. If the 'name' is not specified then a unique name will be generated from the title.
-
-Example
-=======
-
-+-------+-----------+-----------+--------------------------------+------------------------+---+
-| | A | B | C | D | E |
-+=======+===========+===========+================================+========================+===+
-| **1** | Packages | | | | |
-+-------+-----------+-----------+--------------------------------+------------------------+---+
-| **2** | | | | | |
-+-------+-----------+-----------+--------------------------------+------------------------+---+
-| **3** | name | title | resource-0-url | tags | |
-+-------+-----------+-----------+--------------------------------+------------------------+---+
-| **4** | wikipedia | Wikipedia | http://download.wikimedia.org/ | encyclopedia reference | |
-+-------+-----------+-----------+--------------------------------+------------------------+---+
-| **5** | tviv | TV IV | http://tviv.org/Category:Grids | tv encyclopaedia | |
-+-------+-----------+-----------+--------------------------------+------------------------+---+
-| **6** | | | | | |
-+-------+-----------+-----------+--------------------------------+------------------------+---+
-
-Fields
-======
-
-Each package has many fields.
-
-+------------------------+----------------------------------------+------------------------------------+
-| Name | Example value | Notes |
-+========================+========================================+====================================+
-| name | wikipedia-blind | |
-+------------------------+----------------------------------------+------------------------------------+
-| title | Wikipedia for the Blind | |
-+------------------------+----------------------------------------+------------------------------------+
-| notes | Maintained until 2008 | |
-+------------------------+----------------------------------------+------------------------------------+
-| url | http://blind.wikipedia.org/ | |
-+------------------------+----------------------------------------+------------------------------------+
-| resource-0-url | http://blind.wikipedia.org/dump-en.csv | Number resources from 0. |
-+------------------------+----------------------------------------+------------------------------------+
-| resource-0-format | csv | |
-+------------------------+----------------------------------------+------------------------------------+
-| resource-0-description | English version | |
-+------------------------+----------------------------------------+------------------------------------+
-| resource-0-hash | e0d123e5f31 | Hash of the resource |
-+------------------------+----------------------------------------+------------------------------------+
-| resource-1-url | http://blind.wikipedia.org/dump-fr.csv | |
-+------------------------+----------------------------------------+------------------------------------+
-| resource-1-format | csv | |
-+------------------------+----------------------------------------+------------------------------------+
-| resource-1-description | French version | |
-+------------------------+----------------------------------------+------------------------------------+
-| resource-1-hash | 78bfdf5a008 | |
-+------------------------+----------------------------------------+------------------------------------+
-| tags | encyclopedia blind format-csv | Space separated list |
-+------------------------+----------------------------------------+------------------------------------+
-| author | John Doe | |
-+------------------------+----------------------------------------+------------------------------------+
-| author_url | john at doe.com | |
-+------------------------+----------------------------------------+------------------------------------+
-| maintainer | John Doe | |
-+------------------------+----------------------------------------+------------------------------------+
-| maintainer_url | john at doe.com | |
-+------------------------+----------------------------------------+------------------------------------+
-| license | OKD Compliant::UK Click Use PSI | License name |
-+------------------------+----------------------------------------+------------------------------------+
-| [arbitrary] | | Any field name and a string value. |
-+------------------------+----------------------------------------+------------------------------------+
-| groups | blind-picks | Space separated list of group |
-| | | names to add package to. |
-+------------------------+----------------------------------------+------------------------------------+
-
--- a/doc/index.rst Thu Jul 28 16:26:50 2011 +0100
+++ b/doc/index.rst Thu Jul 28 17:20:40 2011 +0100
@@ -1,34 +1,40 @@
-==========================================
-Welcome to CKAN's Developer Documentation!
-==========================================
+=======================================
+Welcome to CKAN's Administration Guide
+=======================================
-This Developer Documentation provides more advanced help for `CKAN <http://ckan.org>`_ development and administration. The latest released version of this documentation is found at: http://packages.python.org/ckan/
+This Administration Guide covers how to set up and manage `CKAN <http://ckan.org>`_ (Comprehensive Knowledge Archive Network) software.
+* The first two sections cover your two options for installing CKAN: package or source install.
+* The rest of the first half of the Guide, up to :doc:`authorization`, cover setup and basic admin.
+* The second half of the Guide, from :doc:`prepare-extensions` onwards, covers advanced tasks, including extensions and forms.
+
+For high-level information on what CKAN is, see the `CKAN website <http://ckan.org>`_.
Contents:
.. toctree::
:maxdepth: 2
- README
- deployment
- configuration
+ install-from-package
+ install-from-source
+ post-installation
+ theming
+ loading_data
paster
- api
+ authorization
+ prepare-extensions
+ extensions
plugins
- feeds
- loader_scripts
- importer
forms
form-integration
- model
- authentication
- authorization
- load_testing
database_dumps
- deb
- vm
+ upgrade
+ i18n
+ configuration
+ api
+ test
buildbot
+ about
Other material
==============
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/doc/install-from-package.rst Thu Jul 28 17:20:40 2011 +0100
@@ -0,0 +1,429 @@
+==============================
+Option 1: Package Installation
+==============================
+
+This section describes how to install CKAN from packages. This is the recommended and by far the easiest way to install CKAN.
+
+Package install requires you to use 64-bit Ubuntu 10.04: either locally, through a virtual machine or Amazon EC2. Your options are as follows:
+
+* Using 64-bit Ubuntu 10.04 directly.
+* :ref:`using-virtualbox`. This is suitable if you want to host your CKAN instance on a machine running any other OS.
+* :ref:`using-amazon`. This is suitable if you want to host your CKAN instance in the cloud, on a ready-made Ubuntu OS.
+
+.. note:: We recommend you use package installation unless you are a core CKAN developer or have no access to Ubuntu 10.04 through any of the methods above, in which case, you should use :doc:`install-from-source`.
+
+For support during installation, please contact `the ckan-dev mailing list <http://lists.okfn.org/mailman/listinfo/ckan-dev>`_.
+
+Prepare your System
+--------------------
+
+CKAN runs on 64-bit Ubuntu 10.04. If you are already using Ubuntu 10.04, you can continue straight to :ref:`run-package-installer`.
+
+However, if you're not, you can either use VirtualBox to set up an Ubuntu VM on Windows, Linux, Macintosh and Solaris. Alternatively, you can use an Amazon EC2 instance.
+
+.. _using-virtualbox:
+
+Option A: Using VirtualBox
+++++++++++++++++++++++++++
+
+This option is suitable if you want to install CKAN on a machine running an OS other than Ubuntu 10.04. `VirtualBox <http://www.virtualbox.org>`_ lets you set up a virtual machine to run Ubuntu 10.04.
+
+Pre-requisites and Downloads
+****************************
+
+First, check your machine meets `the pre-requisites for VirtualBox <http://www.virtualbox.org/wiki/End-user_documentation>`_. These include a fairly recent processor and some spare memory.
+
+Then download the installation files.
+
+* `Download the VirtualBox installer <http://www.virtualbox.org/wiki/Downloads>`_.
+* `Download the Ubuntu image <http://www.ubuntu.com/download/ubuntu/download>`_ - make sure you choose Ubuntu 10.04 and 64-bit.
+
+Install VirtualBox
+******************
+
+.. note::
+
+ This tutorial is for a Mac, but you can find instructions for installing VirtualBox on any OS `in the VirtualBox Manual <http://www.virtualbox.org/manual/ch02.html>`_.
+
+To install, double-click on the VirtualBox installer:
+
+.. image:: images/virtualbox1-package.png
+ :width: 807px
+ :alt: The VirtualBox installer - getting started
+
+Click Continue to begin the installation process. Enter your password when required, and wait for the installation to finish.
+
+Create Your Virtual Machine
+***************************
+
+Go to Applications and open VirtualBox, then click New:
+
+.. image:: images/virtualbox4-newvm.png
+ :width: 807px
+ :alt: The VirtualBox installer - the New Virtual Machine Wizard
+
+Give your VM a name - we'll call ours ``ubuntu_ckan``. Under **OS Type**, choose **Linux** and **Ubuntu 64-bit**.
+
+.. image:: images/virtualbox5-vmtype.png
+ :width: 807px
+ :alt: The VirtualBox installer - choosing your operating system
+
+Leave the memory size as 512MB, and choose **Create new hard disk**. This will open a new wizard:
+
+.. image:: images/virtualbox6-vmloc.png
+ :width: 807px
+ :alt: The VirtualBox installer - creating a new hard disk
+
+You can leave the defaults unchanged here too - click **Continue**, and then **Done**, and **Done** again, to create a new VM.
+
+Next, choose your VM from the left-hand menu, and click **Start**:
+
+.. image:: images/virtualbox7-startvm.png
+ :width: 807px
+ :alt: Starting your new VM
+
+This will open the First Run Wizard:
+
+.. image:: images/virtualbox8-firstrun.png
+ :width: 807px
+ :alt: The VirtualBox First Run Wizard
+
+After clicking **Continue**, you'll see **Select Installation Media**. This is where we need to tell our VM to boot from Ubuntu. Click on the file icon, and find your Ubuntu ``.iso`` file:
+
+.. image:: images/virtualbox9-iso.png
+ :width: 807px
+ :alt: When you get to Select Installation Media, choose your Ubuntu .iso file
+
+Click **Done**, wait for a few seconds, and you will see your Ubuntu VM booting.
+
+Set Up Ubuntu
+*************
+
+During boot, you will be asked if you want to try Ubuntu, or install it. Choose **Install Ubuntu**:
+
+.. image:: images/virtualbox11-ubuntu.png
+ :width: 807px
+ :alt: Booting Ubuntu - choose the Install Ubuntu option
+
+You can then follow the usual Ubuntu installation process.
+
+After Ubuntu is installed, from the main menu, choose **System > Administration > Update Manager**. You'll be asked if you want to install updates - say yes.
+
+When all the updates have been downloaded and installed, you'll be prompted to reboot Ubuntu.
+
+At this point, you can proceed to :ref:`run-package-installer`.
+
+.. _using-amazon:
+
+Option B: Using Amazon EC2
+++++++++++++++++++++++++++
+
+If you prefer to run your CKAN package install in the cloud, you can use an Amazon EC2 instance, which is a fairly cheap and lightweight way to set up a server.
+
+Create an Amazon Account
+************************
+
+If you don't already have an Amazon AWS account you'll need to create one first. You can `create an Amazon AWS account for EC2 here <http://aws.amazon.com/ec2/>`_.
+
+Configure EC2
+*************
+
+Once you have an EC2 account, you'll need to configure settings for your CKAN instance.
+
+Start by logging into your `Amazon AWS Console <https://console.aws.amazon.com/s3/home>`_ and click on the EC2 tab.
+
+Select the region you want to run your CKAN instance in - the security group you set up is region-specific. In this tutorial, we use EU West, so it will be easier to follow if you do too.
+
+.. image :: images/1.png
+
+Set up a Security Group
+^^^^^^^^^^^^^^^^^^^^^^^
+
+Click the **Security Groups** link in the **My Resources** section in the right-hand side of the dashboard.
+
+.. image :: images/2.png
+ :width: 807px
+
+Create a security group called ``web_test`` that gives access to ports 22, 80 and 5000 as shown below. This is needed so that you'll actually be able to access your server once it is created. You can't change these settings once the instance is running, so you need to do so now.
+
+.. image :: images/3a.png
+ :width: 807px
+
+.. image :: images/3b.png
+ :width: 807px
+
+Create a Keypair
+^^^^^^^^^^^^^^^^
+
+Now create a new keypair ``ckan_test`` to access your instance:
+
+.. image :: images/4.png
+ :width: 807px
+
+When you click **Create**, your browser will prompt you to save a keypair called ``ckan_test.pem``:
+
+.. image :: images/5.png
+ :width: 807px
+
+In this tutorial, we save the keypair in ``~/Downloads/ckan_test.pem``, but you should save it
+somewhere safe.
+
+.. note :: If you plan to boot your EC2 instance from the command line, you need to remember where you've put this file.
+
+
+Boot the EC2 Image
+******************
+
+CKAN requires Ubuntu 10.04 to run (either the i386 or amd64
+architectures). Luckily Canonical provide a `range of suitable images <http://uec-images.ubuntu.com/releases/10.04/release/>`_.
+
+The cheapest EC2 instance is the micro one, but that isn't very powerful, so in this tutorial,
+we'll use the 32-bit small version.
+
+We're in ``eu-west-1`` and we'll use an instance-only image (i.e. all the data will be lost when you shut it down) so we need the `ami-3693a542 <https://console.aws.amazon.com/ec2/home?region=eu-west-1#launchAmi=ami-3693a542>`_ AMI.
+
+.. note ::
+
+ There are more recent Ubuntu images at http://cloud.ubuntu.com/ami/ but we need the older 10.04 LTS release.
+
+At this point, you can either boot this image from the AWS
+console or launch it from the command line.
+
+
+Option 1: Boot the EC2 Image AMI via the AWS Console
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+From the EC2 dashboard, choose **Launch instance >**:
+
+.. image :: images/2.png
+ :width: 807px
+ :alt: Choose launch instance from the EC2 dashboard
+
+Now work through the wizard as shown in the following screenshots.
+
+In the first step search for ``ami-3693a542`` and select it from the results (it may take a few seconds for Amazon to find it).
+
+.. warning ::
+
+ No image other than ``ami-3693a542`` will work with CKAN.
+
+.. image :: images/i1.png
+ :width: 807px
+ :alt: Search for image ami-3693a542
+
+You can keep the defaults for all of the following screens:
+
+.. image :: images/i2.png
+ :width: 807px
+ :alt: Keep the defaults while setting up your instance
+.. image :: images/i3.png
+ :width: 807px
+ :alt: Keep the defaults while setting up your instance
+.. image :: images/i4.png
+ :width: 807px
+ :alt: Keep the defaults while setting up your instance
+.. image :: images/i5.png
+ :width: 807px
+ :alt: Keep the defaults while setting up your instance
+
+Choose the ``web_test`` security group you created earlier:
+
+.. image :: images/i6.png
+ :width: 807px
+ :alt: Choose the web_test security group you created earlier
+
+Then finish the wizard:
+
+.. image :: images/i7.png
+ :width: 807px
+ :alt: Finish the wizard
+
+Finally click the **View your instances on the Instances page** link:
+
+.. image :: images/i8.png
+ :width: 807px
+ :alt: View your instance
+
+After a few seconds you'll see your instance has booted. Now skip to :ref:`log-in-to-instance`.
+
+Option 2: Boot the EC2 Image AMI from the Command Line
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+[You can skip this section if you've just booted from the AWS console and go straight to :ref:`log-in-to-instance`]
+
+To boot from the command line you still need the same information but you enter it in one command. I'll show you now.
+
+Install The EC2 Tools Locally
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you are on Linux, you can just install the tools like this:
+
+::
+
+ sudo apt-get install ec2-ami-tools
+ sudo apt-get install ec2-api-tools
+
+If you are on Windows or Mac you'll need to `download them from the Amazon website <http://aws.amazon.com/developertools/351>`_.
+
+Once the software is installed you can use the files you've just downloaded to do create your instance.
+
+Get Security Certificates
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Next click on the **Account** link, right at the top of the screen, and you'll see this screen:
+
+.. image :: images/6.png
+ :width: 807px
+ :alt: The Account screen
+
+From this screen choose **Security Credentials** from the left hand side. Once
+the page has loaded scroll down and you'll see the **Access Credentials**
+section. Click on the **X.509 Certificate** tab:
+
+.. image :: images/7.png
+ :width: 807px
+ :alt: The Access Credentials screen
+
+Here you'll be able to create an X.509 certificate and private key.
+
+.. tip ::
+
+ You can only have two X.509 certificates at any given time, so you might need
+ to inactivate an old one first and then delete it before you are allowed to
+ create a new one, as in the screenshot above.
+
+Once you click the **Create New Certificate** link you get a popup which allows
+you to download the certificate and private key - do this. Once again, ours are in
+``~/Downloads``, but you should save it somewhere safe.
+
+.. image :: images/8.png
+ :width: 807px
+ :alt: Download your certificate
+
+.. tip ::
+
+ Amazon will only give you a private key file once when you create it so
+ although you can always go back to get a copy of the certificate, you can only
+ get the private key once. Make sure you save it in a safe place.
+
+You now have:
+
+* Your private key (``pk-[ID].pem``)
+* Your certificate file (``cert-[ID].pem``)
+* Your new keypair (``ckan-test.pem``)
+
+The private key and the certificate files have the same name in the ``ID`` part.
+
+Create an Ubuntu Instance
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Once the tools are installed, run this command:
+
+::
+
+ ec2-run-instances ami-3693a542 --instance-type m1.small --region eu-west-1 --group web_test \
+ --key ckan_test \
+ --private-key ~/Downloads/pk-[ID].pem \
+ --cert ~/Downloads/cert-[ID].pem
+
+
+.. note ::
+
+ The ``--key`` argument is the name of the keypair (``ckan_test``), not the certificate
+ itself (``ckan_test.pem``).
+
+.. warning ::
+
+ Amazon charge you for a minimum of one hour usage, so you shouldn't create and
+ destroy lots of EC2 instances unless you want to be charged a lot.
+
+.. _log-in-to-instance:
+
+Log in to the Instance
+**********************
+
+Once your instance has booted, you will need to find out its public DNS. Give it
+a second or two for the instance to load then browse to the running instance in
+the AWS console. If you tick your instance you'll be able to find the public
+DNS by scrolling down to the bottom of the **Description** tag.
+
+.. image :: images/8a.png
+ :width: 807px
+ :alt: Find the public DNS
+
+Here you can see that our public DNS is
+``ec2-79-125-86-107.eu-west-1.compute.amazonaws.com``. The private DNS only works
+from other EC2 instances so isn't any use to us.
+
+Once you've found your instance's public DNS, ensure the key has the correct permissions:
+
+::
+
+ chmod 0600 "ckan_test.pem"
+
+You can then log in like this:
+
+::
+
+ ssh -i ~/Downloads/ckan_test.pem ubuntu at ec2-46-51-149-132.eu-west-1.compute.amazonaws.com
+
+The first time you connect you'll see this, choose ``yes``:
+
+::
+
+ RSA key fingerprint is 6c:7e:8d:a6:a5:49:75:4d:9e:05:2e:50:26:c9:4a:71.
+ Are you sure you want to continue connecting (yes/no)? yes
+ Warning: Permanently added 'ec2-79-125-86-107.eu-west-1.compute.amazonaws.com,79.125.86.107' (RSA) to the list of known hosts.
+
+When you log in you'll see a welcome message. You can now proceed to :ref:`run-package-installer`.
+
+
+.. note ::
+
+ If this is a test install of CKAN, when you have finished using CKAN, you can shut down your EC2 instance through the AWS console.
+
+.. warning ::
+
+ Shutting down your EC2 instance will lose all your data. Also, Amazon charge you for a minimum usage of one hour, so don't create and destroy lots of EC2 instances unless you want to be charged a lot!
+
+
+.. _run-package-installer:
+
+Run the Package Installer
+-------------------------
+
+On your Ubuntu 10.04 system, open a terminal window and switch to the root user:
+
+::
+
+ sudo -s
+
+Install the CKAN packages as follows:
+
+::
+
+ echo 'deb http://apt.okfn.org/ubuntu_ckan-std_dev lucid universe' > /etc/apt/sources.list.d/okfn.list
+ wget -qO- http://apt.okfn.org/packages.okfn.key | sudo apt-key add -
+ apt-get update
+ apt-get install ckan-std
+
+Wait for the output to finish, then create your CKAN instance:
+
+::
+
+ ckan-std-install
+
+If you are using Amazon EC2, you will additionally need to set the hostname of your server. To do this, run the command below, replacing ``ec2-46-51-149-132.eu-west-1.compute.amazonaws.com`` with the public DNS of your EC2 instance. Leave the ``/`` at the end, as it is part of the ``sed`` command. Then restart Apache. You can skip this if installing on VirtualBox or a local server.
+
+::
+
+ sudo sed -e "s/ServerAlias \(.*\)/ServerAlias ec2-46-51-149-132.eu-west-1.compute.amazonaws.com/" \
+ -i /etc/apache2/sites-available/std.common
+ sudo /etc/init.d/apache2 restart
+
+Finally visit your CKAN instance - either at your Amazon EC2 hostname, or at http://localhost. You'll be redirected to the login screen because you won't have set up any permissions yet, so the welcome screen will look something like this.
+
+.. image :: images/9.png
+ :width: 807px
+
+You can now proceed to :doc:`post-installation`.
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/doc/install-from-source.rst Thu Jul 28 17:20:40 2011 +0100
@@ -0,0 +1,270 @@
+=============================
+Option 2: Install from Source
+=============================
+
+This section describes how to install CKAN from source. This removes the requirement for Ubuntu 10.04 that exists with :doc:`install-from-package`.
+
+.. warning:: This option is more complex than :doc:`install-from-package`, so we suggest only doing it this way if you plan to work on CKAN core, or have no access to Ubuntu 10.04 via any of the suggested methods.
+
+For support during installation, please contact `the ckan-dev mailing list <http://lists.okfn.org/mailman/listinfo/ckan-dev>`_.
+
+Install the Source
+------------------
+
+These are instructions to get developing with CKAN.
+
+Before you start, it may be worth `checking CKAN has passed the auto build and
+tests <http://buildbot.okfn.org/waterfall>`_.
+
+
+1. Ensure the required packages are installed.
+
+ If you have access to ``apt-get``, you can install these packages as follows:
+
+ ::
+
+ sudo apt-get install build-essential libxml2-dev libxslt-dev
+ sudo apt-get install wget mercurial postgresql libpq-dev git-core
+ sudo apt-get install python-dev python-psycopg2 python-virtualenv
+ sudo apt-get install subversion
+
+ Otherwise, you should install these packages from source.
+
+ ===================== ===============================================
+ Package Description
+ ===================== ===============================================
+ mercurial `Source control <http://mercurial.selenic.com/>`_
+ python `Python v2.5-2.7 <http://www.python.org/getit/>`_
+ postgresql `PostgreSQL database <http://www.postgresql.org/download/>`_
+ libpq `PostgreSQL library <http://www.postgresql.org/docs/8.1/static/libpq.html>`_
+ psycopg2 `PostgreSQL python module <http://initd.org/psycopg/install/>`_
+ libxml2 `XML library development files <http://xmlsoft.org/>`_
+ libxslt `XSLT library development files <http://www.linuxfromscratch.org/blfs/view/6.3/general/libxslt.html>`_
+ virtualenv `Python virtual environments <http://pypi.python.org/pypi/virtualenv>`_
+ wget `Command line tool for downloading from the web <http://www.gnu.org/s/wget/>`_
+ build-essential Tools for building source code (or up-to-date Xcode on Mac)
+ git `Git source control (for getting MarkupSafe src) <http://book.git-scm.com/2_installing_git.html>`_
+ subversion `Subversion source control (for pyutilib) <http://subversion.apache.org/packages.html>`_
+ ===================== ===============================================
+
+
+
+2. Create a Python virtual environment.
+
+ In your home directory run the command below. It is currently important to
+ call your virtual environment ``pyenv`` so that the automated deployment tools
+ work correctly.
+
+ ::
+
+ cd ~
+ virtualenv pyenv
+
+ .. tip ::
+
+ If you don't have a ``python-virtualenv`` package in your distribution
+ you can get a ``virtualenv.py`` script from within the
+ `virtualenv source distribution <http://pypi.python.org/pypi/virtualenv/>`_
+ and then run ``python virtualenv.py pyenv`` instead.
+
+ To help with automatically installing CKAN dependencies we use a tool
+ called ``pip``. Make sure you have activated your environment (see step 3)
+ and then install it from an activated shell like this:
+
+ ::
+
+ easy_install pip
+
+3. Activate your virtual environment.
+
+ To work with CKAN it is best to adjust your shell settings so that your
+ shell uses the virtual environment you just created. You can do this like
+ so:
+
+ ::
+
+ . pyenv/bin/activate
+
+ When your shell is activated you will see the prompt change to something
+ like this:
+
+ ::
+
+ (pyenv)[ckan at host ~/]$
+
+ An activated shell looks in your virtual environment first when choosing
+ which commands to run. If you enter ``python`` now it will actually
+ run ``~/pyenv/bin/python`` which is what you want.
+
+4. Install CKAN code and required Python packages into the new environment.
+
+ First you'll need to install CKAN. For the latest version run:
+
+ ::
+
+ pip install --ignore-installed -e hg+http://bitbucket.org/okfn/ckan#egg=ckan
+
+ CKAN has a set of dependencies it requires which you should install too:
+
+ ::
+
+ pip install --ignore-installed -r pyenv/src/ckan/requires/lucid_missing.txt -r pyenv/src/ckan/requires/lucid_conflict.txt
+
+ The ``--ignore-installed`` option ensures ``pip`` installs software into
+ this virtual environment even if it is already present on the system.
+
+ If you are using Ubuntu Lucid you can install the rest of the dependencies
+ from the system versions like this:
+
+ ::
+
+ sudo apt-get install python-psycopg2 python-lxml python-sphinx
+ sudo apt-get install python-pylons python-formalchemy python-repoze.who
+ sudo apt-get install python-repoze.who-plugins python-tempita python-zope.interface
+
+ If you are not using Ubuntu Lucid you'll still need to install all the
+ dependencies that would have been met in the ``apt-get install`` command
+ at the start. You can do so like this:
+
+ ::
+
+ pip install --ignore-installed -r pyenv/src/ckan/requires/lucid_present.txt
+
+ This will take a **long** time. Particularly the install of the ``lxml``
+ package.
+
+ At this point you will need to deactivate and then re-activate your
+ virtual environment to ensure that all the scripts point to the correct
+ locations:
+
+ ::
+
+ deactivate
+ . pyenv/bin/activate
+
+5. Setup a PostgreSQL database.
+
+ List existing databases:
+
+ ::
+
+ psql -l
+
+ It is advisable to ensure that the encoding of databases is 'UTF8', or
+ internationalisation may be a problem. Since changing the encoding of PostgreSQL
+ may mean deleting existing databases, it is suggested that this is fixed before
+ continuing with the CKAN install.
+
+ Next you'll need to create a database user if one doesn't already exist.
+
+ .. tip ::
+
+ If you choose a database name, user or password which are different from those
+ suggested below then you'll need to update the configuration file you'll create in
+ the next step.
+
+ Here we choose ``ckantest`` as the database and ``ckanuser`` as the user:
+
+ ::
+
+ sudo -u postgres createuser -S -D -R -P ckantest
+
+ It should prompt you for a new password for the CKAN data in the database.
+ It is suggested you enter ``pass`` for the password.
+
+ Now create the database, which we'll call ``ckantest`` (the last argument):
+
+ ::
+
+ sudo -u postgres createdb -O ckantest ckantest
+
+6. Create a CKAN config file.
+
+ Make sure you are in an activated environment (see step 3) so that Python
+ Paste and other modules are put on the python path (your command prompt will
+ start with ``(pyenv)`` if you have) then change into the ``ckan`` directory
+ which will have been created when you installed CKAN in step 4 and create the
+ config file ``development.ini`` using Paste:
+
+ ::
+
+ cd pyenv/src/ckan
+ paster make-config ckan development.ini
+
+ You can give your config file a different name but the tests will expect you
+ to have used ``development.ini`` so it is strongly recommended you use this
+ name, at least to start with.
+
+ If you used a different database name or password when creating the database
+ in step 5 you'll need to now edit ``development.ini`` and change the
+ ``sqlalchemy.url`` line, filling in the database name, user and password you used.
+
+ ::
+
+ sqlalchemy.url = postgresql://ckantest:pass@localhost/ckantest
+
+ If you're using a remote host with password authentication rather than SSL authentication, use::
+
+ sqlalchemy.url = postgresql://<user>:<password>@<remotehost>/ckan?sslmode=disable
+
+ .. caution ::
+
+ Advanced users: If you are using CKAN's fab file capability you currently need to create
+ your config file as ``pyenv/ckan.net.ini`` so you will probably have
+ ignored the advice about creating a ``development.ini`` file in the
+ ``pyenv/src/ckan`` directory. This is fine but CKAN probably won't be
+ able to find your ``who.ini`` file. To fix this edit ``pyenv/ckan.net.ini``,
+ search for the line ``who.config_file = %(here)s/who.ini`` and change it
+ to ``who.config_file = who.ini``.
+
+ We are moving to a new deployment system where this incompatibility
+ will be fixed.
+
+7. Create database tables.
+
+ Now that you have a configuration file that has the correct settings for
+ your database, you'll need to create the tables. Make sure you are still in an
+ activated environment with ``(pyenv)`` at the front of the command prompt and
+ then from the ``pyenv/src/ckan`` directory run this command:
+
+ ::
+
+ paster db init
+
+ You should see ``Initialising DB: SUCCESS``. If you are not in the
+ ``pyenv/src/ckan`` directory or you don't have an activated shell, the command
+ will not work.
+
+ If the command prompts for a password it is likely you haven't set up the
+ database configuration correctly in step 6.
+
+8. Create the cache directory.
+
+ You need to create the Pylon's cache directory specified by 'cache_dir'
+ in the config file.
+
+ (from the ``pyenv/src/ckan`` directory):
+
+ ::
+
+ mkdir data
+
+
+9. Run the CKAN webserver.
+
+ NB If you've started a new shell, you'll have to activate the environment
+ again first - see step 3.
+
+ (from the ``pyenv/src/ckan`` directory):
+
+ ::
+
+ paster serve development.ini
+
+10. Point your web browser at: http://127.0.0.1:5000/
+
+ The CKAN homepage should load.
+
+Finally, make sure that tests pass, as described in :ref:`basic-tests`.
+
+You can now proceed to :doc:`post-installation`.
--- a/doc/load_testing.rst Thu Jul 28 16:26:50 2011 +0100
+++ /dev/null Thu Jan 01 00:00:00 1970 +0000
@@ -1,70 +0,0 @@
-=================
-Load Testing CKAN
-=================
-
-What we did
-===========
-
-1. Put db in some standard state and start server::
-
- paster db clean && paster db create && paster create-test-data search
- # alternatively we could use a dump of the live db
- # psql -h localhost --user tester ckantest < ....
-
- # may want to create a production config
- paster serve development.ini
-
-2. Do tests::
-
- # 5 results for this query
- ab -n 100 -c 10 -e myresults.csv "http://localhost:5000/api/search/package?q=government&all_fields=1"
- # remote test host
- # ab -n 100 -c 10 -e myresults.csv "http://test.ckan.net/api/search/package?q=geo&all_fields=1"
-
-3. Examine results, fix, and repeat!
-
-
-Research
-========
-
-Testing Tools
-+++++++++++++
-
-Apache Benchmarking Tool
-------------------------
-
- * http://httpd.apache.org/docs/2.0/programs/ab.html
- * Well known and standard.
- * Can run against local or remote but local preferred
- * ab -n 100 -c 10 -e myresults.csv http://www.okfn.org/
-
-funkload
---------
-
- * Mature and python based
- * http://funkload.nuxeo.org/
- * Functional testing primarily but with load testing support
-
-Individual scripts
-------------------
-
- * http://code.google.com/appengine/articles/load_test.html
-
-
-High Performance Servers and Caching Utilities
-----------------------------------------------
-
-memcached
-+++++++++
-
- * http://memcached.org/
- * In memory caching with access via a port
- * Very heavily used
-
-Tornado
-+++++++
-
- * http://www.tornadoweb.org/
- * Built by friendfeed and open-sourced
- * Asynchronous
-
--- a/doc/loader_scripts.rst Thu Jul 28 16:26:50 2011 +0100
+++ /dev/null Thu Jan 01 00:00:00 1970 +0000
@@ -1,31 +0,0 @@
-==============
-Loader scripts
-==============
-
-Introduction
-============
-
-'Loader scripts' provide a simple way to take any format metadata and bulk upload it to a remote CKAN instance. Essentially each custom script has to convert the metadata to the standard 'package' dictionary format, and the loader does the work of loading it into CKAN in an intelligent way.
-
-
-Structure
-=========
-
-First you need an importer that derives from `PackageImporter` (ckan/lib/importer.py). This takes whatever format the metadata is in and sorts it into records of type `DataRecord`. Then each DataRecord is converted into the correct fields for a package using the `record_2_package` method. This results in package dictionaries.
-
-Note: for CSV and Excel formats, there is already the `SpreadsheetPackageImporter` (ckan/lib/spreadsheet_importer.py) which wraps the file in `SpreadsheetData` before extracting the records into `SpreadsheetDataRecords`.
-
-The `PackageLoader` takes the package dictionaries and loads them onto a CKAN instance using the ckanclient. There are various settings to determine:
- * how to identify the same package, previously been loaded into CKAN. This canbe simply by name or by an identifier stored in another field.
- * how to merge in changes to an existing packages. It can simply replace it or maybe merge in resources etc.
-
-Loaders generally go into the `ckanext` repository.
-
-The loader shoud be given a command line interface using the `Command` base class (ckanext/command.py). You need to add a line to the setup.py `[console_scripts]` and when you run ``python setup.py develop`` it creates a script for you in your python environment.
-
-
-Example
-=======
-
-To get a flavour of what these scripts look like, take a look at the ONS scripts: https://bitbucket.org/okfn/ckanext-dgu/src/default/ckanext/dgu/ons/
-
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/doc/loading_data.rst Thu Jul 28 17:20:40 2011 +0100
@@ -0,0 +1,99 @@
+=============
+Load Datasets
+=============
+
+You can upload individual datasets through the CKAN front-end, but for importing datasets on masse, you have two choices:
+
+* :ref:`load-data-api`. You can use the `CKAN API <api.html>`_ to script import. To simplify matters, we offer provide standard loading scripts for Google Spreadsheets, CSV and Excel.
+
+* :ref:`load-data-harvester`. The `CKAN harvester extension <https://bitbucket.org/okfn/ckanext-harvest/>`_ provides web and command-line interfaces for larger import tasks.
+
+If you need advice on data import, `contact the ckan-dev mailing list <http://lists.okfn.org/mailman/listinfo/ckan-dev>`_.
+
+.. note :: If loading your data requires scraping a web page regularly, you may find it best to write a scraper on `ScraperWiki <http://www.scraperwiki.com>`_ and combine this with either of the methods above.
+
+.. _load-data-api:
+
+Import Data with the CKAN API
+-----------------------------
+
+You can use the `CKAN API <api.html>`_ to upload datasets directly into your CKAN instance.
+
+The Simplest Approach - ckanclient
+++++++++++++++++++++++++++++++++++
+
+The most basic way to automate package loading is with a Python script using the `ckanclient library <http://pypi.python.org/pypi/ckanclient>`_. You will need to register for an API key first.
+
+You can install ckanclient with::
+
+ pip install ckanclient
+
+Here is an example script to register a new package::
+
+ import ckanclient
+ # Instantiate the CKAN client.
+ ckan = ckanclient.CkanClient(api_key=my_api_key, base_location="http://myckaninstance.com/api")
+ # Describe the package.
+ package_entity = {
+ 'name': my_package_name,
+ 'url': my_package_url,
+ 'download_url': my_package_download_url,
+ 'tags': my_package_keywords,
+ 'notes': my_package_long_description,
+ }
+ # Register the package.
+ ckan.package_register_post(package_entity)
+
+Loader Scripts
+++++++++++++++
+
+'Loader scripts' provide a simple way to take any format metadata and bulk upload it to a remote CKAN instance.
+
+Essentially each set of loader scripts converts the dataset metadata to the standard 'package' format, and then loads it into CKAN.
+
+Loader scripts are generally stored into the `ckanext` repository. To get a flavour of what loader scripts look like, take a look at `the ONS scripts <https://bitbucket.org/okfn/ckanext-dgu/src/default/ckanext/dgu/ons/>`_.
+
+Loader Scripts for CSV and Excel
+********************************
+
+For CSV and Excel formats, the `SpreadsheetPackageImporter` (found in ``ckan/lib/spreadsheet_importer.py``) loader script wraps the file in `SpreadsheetData` before extracting the records into `SpreadsheetDataRecords`.
+
+SpreadsheetPackageImporter copes with multiple title rows, data on multiple sheets, dates. The loader can reload packages based on a unique key column in the spreadsheet, choose unique names for packages if there is a clash, add/merge new resources for existing packages and manage package groups.
+
+Loader Scripts for Google Spreadsheets
+**************************************
+
+The `GoogleSpreadsheetReader` class (found in ``ckanclient.loaders``) simplifies the process of loading data from Google Spreadsheets.
+
+`This script <https://bitbucket.org/okfn/ckanext/src/default/bin/ckanload-italy-nexa>`_ has a simple example of loading data from Google Spreadsheets.
+
+Write Your Own Loader Script
+****************************
+
+## this needs work ##
+
+First, you need an importer that derives from `PackageImporter` (found in ``ckan/lib/importer.py``). This takes whatever format the metadata is in and sorts it into records of type `DataRecord`.
+
+Next, each DataRecord is converted into the correct fields for a package using the `record_2_package` method. This results in package dictionaries.
+
+The `PackageLoader` takes the package dictionaries and loads them onto a CKAN instance using the ckanclient. There are various settings to determine:
+
+ * ##how to identify the same package, previously been loaded into CKAN.## This can be simply by name or by an identifier stored in another field.
+ * how to merge in changes to an existing packages. It can simply replace it or maybe merge in resources etc.
+
+The loader should be given a command-line interface using the `Command` base class (``ckanext/command.py``).
+
+You need to add a line to the CKAN ``setup.py`` (under ``[console_scripts]``) and when you run ``python setup.py develop`` it creates a script for you in your Python environment.
+
+.. _load-data-harvester:
+
+Import Data with the Harvester Extension
+----------------------------------------
+
+The `CKAN harvester extension <https://bitbucket.org/okfn/ckanext-harvest/>`_ provides useful tools for more advanced data imports.
+
+These include a command-line interface and a web user interface for running harvesting jobs.
+
+To use the harvester extension, derive from the `base class of the harvester extension <https://bitbucket.org/okfn/ckanext-harvest/src/61844c8d2374/ckanext/harvest/harvesters/base.py>`_ and then write a custom ``_create_or_update_package`` method for your data.
+
+For more information on working with extensions, see :doc:`extensions`.
--- a/doc/model.rst Thu Jul 28 16:26:50 2011 +0100
+++ /dev/null Thu Jan 01 00:00:00 1970 +0000
@@ -1,96 +0,0 @@
-=====
-Model
-=====
-
-The structure of the CKAN data is described in the 'model'. This is in the code at::
-
- ckan/model
-
-Many of the domain objects are Revisioned and some are Stateful. These are concepts introduced by vdm.
-
-Migration
-=========
-
-When edits are made to the model code, then before the code can be used on a CKAN instance with existing data, the existing data has to be migrated. This is achieved with a migration script. CKAN currently uses `SqlAlchemy Migrate <http://code.google.com/p/sqlalchemy-migrate/>`_ to manage these scripts.
-
-When you deploy new code to a CKAN instance, as part of the process you run any required migration scripts with::
-
- paster db upgrade
-
-The scripts give their model version numbers in their filenames and are stored here::
-
- ckan/migration/versions/
-
-The current version the database is migrated to is also stored in the database. When you run the upgrade, as each migration script is run it prints to the console something like ``11->12``. If no upgrade is required because it is up to date, then nothing is printed.
-
-Creating a new migration script
-===============================
-
-A migration script should be checked into CKAN at the same time as the model changes it is related to. Before pushing the changes, ensure the tests pass when running against the migrated model, which requires the ``--ckan-migration`` setting - see `<README.html#migrationtesting>`_.
-
-To create a new migration script, create a python file in ckan/migration/versions/ and name it with a prefix numbered one higher than the previous one and some words describing the change.
-
-You need to use the special engine provided by the SqlAlchemy Migrate. Here is the standard header for your migrate script::
-
- from sqlalchemy import *
- from migrate import *
-
-
-The migration operations go in the upgrade function::
-
- def upgrade(migrate_engine):
- metadata = MetaData()
- metadata.bind = migrate_engine
-
-The following process should be followed when doing a migration. This process is here to make the process easier and to validate if any mistakes have been made.
-
-1. Get a dump of the database schema before you add your new migrate scripts.
-
- paster db clean
- paster db upgrade
- pg_dump -h host -s -f old.sql dbname
-
-2. Get a dump of the database as you have specified it in the model.
-
- paster db clean
- paster db create-test #this makes the database as defined in the model
- pg_dump -h host -s -f new.sql dbname
-
-3. Get agpdiff (apt-get it). It produces sql it thinks that you need to run on the database in order to get it to the updated schema.
-
- apgdiff old.sql new.sql > upgrade.diff
- (or if you don't want to install java use http://apgdiff.startnet.biz/diff_online.php)
-
-4. The upgrade.diff file created will have all the changes needed in sql. Delete the drop index lines as they are not created in the model.
-
-5. Put the resulting sql in your migrate script.
-
- eg migrate_engine.execute('''update table .........; update table ....''')
-
-6. Do a dump again, then a diff again to see if the the only thing left are drop index statements.
-
-7. run nosetests with --ckan-migration flag.
-
-Its that simple. Well almost..
-
-* If you are doing any table/field renaming adding that to your new migrate script first and use this as a base for your diff (i.e add a migrate script with these renaming before 1). This way the resulting sql wont try and drop and recreate the field/table!!
-* It sometimes drops the foreign key constraints in the wrong order causing an error so you may need to rearrange the order in the resulting upgrade.diff.
-* If you need to do any data transfer in the migrations then do it between the dropping of the constraints and adding of new ones.
-* May need to add some tests if you are doing data migrations.
-
-An example of a script doing it this way is 034_resource_group_table.py. This script copies the definitions of the original tables in order to do the renaming the tables/fields.
-
-In order to do some basic data migration testing extra assertions should be added to the migration script.
-
-Examples of this can also be found in 034_resource_group_table.py for example.
-
-This statement is run at the top of the migration script to get the count of rows::
-
- package_count = migrate_engine.execute('''select count(*) from package''').first()[0]
-
-And the following is run after to make sure that row count is the same::
-
- resource_group_after = migrate_engine.execute('''select count(*) from resource_group''').first()[0]
- assert resource_group_after == package_count
-
-
--- a/doc/paster.rst Thu Jul 28 16:26:50 2011 +0100
+++ b/doc/paster.rst Thu Jul 28 17:20:40 2011 +0100
@@ -1,16 +1,81 @@
-Paster commands
-+++++++++++++++
+=================
+Common CKAN Tasks
+=================
-Paster commands provide a number of useful ways to administer a CKAN installation. These are run on the command line on the server running CKAN.
+The majority of common CKAN administration tasks are carried out using the **paster** script.
-Commands
-========
+Paster is run on the command line on the server running CKAN. This section covers:
+
+* :ref:`paster-understanding`. Understanding paster syntax and getting help.
+* :ref:`paster-tasks`. How to carry out common CKAN admin tasks using paster.
+
+.. _paster-understanding:
+
+Understanding Paster
+====================
+
+The basic paster format is::
+
+ paster --plugin=ckan <ckan commands> --config=<config file>
+
+For example, to initialise a database::
+
+ paster --plugin=ckan db init --config=/etc/ckan/std/std.ini
+
+
+.. _paster-help:
+
+Getting Help on Paster
+----------------------
+
+To get a full list of paster commands (i.e. including CKAN commands)::
+
+ paster --plugin=ckan --help
+
+And to get more detailed help on each command (e.g. on ``db``)::
+
+ paster --plugin=ckan --help db
+
+
+Position of Paster Parameters
+-----------------------------
+
+The position of paster parameters matters.
+
+``--plugin`` is a parameter to paster, so needs to come before the CKAN command. To do this, the first parameter to paster is normally ``--plugin=ckan``.
+
+.. note:: The default value for ``--plugin`` is ``setup.py`` in the current directory. If you are running paster from the directory where CKAN's ``setup.py`` file is located, you don't need to specify the plugin parameter..
+
+Meanwhile, ``--config`` is a parameter to CKAN, so needs to come after the CKAN command. This specifies the CKAN config file for the instance you want to use, e.g. ``--config=/etc/ckan/std/std.ini``
+
+.. note:: The default value for ``--config`` is ``development.ini`` in the current directory. If you are running a package install of CKAN (as described in :doc:`install-from-package`), you should explicitly specify ``std.ini``.
+
+The position of the CKAN command itself is less important, as longs as it follows ``--plugin``. For example, both the following commands have the same effect:::
+
+ paster --plugin=ckan db --config=development.ini init
+ paster --plugin=ckan db init --config=development.ini
+
+
+Running a Paster Shell
+----------------------
+
+If you want to run a "paster shell", which can be useful for development, then the plugin is pylons. e.g. ``paster --plugin=pylons shell``.
+
+Often you will want to run this as the same user as the web application, to ensure log files are written as the same user. And you'll also want to specify a config file (note that this is not specified using the ``--config`` parameter, but simply as the final argument). For example::
+
+ sudo -u www-data paster --plugin=pylons shell std.ini
+
+
+.. _paster-tasks:
+
+Common Tasks Using Paster
+=========================
+
+The following tasks are supported by paster.
================= ==========================================================
- changes Distribute changes
create-test-data Create test data in the database.
db Perform various tasks on the database.
- notify Send out modification notifications.
ratings Manage the ratings stored in the db
rights Commands relating to per-object and system-wide access rights.
roles Commands relating to roles and actions.
@@ -20,55 +85,98 @@
================= ==========================================================
-Running paster commands
-=======================
+For the full list of tasks supported by paster, you can run::
+
+ paster --plugin=ckan --help
-Basically the format is::
- paster --plugin=ckan <ckan commands> --config=<config file>
+create-test-data: Create test data
+----------------------------------
-e.g.::
+As the name suggests, this command lets you load test data when first setting up CKAN. See :ref:`create-test-data` for details.
- paster --plugin=ckan db init --config=myckan.ini
-
-To get a list of paster commands (i.e. including CKAN commands)::
-
- paster --plugin=ckan --help
-
-And you can get help for each command. e.g.::
-
- paster --plugin=ckan --help db
-
-
-``--plugin`` parameter
+db: Manage databases
--------------------
-Paster is a system-wide command, so you need to tell it to find the commands defined in the ckan code. So first parameter to paster is normally ``--plugin=ckan``.
+Lets you initialise, upgrade, and dump database files in various formats.
-NB: If your current directory is where CKAN's setup.py is, you don't need to specify this parameter.
+For example, to initialise the CKAN database, creating the tables that CKAN uses (note that you don't need to do this during setup if you have run ``create-test-data``)::
-If you want to run a "paster shell" then the plugin is pylons. e.g. ``paster --plugin=pylons shell``. Often you will want to run this as the same user as the web application, to ensure log files are written as the same user. And you'll also want to specify a config file. e.g.::
+ paster --plugin=ckan db init --config=/etc/ckan/std/std.ini
- sudo -u www-data paster --plugin=pylons shell myckan.ini
+When you upgrade CKAN software by any method *other* than the package update described in :doc:`upgrade`, before you restart it, you should run 'db upgrade', to migrate the database tables if necessary::
+ paster --plugin=ckan db upgrade --config=/etc/ckan/std/std.ini
-``--config`` parameter
-----------------------
+For information on using ``db`` to create dumpfiles, see :doc:`database_dumps`.
-Each server can have multiple CKAN instances running, so use this parameter to specify the CKAN config file for the instance you want to operate on. e.g. ``--config=myckan.ini``
-NB: Developers tend to use development.ini, and this is the default value (looking in the current directory), so in this situation you don't need to specify this parameter.
+ratings: Manage package ratings
+-------------------------------
+Manages the ratings stored in the database, and can be used to count ratings, remove all ratings, or remove only anonymous ratings.
-Position of paster parameters
+For example, to remove anonymous ratings from the database::
+
+ paster --plugin=ckan ratings clean-anonymous --config=/etc/ckan/std/std.ini
+
+
+rights: Set user permissions
+----------------------------
+
+Sets the authorization roles of a specific user on a given object within the system.
+
+For example, to give the user named 'bar' the 'admin' role on the package 'foo'::
+
+ paster --plugin=ckan rights make bar admin package:foo --config=/etc/ckan/std/std.ini
+
+To list all the rights currently specified::
+
+ paster --plugin=ckan rights list --config=/etc/ckan/std/std.ini
+
+For more information and examples, see :doc:`authorization`.
+
+
+roles: Manage system-wide permissions
+--------------------------------------
+
+This important command gives you fine-grained control over CKAN permissions, by listing and modifying the assignment of actions to roles.
+
+The ``roles`` command has its own section: see :doc:`authorization`.
+
+
+search-index: Rebuild search index
+----------------------------------
+
+Rebuilds the search index defined in the :ref:`config-search-backend` config setting. This is useful to prevent search indexes from getting out of sync with the main database.
+
+For example::
+
+ paster --plugin=ckan search-index --config=/etc/ckan/std/std.ini
+
+
+sysadmin: Give sysadmin rights
+------------------------------
+
+Gives sysadmin rights to a named user. This means the user can perform any action on any object.
+
+For example, to make a user called 'admin' into a sysadmin::
+
+ paster --plugin=ckan sysadmin add admin --config=/etc/ckan/std/std.ini
+
+
+.. _paster-user:
+
+user: Create and manage users
-----------------------------
-``--plugin`` is a paster parameter, so needs to come before the CKAN command, but <code>--config</code> is a ckan parameter so needs to come after the CKAN command.
+Lets you create, remove, list and manage users.
-Here are three ways to express the same thing::
+For example, to create a new user called 'admin'::
- paster db init
- paster --plugin=ckan db --config=development.ini init
- paster --plugin=ckan db init --config=development.ini
+ paster --plugin=ckan user add admin --config=/etc/ckan/std/std.ini
+To delete the 'admin' user::
+
+ paster --plugin=ckan user delete admin --config=/etc/ckan/std/std.ini
--- a/doc/plugins.rst Thu Jul 28 16:26:50 2011 +0100
+++ b/doc/plugins.rst Thu Jul 28 17:20:40 2011 +0100
@@ -1,44 +1,24 @@
-CKAN Extensions, Plugins Interfaces and Workers
-+++++++++++++++++++++++++++++++++++++++++++++++
+===============================
+Understand and Write Extensions
+===============================
-**Note: the terms "extension", "plugin interface" and "worker"
-now have very precise meanings and that the use of the generic word "plugin" to
-describe any way in which CKAN might be extended is deprecated.**
+If you want to extend CKAN core functionality, the best way to do so is by writing extensions.
-.. contents ::
+Extensions allow you to customise CKAN for your own requirements, without interfering with the basic CKAN system.
-Background
-----------
-
-The CKAN codebase is open source so that different organisations can customise
-it for their own needs. As CKAN has grown in popularity it has become
-especially important to organise the code in a way that can accommodate the
-customisations which different organisations require without those changes
-interfering with customizations required by other organisations.
-
-To meet this need we have introduced the concepts of CKAN extensions, plugin
+To meet the need to customize CKAN efficiently, we have introduced the concepts of CKAN extensions, plugin
interfaces and workers. These work together to provide a simple mechanism to
extend core CKAN functionality.
-Let's start by looking at extensions.
+.. warning:: This is an advanced topic. At the moment, you need to have prepared your system to work with extensions, as described in :doc:`prepare-extensions`. We are working to make the most popular extensions more easily available as Debian packages.
+
+.. note:: The terms **extension**, **plugin interface** and **worker** have very precise meanings: the use of the generic word **plugin** to describe any way in which CKAN might be extended is deprecated.
+
+.. contents ::
CKAN Extensions
---------------
-CKAN currently has the following extensions:
-
-* disqus_ extension for user comments
-* Solr extension for full text search and result faceting
-* Asyncronous queue extension based on AMQP for model update, harvesting and other operations
-* Custom form extensions for different governments
-
-.. note ::
-
- The form extension does not currently behave quite the same way as the other
- extensions, it will do soon.
-
-All CKAN extensions have package names of the form ``ckanext-name`` so the
-queue extension code is stored in the package ``ckanext-queue`` for example.
Extensions are implemented as *namespace packages* under the ``ckanext``
package which means that they can be imported like this:
@@ -47,48 +27,11 @@
$ python
>>> import ckanext.queue
-Extensions are used for any code that is not required for the core CKAN code to
-operate but which are nethertheless and important part of one or more CKAN
-instances.
-
Individual CKAN *extensions* may implement one or more *plugin interfaces* or
*workers* to provide their functionality. You'll learn about these later on.
-All CKAN extensions are described on the CKAN public wiki at
-http://wiki.okfn.org/ckan/extensions. If you write an extension then do share
-details of it there but also document the plugin interfaces the extension
-implements so that when people install it, they will know which plugin
-interfaces they need to set up in their config file.
-Installing an extension
-~~~~~~~~~~~~~~~~~~~~~~~
-
-To install an extension on a CKAN instance:
-
-1. Install the extension package code using pip. The -E parameter is for your
-CKAN python environment (e.g. ``~/var/srvc/ckan.net/pyenv``). Prefix the source
-url with the repo type (``hg+`` for Mercurial, ``git+`` for Git). For example::
-
- $ pip install -E ~/var/srvc/ckan.net/pyenv hg+http://bitbucket.org/okfn/ckanext-disqus
-
-2. Add the names of any plugin implementations the extension uses to the CKAN
-config. The config file may have a filepath something like:
-``~/var/srvc/ckan.net/ckan.net.ini``. The plugins variable is in the '[app:main]'
-section under 'ckan.plugins'. e.g.::
-
- [app:main]
- ckan.plugins = disqus
-
- If your extension implemented multiple different plugin interfaces, separate them with spaces::
-
- ckan.plugins = disqus amqp myplugin
-
-3. Restart WSGI, which usually means restarting Apache::
-
- $ sudo /etc/init.d/apache2 restart
-
-
-Creating your own extension
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Create Your Own Extension
+~~~~~~~~~~~~~~~~~~~~~~~~~
All CKAN extensions must start with the name ``ckanext-``. You can create your
own CKAN extension like this:
@@ -146,9 +89,35 @@
To build useful extensions you need to be able to "hook into" different parts
of CKAN in order to extend its functionality. You do this using CKAN's plugin
-architeture. We'll look at this in the next section. If you do write a CKAN
-extension you may well want to publish it publicly so others can use it too.
-See the `Publishing your extension`_ section below to find out how.
+architecture. We'll look at this in the next section.
+
+Testing Extensions
+``````````````````
+
+CKAN extensions ordinarily have their own ``test.ini`` that refers to the CKAN ``test.ini``, so you can run them in exactly the same way. For example::
+
+ cd ckanext-dgu
+ nosetests ckanext/dgu/tests --ckan
+ nosetests ckanext/dgu/tests --ckan --with-pylons=test-core.ini
+
+To test your changes you'll need to use the ``paster serve`` command from the ``ckan`` directory:
+
+::
+
+ cd /home/ubuntu/pyenv/src/ckan
+ . ../../bin/activate
+ paster make-config ckan development.ini
+
+Then make any changes to the ``development.ini`` file that you need before continuing:
+
+::
+
+ paster db upgrade
+ paster serve --reload
+
+You should also make sure that your CKAN installation passes the developer tests, as described in :doc:`test`.
+
+Finally, if you write a CKAN, extension you may well want to publish it so others can use it too. See the `Publishing your extension`_ section below for details.
Plugins
-------
@@ -232,11 +201,11 @@
.. tip ::
- This example is based on real code used to implement the ``ckanext-discus`` plugin
+ This example is based on real code used to implement the ``ckanext-disqus`` plugin
to add commenting to packages. You can see the latest version of this code at
http://bitbucket.org/okfn/ckanext-disqus/src/tip/ckanext/plugins/disqus/__init__.py.
-First we set up logging and some helpers we'll need from Genshi to transfor the stream:
+First we set up logging and some helpers we'll need from Genshi to transfer the stream:
::
@@ -256,11 +225,11 @@
from ckan.plugins.interfaces import IConfigurable, IGenshiStreamFilter
In this case we are implementing both the ``IConfigurable`` and
-``IGenshiStreamFilter`` plugin interfaces in out plugin class. The
+``IGenshiStreamFilter`` plugin interfaces in our plugin class. The
``IConfigurable`` plugin interface defines a ``configure()`` method which will
be is called on out plugin to let it know about configuration options. The
``IGenshiStreamFilter`` plugin interface defines a ``filter()`` method which
-will be called on the plugin to give it the oppurtunity to change the template
+will be called on the plugin to give it the opportunity to change the template
before the HTML is returned to the browser.
Let's have a look at the code:
@@ -337,7 +306,7 @@
interfaces to define your own "hooks". Before we can use the ``Disqus`` plugin
there is one more thing to do: add it to the extension and set an *entry point*.
-Setting the entry point
+Setting the Entry Point
~~~~~~~~~~~~~~~~~~~~~~~
Imagine the code above was saved into a file named ``disqus.py`` in the
@@ -359,7 +328,6 @@
CKAN finds plugins by searching for entry points in the group ``ckan.plugin``.
-
Entry points are defined in a package's ``setup.py`` file. If you look in the
``setup.py`` file for the ``ckanext-myname`` extension you'll see these
lines commented out towards the end:
@@ -411,10 +379,10 @@
``python setup.py develop`` if needed). Your users will then need to add the
new entry point name to their ``ckan.plugins`` config option too.
-Writing a database plugin
+Writing a Database Plugin
~~~~~~~~~~~~~~~~~~~~~~~~~
-You've seen how to use ``IConfigurable`` and ``IGenshiStreamFilter``, here's
+You've seen how to use ``IConfigurable`` and ``IGenshiStreamFilter``. Here's
another example which implements ``IMapperExtension`` to log messages after any
record is inserted into the database.
@@ -447,10 +415,11 @@
Publishing Your Extension
~~~~~~~~~~~~~~~~~~~~~~~~~
-At this point you might want to share your extension with the public. First
-check you have chosen an open source license (`MIT
-<http://opensource.org/licenses/mit-license.html>`_ license is fine for
-example) and then update the ``long_description`` variable in ``setup.py`` to
+At this point you might want to share your extension with the public.
+
+First check you have chosen an open source licence (e.g. the `MIT
+<http://opensource.org/licenses/mit-license.html>`_ licence) and then
+update the ``long_description`` variable in ``setup.py`` to
explain what the extension does and which entry point names a user of the
extension will need to add to their ``ckan.plugins`` configuration.
@@ -462,14 +431,14 @@
python setup.py register
python setup.py sdist upload
-You'll then see your extension at http://pypi.python.org/pypi. Others will then
+You'll then see your extension at http://pypi.python.org/pypi. Others will
be able to install your plugin with ``pip``.
-You should also add a summary of your extension and its entry points to
-http://wiki.okfn.org/ckan/extensions. You can create a new account if you need
-to.
+Finally, please also add a summary of your extension and its entry points to the Extensions page on
+http://wiki.ckan.net.
-Writing a plugin interface
+
+Writing a Plugin Interface
~~~~~~~~~~~~~~~~~~~~~~~~~~
This describes how to add a plugin interface to make core CKAN code pluggable.
@@ -509,11 +478,11 @@
your class via ``self.plugin``.
See the pyutilib_ documentation for more information on creating interfaces and
-plugins. Be aware though that pyutilib uses slightly different terminology. It
+plugins. However, be aware that pyutilib uses slightly different terminology. It
calls ``PluginImplementations`` ``ExtensionPoint`` and it calls instances of a
plugin object a *service*.
-Testing plugins
+Testing Plugins
~~~~~~~~~~~~~~~
When writing tests for your plugin code you will need setup and teardown code
@@ -552,23 +521,22 @@
cls.app = paste.fixture.TestApp(wsgiapp)
At this point you should be able to write your own plugins and extensions
-together with their tests. Over time we hope to move more functionality out
-into CKAN extensions.
+together with their tests.
-Ordering of extensions
+Ordering of Extensions
~~~~~~~~~~~~~~~~~~~~~~
.. caution ::
- The order that extensions are initially loaded is **different** to the order that their plugins are run.
+ The order in which extensions are initially loaded is **different** to the order that their plugins are run.
-The order that extensions are initially loaded is as follows:
+The order in which extensions are initially loaded is as follows:
1. System plugins (in setup.py under ``ckan.system_plugins``).
2. In order of the plugins specified in the config file: ``plugins =``.
-3. If more than one module has a plug-in with the same name specified in the config, then all those are loaded, in the order the modules appear in ``sys.path``.
+3. If more than one module has a plugin with the same name specified in the config, then all those are loaded, in the order the modules appear in ``sys.path``.
The order that a plugins are run in, for example the order that IRoutes extensions have their ``before_map`` method run, is alphabetical by the plugin class.
@@ -577,7 +545,7 @@
(This alphabetical ordering is done by ``pyutilib.component.core:ExtensionPoint.extensions()``)
-Plugin API documentation
+Plugin API Documentation
~~~~~~~~~~~~~~~~~~~~~~~~
.. automodule:: ckan.plugins.core
@@ -595,8 +563,8 @@
The Queue Extension
-------------------
-** Note: the queue extension currently isn't working correctly. These docs
-may not work for you**
+.. warning:: The queue extension is currently under development. These docs may not work for you.
+
Certain tasks that CKAN performs lend themselves to the use of a queue system.
Queue systems can be very simple. At their heart is the idea that you have two
@@ -641,7 +609,7 @@
yum install ncurses-devel flex.x86_64 m4.x86_64 openssl-devel.x86_64 unixODBC-devel.x86_64
-Install erlang like this:
+Install Erlang like this:
::
@@ -661,7 +629,7 @@
wget http://www.rabbitmq.com/releases/rabbitmq-server/v2.2.0/rabbitmq-server-2.2.0-1.noarch.rpm
rpm -Uhv --no-deps rabbitmq-server-2.2.0-1.noarch.rpm
-Finally edit the ``/etc/init.d/rabbitmq-server`` script so that it uses the correct path for your erlang install. Change this line
+Finally edit the ``/etc/init.d/rabbitmq-server`` script so that it uses the correct path for your Erlang install. Change this line
::
@@ -691,7 +659,7 @@
Working Directly with Carrot
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-As you learned earlier, CKAN uses carrot with the ``pyamqplib`` backend. Carrot is well documented at this URL:
+As you learned earlier, CKAN uses carrot with the ``pyamqplib`` backend. Carrot is well-documented at this URL:
http://ask.github.com/carrot/introduction.html
@@ -778,7 +746,7 @@
Working with CKANext Queue
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~
Rather than working with carrot publishers and consumers directly,
``ckanext-queue`` provides two useful Python objects to help you:
@@ -799,16 +767,16 @@
``ckanext.queue.worker.Worker``
This is a base class which you can inherit from. You can override its
- ``consume()`` method to asyncronously pull items from the queue to do useful
+ ``consume()`` method to asynchronously pull items from the queue to do useful
things
.. note ::
- To use the queue extension you don't need to implenent any new plugin
+ To use the queue extension you don't need to implement any new plugin
interfaces, you just need to use the ``get_publisher(config).send()`` method and the
``Worker`` class. Of course your own extension might use plugins to hook into
- other parts of CKAN to get information to put or retireve from the queue.
+ other parts of CKAN to get information to put or retrieve from the queue.
The worker implementation runs outside the CKAN server process, interacting
directly with both the AMQP queue and the CKAN API (to get CKAN data). The
@@ -862,7 +830,7 @@
You'll see that once again the consumer picks up the message.
-Writing a worker
+Writing a Worker
````````````````
Now let's replace the consumer with a worker.
@@ -1000,17 +968,3 @@
Now that you know about extensions, plugins and workers you should be able to
extend CKAN in lots of new and interesting ways.
-
-.. Links
-.. -----
-..
-.. Etherpad discussion: http://ckan.okfnpad.org/plugins
-..
-.. Existing plugin implementations (using the old API), comments from are pudo:
-..
-.. - Comments: http://bitbucket.org/pudo/ckandisqus
-.. - Weird stuff: http://bitbucket.org/pudo/ckanextdeliverance
-.. - Shouldn't be a plugin, but typical for localized versions: http://bitbucket.org/pudo/offenedaten
-.. - and probably the largest yet least plugin-ish: http://bitbucket.org/okfn/ckanextiati
-.. - this is what I want to avoid: http://bitbucket.org/okfn/ckanextiati/src/tip/ckanext/iati/authz.py
-
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/doc/post-installation.rst Thu Jul 28 17:20:40 2011 +0100
@@ -0,0 +1,63 @@
+========================
+Post-Installation Setup
+========================
+
+After you have completed installation (from either package or source), follow this section for instructions on setting up an initial user, loading test data, and notes on deploying CKAN.
+
+.. _create-admin-user:
+
+Create an Admin User
+====================
+
+By default, CKAN has a set of locked-down permissions. To begin
+working with it you need to set up a user and some permissions.
+
+First create an admin account from the command line (you must be root, ``sudo -s``):
+
+::
+
+ paster --plugin=ckan user add admin --config=/etc/ckan/std/std.ini
+
+When prompted, enter a password - this is the password you will use to log in to CKAN. In the resulting output, note that you will also get assigned a CKAN API key.
+
+.. note :: This command is your first introduction to some important CKAN concepts. **paster** is the script used to run CKAN commands. **std.ini** is the CKAN config file. You can change options in this file to configure CKAN.
+
+For exploratory purposes, you might was well make the ``admin`` user a
+sysadmin. You obviously wouldn't give most users these rights as they would
+then be able to do anything. You can make the ``admin`` user a sysadmin like
+this:
+
+::
+
+ paster --plugin=ckan sysadmin add admin --config=/etc/ckan/std/std.ini
+
+You can now login to the CKAN frontend with the username ``admin`` and the password you set up.
+
+.. _create-test-data:
+
+Load Test Data
+==============
+
+It can be handy to have some test data to start with. You can get test data like this:
+
+::
+
+ paster --plugin=ckan create-test-data --config=/etc/ckan/std/std.ini
+
+You now have a CKAN instance that you can log in to, with some test data to check everything
+works.
+
+.. _deployment-notes:
+
+Deployment
+==========
+
+You may want to deploy your CKAN instance at this point, to share with others.
+
+If you have installed CKAN from packages, then Apache and WSGI deployment scripts are already configured for you in standard locations.
+
+If you have installed CKAN from source, then the standard production deployment of CKAN is Apache and WSGI, which you will need to configure yourself. For more information, see http://wiki.ckan.net/Deployment
+
+CKAN has been successfully deployed by a variety of other methods including Apache reverse proxy + paster, nginx reverse proxy + paster, and nginx + uwsgi.
+
+You can now proceed to :doc:`theming`.
\ No newline at end of file
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/doc/prepare-extensions.rst Thu Jul 28 17:20:40 2011 +0100
@@ -0,0 +1,31 @@
+=========================
+Prepare to Use Extensions
+=========================
+
+If you are running a package installation of CKAN, before you start using and testing extensions (described in :doc:`extensions`) you need to prepare your system.
+
+Firstly, you'll need to set up and enter a virtual Python environment, as follows:
+
+::
+
+ sudo apt-get install virtualenv pip mercurial
+ virtualenv /home/ubuntu/pyenv
+ . /home/ubuntu/pyenv/bin/activate
+
+Then, you need to install the CKAN source into your virtual environment. You can install CKAN like this:
+
+::
+
+ pip install -e hg+http://bitbucket.org/okfn/ckan#egg=ckan
+
+Your new CKAN developer install will be running on http://localhost:5000/
+
+When you start using extensions, you should install any of the developer versions of the CKAN extensions you want to work on like this (using the appropriate URL):
+
+::
+
+ pip install -e hg+http://bitbucket.org/okfn/<dependency-name>@<version>#egg=<egg-name>
+
+The dependency you've installed will appear in ``/home/ubuntu/pyenv/src/`` where you can work on it.
+
+After working on extensions, you should make sure that your deployment passes the tests, as described in :doc:`test`.
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/doc/test.rst Thu Jul 28 17:20:40 2011 +0100
@@ -0,0 +1,118 @@
+======================
+Testing for Developers
+======================
+
+If you are installing CKAN from source, or developing extensions, then you need to know how to run CKAN tests.
+
+This section describes testing topics for developers, including basic tests, migration testing and testing against PostgreSQL.
+
+.. _basic-tests:
+
+Basic Tests
+-----------
+
+After completing your source installation of CKAN, you should check that tests pass. You should also check this before checking in changes to CKAN code.
+
+Make sure you've created a config file at ``pyenv/ckan/development.ini``. Then activate the Python environment::
+
+ . pyenv/bin/activate
+
+Install nose into your virtual environment::
+
+ pip install --ignore-installed nose
+
+At this point you will need to deactivate and then re-activate your
+virtual environment to ensure that all the scripts point to the correct
+locations:
+
+::
+
+ deactivate
+ . pyenv/bin/activate
+
+Then run the quick development tests::
+
+ cd pyenv/src/ckan
+ nosetests ckan/tests --ckan
+
+You *must* run the tests from the CKAN directory as shown above, otherwise the
+``--ckan`` plugin won't work correctly.
+
+.. warning ::
+
+ By default, the test run is 'quick and dirty' - only good enough as an initial check.
+
+
+Testing against PostgreSQL
+--------------------------
+
+The default way to run tests is defined in ``test.ini`` (which is the default config file for nose - change it with option ``--with-pylons``). This specifies using SQLite and sets ``faster_db_test_hacks``, which are compromises.
+
+::
+
+ cd pyenv/src/ckan
+ nosetests ckan/tests --ckan
+
+Although SQLite is useful for testing a large proportion of CKAN, actually in deployment, CKAN must run with PostgreSQL.
+
+Running the tests against PostgreSQL is slower but more thorough for two reasons:
+
+ 1. You test subtleties of PostgreSQL
+ 2. CKAN's default search relies on PostgreSQL's custom full-text search, so these (100 or so) tests are skipped when running against SQLite.
+
+So when making changes to anything involved with search or closely related to the database, it is wise to test against PostgreSQL.
+
+To test against PostgreSQL:
+
+ 1. Edit your local ``development.ini`` to specify a PostgreSQL database with the ``sqlalchemy.url`` parameter.
+ 2. Tell nose to use ``test-core.ini`` (which imports settings from ``development.ini``)
+
+::
+
+ nosetests ckan/tests --ckan --with-pylons=test-core.ini
+
+The test suite takes a long time to run against standard PostgreSQL (approx. 15 minutes, or close to an hour on Ubuntu/10.04 Lucid).
+
+This can be improved to between 5 and 15 minutes by running PostgreSQL in memory and turning off durability, as described `in the PostgreSQL documentation <http://www.postgresql.org/docs/9.0/static/non-durability.html>`_.
+
+.. _migrationtesting:
+
+Migration Testing
+-----------------
+
+If your changes require a model change, you'll need to write a migration script. To ensure this is tested as well, you should instead run the tests this way::
+
+ nosetests ckan/tests --ckan --ckan-migrate --with-pylons=test-core.ini
+
+By default, tests are run using the model defined in ``ckan/model``, but by using the ``--ckan-migrate`` option the tests will run using a database that has been created using the migration scripts, which is the way the database is created and upgraded in production. These tests are the most thorough and will take around 20 minutes.
+
+.. caution ::
+
+ Ordinarily, you should set ``development.ini`` to specify a PostgreSQL database
+ so these also get used when running ``test-core.ini``, since ``test-core.ini``
+ inherits from ``development.ini``. If you were to change the ``sqlalchemy.url``
+ option in your ``development.ini`` file to use SQLite, the command above would
+ actually test SQLite rather than PostgreSQL, so always check the setting in
+ ``development.ini`` to ensure you are running the full tests.
+
+.. warning ::
+
+ A common error when wanting to run tests against a particular database is to change ``sqlalchemy.url`` in ``test.ini`` or ``test-core.ini``. The problem is that these are versioned files and people have checked in these by mistake, creating problems for other developers and the CKAN buildbot. This is easily avoided by only changing ``sqlalchemy.url`` in your local ``development.ini`` and testing ``--with-pylons=test-core.ini``.
+
+Common problems running tests
+-----------------------------
+
+* `nose.config.ConfigError: Error reading config file 'setup.cfg': no such option 'with-pylons'`
+
+ This error can result when you run nosetests for two reasons:
+
+ 1. Pylons nose plugin failed to run. If this is the case, then within a couple of lines of running `nosetests` you'll see this warning: `Unable to load plugin pylons` followed by an error message. Fix the error here first.
+
+ 2. The Python module 'Pylons' is not installed into you Python environment. Confirm this with::
+
+ python -c "import pylons"
+
+* `OperationalError: (OperationalError) no such function: plainto_tsquery ...`
+
+ This error usually results from running a test which involves search functionality, which requires using a PostgreSQL database, but another (such as SQLite) is configured. The particular test is either missing a `@search_related` decorator or there is a mixup with the test configuration files leading to the wrong database being used.
+
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/doc/theming.rst Thu Jul 28 17:20:40 2011 +0100
@@ -0,0 +1,101 @@
+=============
+Customization
+=============
+
+After installing CKAN, the next step is probably to re-theme the site with your own logo, site name, and CSS.
+
+Site Name and Description
+=========================
+
+You can change the name and logo of the site by setting options in the CKAN config file.
+
+This is the file called ``std.ini`` that you first encountered in :ref:`create-admin-user`. It is usually located at ``/etc/ckan/std/std.ini``.
+
+Open this file, and change the following options::
+
+ ckan.site_title = My CKAN Site
+ ckan.site_description = The easy way to get, use and share data
+
+After you've edited these options, restart Apache::
+
+ sudo /etc/init.d/apache2 restart
+
+Refresh your home page (clearing the cache if necessary) and you should see your new title and description.
+
+More Advanced Customization
+===========================
+
+If you want to make broader changes to the look and feel of your CKAN site, we offer ways to add custom CSS and over-ride the default CKAN templates.
+
+Custom CSS and Templates
+------------------------
+
+You can add custom CSS, templates, scripts, images etc to your site using the ``extra_template_paths`` and ``extra_public_paths`` options in the CKAN config file::
+
+ extra_template_paths = %(here)s/my-templates
+ extra_public_paths = %(here)s/my-public
+
+All contents of the public directory is mounted directly into the URL space of the site (taking precedence over existing files of the same name).
+
+Furthermore, you can supply multiple public directories, which will be searched in order.
+
+For example, if you set the following option in the CKAN config file::
+
+ extra_public_paths = /path/to/mypublicdir
+
+And then add a file called ``myhtmlfile.html`` in ``/path/to/mypublicdir``, the file would appear on http://yourckan.org/ at ``http://yourckan.org/myhtmlfile.html``.
+
+If you create a file with the same path as one in the main CKAN public directory, your file will override the default CKAN file. For example, if you add ``mypublicdir/css/ckan.css``, then ``http://yourckan.org/css/ckan.css`` will be your file.
+
+Adding a New Logo
+^^^^^^^^^^^^^^^^^
+
+One example is introducing your own logo, which you can do with a new file and a CKAN config option.
+
+Add a logo file at ``mypublicdir/images/mylogo.png``, then set options in the CKAN config file (``/etc/ckan/std/std.ini``) as follows::
+
+ extra_public_paths = /path/to/mypublicdir
+ ckan.site_logo = /images/mylogo.png
+
+
+Adding a New Stylesheet
+^^^^^^^^^^^^^^^^^^^^^^^
+
+Lots of visual changes can be made simply by changing the stylesheet.
+
+The easiest way to override the default CKAN style is to create one or more custom CSS files and load them in the ``layout.html`` template.
+
+Use the 'public' directory as described in the previous section, then add a new file at ``mypublicdir/css/mycss.css``.
+
+Next, copy the ``layout.html`` template and add a reference to the new CSS file. Here is an example of the edited ``layout.html`` template::
+
+ <html xmlns="http://www.w3.org/1999/xhtml"
+ xmlns:i18n="http://genshi.edgewall.org/i18n"
+ xmlns:py="http://genshi.edgewall.org/"
+ xmlns:xi="http://www.w3.org/2001/XInclude"
+ py:strip="">
+ <py:def function="optional_head">
+ <link rel="stylesheet" href="${g.site_url}/css/mycss.css" />
+ </py:def>
+ <xi:include href="layout_base.html" />
+ </html>
+
+Retheming the Site with Templates
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Template files are used as source templates for rendered pages on the site. These templates are just an HTML page but with variables, such as the page title set by each page: ``${page_title}``.
+
+To over-ride a template, set the ``extra_template_paths`` directory as described above, then copy and rewrite the template file you wish to over-ride.
+
+Commonly modified templates are:
+
+ * ``layout_base.html`` - base customizationlayout template for whole site
+ * ``layout.html`` - empty by default
+ * ``home/index.html`` - the home page of the site
+ * ``home/about.html`` - the about page
+
+If you are re-theming the site, we recommend you over-ride ``layout.html``, which is empty but inherits from ``layout_base.html``. This will mean you can upgrade the site more easily in the future.
+
+.. note::
+
+ For more information on the syntax of the CKAN templates, refer to the `Genshi documentation <http://genshi.edgewall.org/wiki/Documentation>`_.
\ No newline at end of file
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/doc/upgrade.rst Thu Jul 28 17:20:40 2011 +0100
@@ -0,0 +1,25 @@
+============
+Upgrade CKAN
+============
+
+Any time you want to upgrade CKAN to a newer version, first back up your database:
+
+::
+
+ paster --plugin=ckan db dump demo_ckan_backup.pg_dump --config=demo.ckan.net.ini
+
+Then run this as root:
+
+::
+
+ apt-get update
+ apt-get dist-upgrade
+ ckan-std-install
+
+You need to use ``dist-upgrade`` rather than ``upgrade`` because it is possible
+more recent versions of CKAN will add new dependencies rather than just
+updating existing ones, and apt will only install new dependencies if you use
+``dist-upgrade``.
+
+The ``ckan-std-install`` will also upgrade your existing CKAN
+instance, keeping a backup of your old CKAN config file.
--- a/doc/vm.rst Thu Jul 28 16:26:50 2011 +0100
+++ /dev/null Thu Jan 01 00:00:00 1970 +0000
@@ -1,247 +0,0 @@
-Testing CKAN in a VM
-++++++++++++++++++++
-
-.. WARNING::
- This document is still under development, use only if you are a member
- of the CKAN team who wishes to be an early adopter and are interested in
- experimenting with virtual machines.
-
-If you aren't running Lucid, you may need to test in a VM.
-
-Repository cache
-================
-
-First set up a cache of the repositories so that you don't need to fetch packages each time you
-build a VM:
-
-::
-
- $ sudo apt-get install apt-proxy
- $ sudo apt-get install uml-utilities
-
-Now find your local host's ethernet port name, IP address, netmask::
-
- $ ifconfig
- eth0 Link encap:Ethernet HWaddr 1c:6f:65:8a:0a:d8
- inet addr:50.56.101.208 Bcast:50.56.101.255 Mask:255.255.255.0
-
-i.e.::
-
- ethernet port name: eth0
- IP address: 50.56.101.208
- netmask: 255.255.255.0
-
-These instructions assume these values, so replace them with yours.
-
-Now edit your apt-get config::
-
- $ sudo vim /etc/apt-proxy/apt-proxy.conf
-
-And set these two lines. The first is your host's IP address::
-
- ;; Server IP to listen on
- address = 50.56.101.208
-
- ;; Server port to listen on
- port = 9998
-
-Now restart it::
-
- $ sudo /etc/init.d/apt-proxy restart
-
-Once this is complete, your (empty) proxy is ready for use on
-http://<host-machine-ip>:9998 and will find Ubuntu repository under ``/ubuntu``.
-
-See also:
-
-* https://help.ubuntu.com/community/AptProxy
-
-
-VM creation
-===========
-
-Install the VM program::
-
- sudo apt-get install python-vm-builder
- sudo apt-get install kvm-pxe
-
-Now create a directory ``~/Vms`` for your virtual machines.
-
-::
-
- mkdir ~/Vms
-
-
-We'll use manual bridging and networking rather than relying on the magic provided by ``libvirt``. Out virtual network for the VMs will be 192.168.100.xxx. You can use any number from 2-253 inclusive for the last bit of the IP. This first machine will have the IP address 192.168.100.2. Each virtual machine afterwards must have a unique IP address.
-
-First set some variables (use your own IP address for HOST_IP):
-
-::
-
- export THIS_IP="4"
- export HOST_IP="50.56.101.208"
-
-Check your CPU has native VM support::
-
- $ kvm-ok
-
-If your CPU doesn't support KVM extensions:
-
-1. change ``kvm`` to ``qemu`` in the vmbuilder step coming up next
-2. use ``/usr/bin/qemu`` instead of ``/usr/bin/kvm``
-
-Now create the VM:
-
-::
-
- cd ${HOME}/Vms
- export BASE_IP="192.168.100"
- sudo vmbuilder kvm ubuntu \
- --mem 512 \
- --cpus 6 \
- --domain ckan_${THIS_IP} \
- --dest ckan_${THIS_IP} \
- --flavour virtual \
- --suite lucid \
- --arch amd64 \
- --hostname ckan${THIS_IP} \
- --user ubuntu \
- --pass ubuntu \
- --rootpass ubuntu \
- --debug -v \
- --ip ${BASE_IP}.${THIS_IP} \
- --mask 255.255.255.0 \
- --net ${BASE_IP}.0 \
- --bcast ${BASE_IP}.255 \
- --gw ${BASE_IP}.254 \
- --dns ${BASE_IP}.254 \
- --proxy http://${HOST_IP}:9998/ubuntu \
- --components main,universe \
- --addpkg vim \
- --addpkg openssh-server \
- --addpkg wget
-
-This assumes you already have an apt mirror set up on port 9998 as described
-above and that you are putting everything in ``~/Vms``.
-
-Now for the networking.
-
-First check you have forwarding enabled on the host:
-
-::
-
- sudo -s
- echo "1" > /proc/sys/net/ipv4/ip_forward
- exit
-
-Now save this as ``~/Vms/start.sh``:
-
-::
-
- #!/bin/bash
-
- if [ "X$1" = "X" ] || [ "X$2" = "X" ] || [ "X$3" = "X" ] || [ "X$4" = "X" ] || [ "X$5" = "X" ]; then
- echo "ERROR: call this script with network device name, tunnel name, amount of memory, number of CPUs and path to the image e.g."
- echo " $0 eth0 qtap0 512M 4 /home/Vms/ckan_2/tmpKfAdeU.qcow2 [extra args to KVM]"
- exit 1
- fi
-
- NETWORK_DEVICE=$1
- TUNNEL=$2
- MEM=$3
- CPUS=$4
- IMAGE=$5
- EXTRA=$6
- MACADDR="52:54:$(dd if=/dev/urandom count=1 2>/dev/null | md5sum | sed 's/^\(..\)\(..\)\(..\)\(..\).*$/\1:\2:\3:\4/')";
-
- echo "Creating bridge..."
- sudo iptables -t nat -A POSTROUTING -o ${NETWORK_DEVICE} -j MASQUERADE
- sudo brctl addbr br0
- sudo ifconfig br0 192.168.100.254 netmask 255.255.255.0 up
- echo "done."
- echo "Creating tunnel..."
- sudo modprobe tun
- sudo tunctl -b -u root -t ${TUNNEL}
- sudo brctl addif br0 ${TUNNEL}
- sudo ifconfig ${TUNNEL} up 0.0.0.0 promisc
- echo "done."
- echo "Starting VM ${IMAGE} on ${TUNNEL} via ${NETWORK_DEVICE} with MAC ${MACADDR}..."
- sudo /usr/bin/kvm -M pc-0.12 -enable-kvm -m ${MEM} -smp ${CPUS} -name dev -monitor pty -boot c -drive file=${IMAGE},if=ide,index=0,boot=on -net nic,macaddr=${MACADDR} -net tap,ifname=${TUNNEL},script=no,downscript=no -serial none -parallel none -usb ${EXTRA}
-
-
-Make it executable:
-
-::
-
- chmod a+x ~/Vms/start.sh
-
-Now you can start it along the lines of this:
-
-::
-
- ./start.sh eth0 qtap0 512M 1 /home/james/Vms/ckan_4/tmpuNIv2h.qcow2
-
-Now login:
-
-::
-
- ssh ubuntu@${BASE_IP}.${THIS_IP}
-
-Once in you'll need some more configuration.
-
-Edit ``/etc/resolv.conf`` to contain just this (the Google DNS servers, handy
-to used a fixed IP so that you don't have to update your ``resolve.conf`` each
-time you move to a different network):
-
-::
-
- nameserver 8.8.8.8
-
-Then change ``/etc/apt/apt.conf`` to comment out the proxy line, you may as
-well get updates directly now.
-
-Finally, run this (swapping the repository name for the one you want to test)
-to allow yourself to install CKAN:
-
-::
-
- sudo apt-get install wget
- wget -qO- http://apt-alpha.ckan.org/packages.okfn.key | sudo apt-key add -
- echo "deb http://apt-alpha.ckan.org/debian lucid universe" | sudo tee /etc/apt/sources.list.d/okfn.list
- sudo apt-get update
-
-Now that you have the repo added you can install and test CKAN as normal.
-
-Here's how mine look:
-
-::
-
- ubuntu at ckan4:~$ cat /etc/network/interfaces
- # This file describes the network interfaces available on your system
- # and how to activate them. For more information, see interfaces(5).
-
- # The loopback network interface
- auto lo
- iface lo inet loopback
-
- # The primary network interface
- auto eth0
- iface eth0 inet static
- address 192.168.100.4
- netmask 255.255.255.0
- network 192.168.100.0
- broadcast 192.168.100.255
- gateway 192.168.100.254
- dns 192.168.100.254
- ubuntu at ckan4:~$ cat /etc/resolv.conf
- nameserver 8.8.8.8
-
-
-VNC access
-==========
-
-To add ability to VNC into the VM as it runs (useful for debug), add this line to the start.sh command::
-
- -vnc :1
-
-Then you can vnc to it when running using a VNC Client.
\ No newline at end of file
Repository URL: https://bitbucket.org/okfn/ckan/
--
This is a commit notification from bitbucket.org. You are receiving
this because you have the service enabled, addressing the recipient of
this email.
More information about the ckan-changes
mailing list