[ckan-changes] commit/ckan: 2 new changesets
Bitbucket
commits-noreply at bitbucket.org
Thu May 19 16:48:17 UTC 2011
2 new changesets in ckan:
http://bitbucket.org/okfn/ckan/changeset/44aea75cf1a9/
changeset: r3095:44aea75cf1a9
user: dread
date: 2011-05-19 18:44:36
summary: [tests][xs]: Extra test added, following query about searching groups.
affected #: 1 file (223 bytes)
--- a/ckan/tests/functional/api/test_package_search.py Thu May 19 16:00:17 2011 +0100
+++ b/ckan/tests/functional/api/test_package_search.py Thu May 19 17:44:36 2011 +0100
@@ -302,6 +302,12 @@
res_dict = self.data_from_res(res)
assert_equal(res_dict['count'], 3)
+ def test_13_just_groups(self):
+ offset = self.base_url + '?groups=roger'
+ res = self.app.get(offset, status=200)
+ res_dict = self.data_from_res(res)
+ assert res_dict['count'] == 1, res_dict
+
def test_strftimestamp(self):
import datetime
t = datetime.datetime(2012, 3, 4, 5, 6, 7, 890123)
http://bitbucket.org/okfn/ckan/changeset/3ba524584591/
changeset: r3096:3ba524584591
user: dread
date: 2011-05-19 18:45:47
summary: [doc][s]: Improvements to docs. Added info on paster and buildbot setup. Improvements to vm and deb docs. Added section to authz about Publisher Mode. Minor edits elsewhere on the way through.
affected #: 9 files (9.7 KB)
--- a/README.txt Thu May 19 17:44:36 2011 +0100
+++ b/README.txt Thu May 19 17:45:47 2011 +0100
@@ -28,6 +28,7 @@
python-dev Python interpreter v2.5 - v2.7 and dev headers
postgresql PostgreSQL database
libpq-dev PostgreSQL library
+ python-psycopg2 PostgreSQL python module
libxml2-dev XML library development files
libxslt-dev XSLT library development files
python-virtualenv Python virtual environments
@@ -37,7 +38,7 @@
subversion Subversion source control (for pyutilib)
===================== ===============================================
- For ubuntu you can install these like so:
+ For Ubuntu you can install these like so:
::
--- a/doc/admin.rst Thu May 19 17:44:36 2011 +0100
+++ /dev/null Thu Jan 01 00:00:00 1970 +0000
@@ -1,20 +0,0 @@
-Administrators and User Rights
-==============================
-
-CKAN allows for certain users to be given administrative rights, i.e. to perform all actions irrespective of their object-specific permissions.
-
-
-Creating and Removing Admins via the Command Line
--------------------------------------------------
-
-If you have CKAN installed, a ``paster`` command line utility will allow you to create new administrators or to remove administrative privileges from existing users. To add an administrator, run::
-
- paster --plugin=ckan sysadmin -c my.ckan.net.ini create USERNAME
-
-Where ``my.ckan.net.ini`` is the name of the configuration file you wish to use. If you've checked out the source code and are developing, you can also execute the same command in the base directory, dropping the ``--plugin`` argument.
-
-Removing administrative access from users works the same::
-
- paster --plugin=ckan sysadmin -c my.ckan.net.ini remove USERNAME
-
-
--- a/doc/authentication.rst Thu May 19 17:44:36 2011 +0100
+++ b/doc/authentication.rst Thu May 19 17:45:47 2011 +0100
@@ -1,7 +1,7 @@
CKAN Authentication and Identification
======================================
-CKAN operates a delegated authenatication model based on openid. Integration of
+CKAN operates a delegated authentication model based on openid. Integration of
authentication into CKAN is provided by repoze.who and the repoze.who.openid
plugin.
--- a/doc/authorization.rst Thu May 19 17:44:36 2011 +0100
+++ b/doc/authorization.rst Thu May 19 17:45:47 2011 +0100
@@ -4,7 +4,7 @@
This document gives an overview of CKAN's authorization capabilities and model
in relation to access control. The authentication/identification aspects of
-access control are dealt with separately in :doc:`authorization`.
+access control are dealt with separately in :doc:`authentication`.
Overview
@@ -45,8 +45,7 @@
Command-line authorization management
-------------------------------------
-To allow for fine-grained control of the authorization system, CKAN has
-several command related to the management of roles and access permissions.
+Although the Admin Extension provides a Web interface for managing authorization, there is a set of more powerful :doc:`paster` commands for fine-grained control.
The ``roles`` command will list and modify the assignment of actions to
roles::
@@ -128,6 +127,34 @@
* A user given the admin right for the System object is a 'System Admin' and can do any action on any object. (A shortcut for creating a System Admin is by using the ``paster sysadmin`` command.)
* A user given the admin right for a particular object can do any action to that object.
+Publisher mode setup
+--------------------
+
+Although ckan.net is forging ahead with the Wikipedia model of allowing anyone to add and improve metadata, some CKAN instances prefer to operate in 'Publisher mode' which allows edits only from authorized users.
+
+To operate in this mode:
+
+ 1. Remove the rights for general public to edit existing packages and create new ones.::
+
+ paster rights remove visitor anon_editor package:all
+ paster rights remove logged_in editor package:all
+ paster rights remove visitor anon_editor system
+ paster rights remove logged_in editor system
+
+ 2. If logged-in users have already created packages in your system then you may also wish to remove admin rights. e.g.::
+
+ paster rights remove bob admin package:all
+
+ 3. Change the default rights for newly created packages. Do this by using these values in your config (.ini file)::
+
+ ckan.default_roles.Package = {"visitor": ["reader"], "logged_in": ["reader"]}
+ ckan.default_roles.Group = {"visitor": ["reader"], "logged_in": ["reader"]}
+ ckan.default_roles.System = {"visitor": ["reader"], "logged_in": ["reader"]}
+ ckan.default_roles.AuthorizationGroup = {"visitor": ["reader"], "logged_in": ["reader"]}
+
+ Note there is also the possibility to restrict package edits by a user's authorization group. See http://wiki.ckan.net/Publisher_Mode
+
+
Examples
--------
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/doc/buildbot.rst Thu May 19 17:45:47 2011 +0100
@@ -0,0 +1,153 @@
+Setting up buildbot
++++++++++++++++++++
+
+These directions provide some rough information for setting up a build bot on a Lucid machine.
+
+
+Apt Installs
+============
+
+Install CKAN core dependencies from Lucid distribution (as per :doc:`README`)::
+
+ sudo apt-get install build-essential libxml2-dev libxslt-dev
+ sudo apt-get install wget mercurial postgresql libpq-dev git-core
+ sudo apt-get install python-dev python-psycopg2 python-virtualenv
+ sudo apt-get install subversion
+
+Maybe need this too::
+
+ sudo apt-get install python-include
+
+Buildbot software::
+
+ sudo apt-get install buildbot
+
+Deb building software (as per :doc:`deb`)::
+
+ sudo apt-get install -y dh-make devscripts fakeroot cdbs
+
+Fabric::
+
+ sudo apt-get install -y fabric
+
+If you get errors with postgres and locales you might need to do these::
+
+ sudo apt-get install language-pack-en-base
+ sudo dpkg-reconfigure locales
+
+
+Postgres setup
+==============
+
+If installation before failed to create a cluster, do this after fixing errors::
+
+ sudo pg_createcluster 8.4 main --start
+
+Create users and databases::
+
+ sudo -u postgres createuser -S -D -R -P buildslave
+ # set this password (matches buildbot scripts): biomaik15
+ sudo -u postgres createdb -O buildslave ckan1
+ sudo -u postgres createdb -O buildslave ckanext
+
+
+Buildslave setup
+================
+
+Rough commands::
+
+ sudo useradd -m -s /bin/bash buildslave
+ sudo chown buildslave:buildslave /home/buildslave
+ sudo su buildslave
+ cd ~
+ hg clone https://dread@bitbucket.org/okfn/buildbot-scripts .
+ ssh-keygen -t rsa
+ cp /home/buildslave/.ssh/id_rsa.pub ~/.ssh/authorized_keys
+ mkdir -p ckan/build
+ cd ckan/build
+ python ~/ckan-default.py
+ buildbot create-slave ~ localhost:9989 okfn <buildbot_password>
+ vim ~/info/admin
+ vim ~/info/host
+ mkdir /home/buildslave/pip_cache
+ virtualenv pyenv-tools
+ pip -E pyenv-tools install buildkit
+
+
+Buildmaster setup
+=================
+
+Rough commands::
+
+ mkdir ~/buildmaster
+ buildbot create-master ~/buildmaster
+ ln -s /home/buildslave/master/master.cfg ~/buildmaster/master.cfg
+ cd ~/buildmaster
+ buildbot checkconfig
+
+
+Startup
+=======
+
+Setup the daemons for master and slave::
+
+ sudo vim /etc/default/buildbot
+
+This file should be edited to be like this::
+
+ BB_NUMBER[0]=0 # index for the other values; negative disables the bot
+ BB_NAME[0]="okfn" # short name printed on startup / stop
+ BB_USER[0]="okfn" # user to run as
+ BB_BASEDIR[0]="/home/okfn/buildmaster" # basedir argument to buildbot (absolute path)
+ BB_OPTIONS[0]="" # buildbot options
+ BB_PREFIXCMD[0]="" # prefix command, i.e. nice, linux32, dchroot
+
+ BB_NUMBER[1]=1 # index for the other values; negative disables the bot
+ BB_NAME[1]="okfn" # short name printed on startup / stop
+ BB_USER[1]="buildslave" # user to run as
+ BB_BASEDIR[1]="/home/buildslave" # basedir argument to buildbot (absolute path)
+ BB_OPTIONS[1]="" # buildbot options
+ BB_PREFIXCMD[1]="" # prefix command, i.e. nice, linux32, dchroot
+
+Start master and slave (according to /etc/default/buildbot)::
+
+ sudo /etc/init.d/buildbot start
+
+Now check you can view buildbot at: http://localhost:8010/
+
+
+Connect ports
+=============
+
+It's preferable to view the buildbot site at port 80 rather than 8010.
+
+If there is no other web service on this machine, you might connect up the addresses using iptables::
+
+ sudo iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 8010
+
+Otherwise it is best to do a reverse proxy. Using apache, edit this file::
+
+ sudo vim /etc/apache2/sites-available/buildbot.okfn.org
+
+to be like this::
+
+ <VirtualHost *:80>
+ ServerName buildbot.okfn.org
+
+ ProxyPassReverse ts Off
+ <Proxy *>
+ Order deny,allow
+ Allow from all
+ </Proxy>
+ ProxyPass / http://127.0.0.1:8010/
+ ProxyPassReverse / http://127.0.0.1:8010/
+ ProxyPreserveHost On
+ </VirtualHost>
+
+Then::
+
+ sudo apt-get install libapache2-mod-proxy-html
+ sudo a2enmod proxy_http
+ sudo a2ensite buildbot.okfn.org
+ sudo /etc/init.d/apache2 reload
+
--- a/doc/deb.rst Thu May 19 17:44:36 2011 +0100
+++ b/doc/deb.rst Thu May 19 17:45:47 2011 +0100
@@ -1,9 +1,10 @@
CKAN's Approach to Dependencies
+++++++++++++++++++++++++++++++
-WARNING: This document is still under development, use only if you are a member
-of the CKAN team who wishes to be an early adopter and are interested in
-experimenting with Debian packaging.
+.. WARNING::
+ This document is still under development, use only if you are a member
+ of the CKAN team who wishes to be an early adopter and are interested in
+ experimenting with virtual machines.
.. contents ::
@@ -216,6 +217,12 @@
actually for a CKAN Python extension called ``python-ckanext-qa`` but the same
process applies).
+hg clone ckan-deps
+
+::
+
+ python -m buildkit.deb /path/to/ckan-deps/.. python-ckan-deps 1.3~01+lucid http://ckan.org
+
Semi-Manual Python Packaging
============================
@@ -325,7 +332,7 @@
The base naming conventions we use for packages are as follows:
``ckan``
- Unstalls CKAN, PostgreSQL, Apache etc. It adds the ``ckan-instance-create`` command which is then the only thing you need to create a new instance.
+ Uninstalls CKAN, PostgreSQL, Apache etc. It adds the ``ckan-instance-create`` command which is then the only thing you need to create a new instance.
``python-ckan``
The CKAN Python library packaged from code at http://bitbucket.org/okfn/ckan
@@ -392,7 +399,7 @@
the web user interface)
In the simple cases, these scripts can then be used in your client application
-CKAN extension's ``posinst`` script to set up the custom instance. In more
+CKAN extension's ``postinst`` script to set up the custom instance. In more
complex cases you may write a ``postinst`` script from scratch. The
``postinst`` script then forms part of the package and is run by the apt system
as part of the package installation or upgrade process to configure your CKAN
@@ -413,8 +420,8 @@
and how to publish your packages to it, let's understand what a user of your
package will do to install it.
-Understaning How a User Installs from an apt repository
-=======================================================
+Understanding How a User Installs from an apt repository
+========================================================
A user will follow the following process:
@@ -605,7 +612,7 @@
python -m buildkit.deb . ckanext-inspire 0.1~03 http://ckan.org python-ckan
python -m buildkit.deb . ckanext-spatial 0.1~04 http://ckan.org python-ckan
- python -m buildkit.deb . ckanext-harvest 0.1~15 htthp://ckan.org python-ckan python-ckanext-spatial python-carrot
+ python -m buildkit.deb . ckanext-harvest 0.1~15 http://ckan.org python-ckan python-ckanext-spatial python-carrot
python -m buildkit.deb . ckanext-csw 0.3~10 http://ckan.org python-ckanext-harvest python-owslib python-ckan
python -m buildkit.deb . ckanext-dgu 0.2~11 http://ckan.org python-ckan python-ckanext-importlib python-ckanext-dgu python-ckanext-csw python-ckan python-ckanext-spatial python-ckanext-inspire
python -m buildkit.deb . ckanext-qa 0.1~19 http://ckan.org python-ckan
--- a/doc/index.rst Thu May 19 17:44:36 2011 +0100
+++ b/doc/index.rst Thu May 19 17:45:47 2011 +0100
@@ -14,6 +14,7 @@
design
deployment
configuration
+ paster
api
plugins
feeds
@@ -29,6 +30,8 @@
database_dumps
admin
deb
+ vm
+ buildbot
Other material
==============
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/doc/paster.rst Thu May 19 17:45:47 2011 +0100
@@ -0,0 +1,74 @@
+Paster commands
++++++++++++++++
+
+Paster commands provide a number of useful ways to administer a CKAN installation. These are run on the command line on the server running CKAN.
+
+Commands
+========
+
+ ================= ==========================================================
+ changes Distribute changes
+ create-test-data Create test data in the database.
+ db Perform various tasks on the database.
+ notify Send out modification notifications.
+ ratings Manage the ratings stored in the db
+ rights Commands relating to per-object and system-wide access rights.
+ roles Commands relating to roles and actions.
+ search-index Creates a search index for all packages
+ sysadmin Gives sysadmin rights to a named user
+ user Manage users
+ ================= ==========================================================
+
+
+Running paster commands
+=======================
+
+Basically the format is::
+
+ paster --plugin=ckan <ckan commands> --config=<config file>
+
+e.g.::
+
+ paster --plugin=ckan db init --config=myckan.ini
+
+
+To get a list of paster commands (i.e. including CKAN commands)::
+
+ paster --plugin=ckan --help
+
+And you can get help for each command. e.g.::
+
+ paster --plugin=ckan --help db
+
+
+``--plugin`` parameter
+--------------------
+
+Paster is a system-wide command, so you need to tell it to find the commands defined in the ckan code. So first parameter to paster is normally ``--plugin=ckan``.
+
+NB: If your current directory is where CKAN's setup.py is, you don't need to specify this parameter.
+
+If you want to run a "paster shell" then the plugin is pylons. e.g. ``paster --plugin=pylons shell``. Often you will want to run this as the same user as the web application, to ensure log files are written as the same user. And you'll also want to specify a config file. e.g.::
+
+ sudo -u www-data paster --plugin=pylons shell myckan.ini
+
+
+``--config`` parameter
+----------------------
+
+Each server can have multiple CKAN instances running, so use this parameter to specify the CKAN config file for the instance you want to operate on. e.g. ``--config=myckan.ini``
+
+NB: Developers tend to use development.ini, and this is the default value (looking in the current directory), so in this situation you don't need to specify this parameter.
+
+
+Position of paster parameters
+-----------------------------
+
+``--plugin`` is a paster parameter, so needs to come before the CKAN command, but <code>--config</code> is a ckan parameter so needs to come after the CKAN command.
+
+Here are three ways to express the same thing::
+
+ paster db init
+ paster --plugin=ckan db --config=development.ini init
+ paster --plugin=ckan db init --config=development.ini
+
--- a/doc/vm.rst Thu May 19 17:44:36 2011 +0100
+++ b/doc/vm.rst Thu May 19 17:45:47 2011 +0100
@@ -1,25 +1,70 @@
Testing CKAN in a VM
++++++++++++++++++++
-WARNING: This document is still under development, use only if you are a member
-of the CKAN team who wishes to be an early adopter and are interested in
-experimenting with virtual machines.
+.. WARNING::
+ This document is still under development, use only if you are a member
+ of the CKAN team who wishes to be an early adopter and are interested in
+ experimenting with virtual machines.
-If you aren't running Lucid, you may need to test in a VM. First set up a cache
-of the repositories so that you don't need to fetch packages each time you
+If you aren't running Lucid, you may need to test in a VM.
+
+Repository cache
+================
+
+First set up a cache of the repositories so that you don't need to fetch packages each time you
build a VM:
::
- sudo apt-get install apt-proxy
+ $ sudo apt-get install apt-proxy
+ $ sudo apt-get install uml-utilities
+
+Now find your local host's ethernet port name, IP address, netmask::
+
+ $ ifconfig
+ eth0 Link encap:Ethernet HWaddr 1c:6f:65:8a:0a:d8
+ inet addr:50.56.101.208 Bcast:50.56.101.255 Mask:255.255.255.0
+
+i.e.::
+
+ ethernet port name: eth0
+ IP address: 50.56.101.208
+ netmask: 255.255.255.0
+
+These instructions assume these values, so replace them with yours.
+
+Now edit your apt-get config::
+
+ $ sudo vim /etc/apt-proxy/apt-proxy.conf
+
+And set these two lines. The first is your host's IP address::
+
+ ;; Server IP to listen on
+ address = 50.56.101.208
+
+ ;; Server port to listen on
+ port = 9998
+
+Now restart it::
+
+ $ sudo /etc/init.d/apt-proxy restart
Once this is complete, your (empty) proxy is ready for use on
-http://mirroraddress:9999 and will find Ubuntu repository under ``/ubuntu``.
+http://<host-machine-ip>:9998 and will find Ubuntu repository under ``/ubuntu``.
See also:
* https://help.ubuntu.com/community/AptProxy
+
+VM creation
+===========
+
+Install the VM program::
+
+ sudo apt-get install python-vm-builder
+ sudo apt-get install kvm-pxe
+
Now create a directory ``~/Vms`` for your virtual machines.
::
@@ -29,14 +74,21 @@
We'll use manual bridging and networking rather than relying on the magic provided by ``libvirt``. Out virtual network for the VMs will be 192.168.100.xxx. You can use any number from 2-253 inclusive for the last bit of the IP. This first machine will have the IP address 192.168.100.2. Each virtual machine afterwards must have a unique IP address.
-First set some variables:
+First set some variables (use your own IP address for HOST_IP):
::
export THIS_IP="4"
- export HOST_IP="192.168.0.2"
+ export HOST_IP="50.56.101.208"
-You can get the host IP by looking at the output from ``ifconifg``.
+Check your CPU has native VM support::
+
+ $ kvm-ok
+
+If your CPU doesn't support KVM extensions:
+
+1. change ``kvm`` to ``qemu`` in the vmbuilder step coming up next
+2. use ``/usr/bin/qemu`` instead of ``/usr/bin/kvm``
Now create the VM:
@@ -46,7 +98,7 @@
export BASE_IP="192.168.100"
sudo vmbuilder kvm ubuntu \
--mem 512 \
- --cpus 4 \
+ --cpus 6 \
--domain ckan_${THIS_IP} \
--dest ckan_${THIS_IP} \
--flavour virtual \
@@ -63,13 +115,13 @@
--bcast ${BASE_IP}.255 \
--gw ${BASE_IP}.254 \
--dns ${BASE_IP}.254 \
- --proxy http://${HOST_IP}:9999/ubuntu \
+ --proxy http://${HOST_IP}:9998/ubuntu \
--components main,universe \
--addpkg vim \
--addpkg openssh-server \
--addpkg wget
-This assumes you already have an apt mirror set up on port 9999 as described
+This assumes you already have an apt mirror set up on port 9998 as described
above and that you are putting everything in ``~/Vms``.
Now for the networking.
@@ -116,17 +168,18 @@
echo "Starting VM ${IMAGE} on ${TUNNEL} via ${NETWORK_DEVICE} with MAC ${MACADDR}..."
sudo /usr/bin/kvm -M pc-0.12 -enable-kvm -m ${MEM} -smp ${CPUS} -name dev -monitor pty -boot c -drive file=${IMAGE},if=ide,index=0,boot=on -net nic,macaddr=${MACADDR} -net tap,ifname=${TUNNEL},script=no,downscript=no -serial none -parallel none -usb ${EXTRA}
+
Make it executable:
::
chmod a+x ~/Vms/start.sh
-Now you can start it:
+Now you can start it along the lines of this:
::
- ./start.sh eth1 qtap0 512M 1 /home/james/Vms/ckan_3/tmpuNIv2h.qcow2
+ ./start.sh eth0 qtap0 512M 1 /home/james/Vms/ckan_4/tmpuNIv2h.qcow2
Now login:
@@ -184,3 +237,11 @@
nameserver 8.8.8.8
+VNC access
+==========
+
+To add ability to VNC into the VM as it runs (useful for debug), add this line to the start.sh command::
+
+ -vnc :1
+
+Then you can vnc to it when running using a VNC Client.
\ No newline at end of file
Repository URL: https://bitbucket.org/okfn/ckan/
--
This is a commit notification from bitbucket.org. You are receiving
this because you have the service enabled, addressing the recipient of
this email.
More information about the ckan-changes
mailing list