[ckan-changes] commit/ckanext-spatial: 4 new changesets
Bitbucket
commits-noreply at bitbucket.org
Thu Sep 29 14:51:41 UTC 2011
4 new changesets in ckanext-spatial:
http://bitbucket.org/okfn/ckanext-spatial/changeset/f01eea986681/
changeset: f01eea986681
user: amercader
date: 2011-09-26 11:26:04
summary: Added tag release-v0.1 for changeset 90466a83f22e
affected #: 1 file (-1 bytes)
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/.hgtags Mon Sep 26 10:26:04 2011 +0100
@@ -0,0 +1,1 @@
+90466a83f22e7b995c98e14e6481a4987b63f882 release-v0.1
http://bitbucket.org/okfn/ckanext-spatial/changeset/54aa0b83fbd3/
changeset: 54aa0b83fbd3
branch: feature-1272-store-extents-from-form
user: amercader
date: 2011-09-26 11:27:20
summary: Tag new version
affected #: 1 file (-1 bytes)
--- a/setup.py Mon Sep 26 10:26:04 2011 +0100
+++ b/setup.py Mon Sep 26 10:27:20 2011 +0100
@@ -1,7 +1,7 @@
from setuptools import setup, find_packages
import sys, os
-version = '0.1'
+version = '0.2'
setup(
name='ckanext-spatial',
http://bitbucket.org/okfn/ckanext-spatial/changeset/b7a5d6c2663e/
changeset: b7a5d6c2663e
branch: feature-1272-store-extents-from-form
user: amercader
date: 2011-09-29 16:40:49
summary: Support for automatic indexing of spatial extent (#1272)
The spatial_query plugin now automatically checks for the existance of an
'spatial' extra when creating, updating or deleting a package and syncs its
value (in GeoJSON format) with the geometry column of the package_extent
table. The extension now uses GeoAlchemy, that makes performing spatial
operation extremely easy. The geometry field has been updated to support
multiple geometry types (Polygons, Points, MultiPolygons...) and the default
srid has been set as WGS 84 lat/lon (EPSG:4326).
affected #: 6 files (-1 bytes)
--- a/README.rst Mon Sep 26 10:27:20 2011 +0100
+++ b/README.rst Thu Sep 29 15:40:49 2011 +0100
@@ -12,7 +12,13 @@
You will need CKAN installed. The present module should be installed at least
with `setup.py develop` if not installed in the normal way with
`setup.py install` or using pip or easy_install.
-
+
+The extension uses the GeoAlchemy_ and Shapely_ libraries. You can install them
+via `pip install -r pip-requirements.txt` from the extension directory.
+
+.. _GeoAlchemy: http://www.geoalchemy.org
+.. _Shapely: https://github.com/sgillies/shapely
+
If you want to use the spatial search API, you will need PostGIS installed
and enable the spatial features of your PostgreSQL database. See the
"Setting up PostGIS" section for details.
@@ -25,14 +31,15 @@
paster spatial initdb [srid] --config=../ckan/development.ini
-You can define the SRID of the geometry column. Default is 4258.
+You can define the SRID of the geometry column. Default is 4326. If you are not
+familiar with projections, we recommend to use the default value.
Problems you may find::
- LINE 1: SELECT AddGeometryColumn('package_extent','the_geom', E'4258...
+ LINE 1: SELECT AddGeometryColumn('package_extent','the_geom', E'4326...
^
HINT: No function matches the given name and argument types. You might need to add explicit type casts.
- "SELECT AddGeometryColumn('package_extent','the_geom', %s, 'POLYGON', 2)" ('4258',)
+ "SELECT AddGeometryColumn('package_extent','the_geom', %s, 'GEOMETRY', 2)" ('4326',)
PostGIS was not installed correctly. Please check the "Setting up PostGIS" section.
@@ -50,9 +57,9 @@
If you are using the spatial search feature, you can define the projection
in which extents are stored in the database with the following option. Use
the EPSG code as an integer (e.g 4326, 4258, 27700, etc). It defaults to
-4258::
+4326::
- ckan.spatial.srid = 4258
+ ckan.spatial.srid = 4326
@@ -65,11 +72,11 @@
initdb [srid]
- Creates the necessary tables. You must have PostGIS installed
and configured in the database.
- You can privide the SRID of the geometry column. Default is 4258.
+ You can privide the SRID of the geometry column. Default is 4326.
extents
- creates or updates the extent geometry column for packages with
- a bounding box defined in extras
+ an extent defined in the 'spatial' extra.
The commands should be run from the ckanext-spatial directory and expect
a development.ini file to be present. Most of the time you will specify
@@ -90,9 +97,28 @@
defined in the database, a CRS must be provided, in one of the following
forms:
-- urn:ogc:def:crs:EPSG::4258
-- EPSG:4258
-- 4258
+- urn:ogc:def:crs:EPSG::4326
+- EPSG:4326
+- 4326
+
+
+Geo-Indexing your packages
+==========================
+
+In order to make a package queryable by location, an special extra must
+be defined, with its key named 'spatial'. The value must be a valid GeoJSON_
+geometry, for example::
+
+ {"type":"Polygon","coordinates":[[[2.05827, 49.8625],[2.05827, 55.7447], [-6.41736, 55.7447], [-6.41736, 49.8625], [2.05827, 49.8625]]]}
+
+or::
+
+ { "type": "Point", "coordinates": [-3.145,53.078] }
+
+.. _GeoJSON: http://geojson.org
+
+Every time a package is created, updated or deleted, the extension will synchronize
+the information stored in the extra with the geometry table.
Setting up PostGIS
@@ -158,7 +184,7 @@
--------------------------
**Note:** If you run the ``initdb`` command, the table was already created for
-you. This sections just describes what's going on for those who want to know
+you. This section just describes what's going on for those who want to know
more.
To be able to store geometries and perform spatial operations, PostGIS
@@ -171,11 +197,12 @@
ALTER TABLE package_extent OWNER TO [your_user];
- SELECT AddGeometryColumn('package_extent','the_geom', 4258, 'POLYGON', 2);
+ SELECT AddGeometryColumn('package_extent','the_geom', 4326, 'POLYGON', 2);
This will add a geometry column in the ``package_extent`` table called
-``the_geom``, with the spatial reference system EPSG:4258. The stored
-geometries will be polygons, with 2 dimensions.
+``the_geom``, with the spatial reference system EPSG:4326. The stored
+geometries will be polygons, with 2 dimensions (The actual table on CKAN
+uses the GEOMETRY type to support multiple geometry types).
Have a look a the table definition, and see how PostGIS has created
three constraints to ensure that the geometries follow the parameters
@@ -193,4 +220,4 @@
Check constraints:
"enforce_dims_the_geom" CHECK (st_ndims(the_geom) = 2)
"enforce_geotype_the_geom" CHECK (geometrytype(the_geom) = 'POLYGON'::text OR the_geom IS NULL)
- "enforce_srid_the_geom" CHECK (st_srid(the_geom) = 4258)
+ "enforce_srid_the_geom" CHECK (st_srid(the_geom) = 4326)
--- a/ckanext/spatial/commands/spatial.py Mon Sep 26 10:27:20 2011 +0100
+++ b/ckanext/spatial/commands/spatial.py Thu Sep 29 15:40:49 2011 +0100
@@ -1,9 +1,12 @@
import sys
import re
from pprint import pprint
+import logging
from ckan.lib.cli import CkanCommand
-from ckanext.spatial.lib import save_extent
+from ckan.lib.helpers import json
+from ckanext.spatial.lib import save_package_extent
+log = logging.getLogger(__name__)
class Spatial(CkanCommand):
'''Performs spatially related operations.
@@ -12,11 +15,11 @@
spatial initdb [srid]
Creates the necessary tables. You must have PostGIS installed
and configured in the database.
- You can provide the SRID of the geometry column. Default is 4258.
+ You can provide the SRID of the geometry column. Default is 4326.
spatial extents
Creates or updates the extent geometry column for packages with
- a bounding box defined in extras
+ an extent defined in the 'spatial' extra.
The commands should be run from the ckanext-spatial directory and expect
a development.ini file to be present. Most of the time you will
@@ -63,19 +66,32 @@
conn = Session.connection()
packages = [extra.package \
for extra in \
- Session.query(PackageExtra).filter(PackageExtra.key == 'bbox-east-long').all()]
+ Session.query(PackageExtra).filter(PackageExtra.key == 'spatial').all()]
- error = False
+ errors = []
+ count = 0
for package in packages:
try:
- save_extent(package)
- except:
- errors = True
-
- if error:
- msg = "There was an error saving the package extent. Have you set up the package_extent table in the DB?"
- else:
- msg = "Done. Extents generated for %i packages" % len(packages)
+ value = package.extras['spatial']
+ log.debug('Received: %r' % value)
+ geometry = json.loads(value)
+
+ count += 1
+ except ValueError,e:
+ errors.append(u'Package %s - Error decoding JSON object: %s' % (package.id,str(e)))
+ except TypeError,e:
+ errors.append(u'Package %s - Error decoding JSON object: %s' % (package.id,str(e)))
+
+ save_package_extent(package.id,geometry)
+
+
+ Session.commit()
+
+ if errors:
+ msg = 'Errors were found:\n%s' % '\n'.join(errors)
+ print msg
+
+ msg = "Done. Extents generated for %i out of %i packages" % (count,len(packages))
print msg
--- a/ckanext/spatial/lib/__init__.py Mon Sep 26 10:27:20 2011 +0100
+++ b/ckanext/spatial/lib/__init__.py Thu Sep 29 15:40:49 2011 +0100
@@ -1,17 +1,21 @@
-from ckan.model import Session, repo
-from ckan.model import Package
+import logging
+
+from ckan.model import Session
from ckan.lib.base import config
+from ckanext.spatial.model import PackageExtent
+from shapely.geometry import asShape
-log = __import__("logging").getLogger(__name__)
+from geoalchemy import WKTSpatialElement
+log = logging.getLogger(__name__)
def get_srid(crs):
"""Returns the SRID for the provided CRS definition
The CRS can be defined in the following formats
- - urn:ogc:def:crs:EPSG::4258
- - EPSG:4258
- - 4258
+ - urn:ogc:def:crs:EPSG::4326
+ - EPSG:4326
+ - 4326
"""
if ':' in crs:
@@ -20,95 +24,51 @@
else:
srid = crs
- return srid
+ return int(srid)
-def save_extent(package,extent=False):
- '''Updates the package extent in the package_extent geometry column
- If no extent provided (as a dict with minx,miny,maxx,maxy and srid keys),
- the values stored in the package extras are used'''
+def save_package_extent(package_id, geometry = None, srid = None):
+ '''Adds, updates or deletes the package extent geometry.
- db_srid = int(config.get('ckan.spatial.srid', '4258'))
- conn = Session.connection()
+ package_id: Package unique identifier
+ geometry: a Python object implementing the Python Geo Interface
+ (i.e a loaded GeoJSON object)
+ srid: The spatial reference in which the geometry is provided.
+ If None, it defaults to the DB srid.
- srid = None
- if extent:
- minx = extent['minx']
- miny = extent['miny']
- maxx = extent['maxx']
- maxy = extent['maxy']
- if 'srid' in extent:
- srid = extent['srid']
- else:
- minx = float(package.extras.get('bbox-east-long'))
- miny = float(package.extras.get('bbox-south-lat'))
- maxx = float(package.extras.get('bbox-west-long'))
- maxy = float(package.extras.get('bbox-north-lat'))
+ Will throw ValueError if the geometry object does not provide a geo interface.
- if srid:
- srid = str(srid)
- try:
+ '''
+ db_srid = int(config.get('ckan.spatial.srid', '4326'))
- # Check if extent already exists
- rows = conn.execute('SELECT package_id FROM package_extent WHERE package_id = %s',package.id).fetchall()
- update =(len(rows) > 0)
- params = {'id':package.id, 'minx':minx,'miny':miny,'maxx':maxx,'maxy':maxy, 'db_srid': db_srid}
+ existing_package_extent = Session.query(PackageExtent).filter(PackageExtent.package_id==package_id).first()
- if update:
- # Update
- if srid and srid != db_srid:
- # We need to reproject the input geometry
- statement = """UPDATE package_extent SET
- the_geom = ST_Transform(
- ST_GeomFromText('POLYGON ((%(minx)s %(miny)s,
- %(maxx)s %(miny)s,
- %(maxx)s %(maxy)s,
- %(minx)s %(maxy)s,
- %(minx)s %(miny)s))',%(srid)s),
- %(db_srid)s)
- WHERE package_id = %(id)s
- """
- params.update({'srid': srid})
+ if geometry:
+ shape = asShape(geometry)
+
+ if not srid:
+ srid = db_srid
+
+ package_extent = PackageExtent(package_id=package_id,the_geom=WKTSpatialElement(shape.wkt, srid))
+
+ # Check if extent exists
+ if existing_package_extent:
+
+ # If extent exists but we received no geometry, we'll delete the existing one
+ if not geometry:
+ existing_package_extent.delete()
+ log.debug('Deleted extent for package %s' % package_id)
+ else:
+ # Check if extent changed
+ if Session.scalar(package_extent.the_geom.wkt) <> Session.scalar(existing_package_extent.the_geom.wkt):
+ # Update extent
+ existing_package_extent.the_geom = package_extent.the_geom
+ existing_package_extent.save()
+ log.debug('Updated extent for package %s' % package_id)
else:
- statement = """UPDATE package_extent SET
- the_geom = ST_GeomFromText('POLYGON ((%(minx)s %(miny)s,
- %(maxx)s %(miny)s,
- %(maxx)s %(maxy)s,
- %(minx)s %(maxy)s,
- %(minx)s %(miny)s))',%(db_srid)s)
- WHERE package_id = %(id)s
- """
- msg = 'Updated extent for package %s'
- else:
- # Insert
- if srid and srid != db_srid:
- # We need to reproject the input geometry
- statement = """INSERT INTO package_extent (package_id,the_geom) VALUES (
- %(id)s,
- ST_Transform(
- ST_GeomFromText('POLYGON ((%(minx)s %(miny)s,
- %(maxx)s %(miny)s,
- %(maxx)s %(maxy)s,
- %(minx)s %(maxy)s,
- %(minx)s %(miny)s))',%(srid)s),
- %(db_srid))
- )"""
- params.update({'srid': srid})
- else:
- statement = """INSERT INTO package_extent (package_id,the_geom) VALUES (
- %(id)s,
- ST_GeomFromText('POLYGON ((%(minx)s %(miny)s,
- %(maxx)s %(miny)s,
- %(maxx)s %(maxy)s,
- %(minx)s %(maxy)s,
- %(minx)s %(miny)s))',%(db_srid)s))"""
- msg = 'Created new extent for package %s'
+ log.debug('Extent for package %s unchanged' % package_id)
+ elif geometry:
+ # Insert extent
+ Session.add(package_extent)
+ log.debug('Created new extent for package %s' % package_id)
- conn.execute(statement,params)
-
- Session.commit()
- log.info(msg, package.id)
- return package
- except Exception,e:
- log.error('An error occurred when saving the extent for package %s: %r' % (package.id,e))
- raise Exception
--- a/ckanext/spatial/model.py Mon Sep 26 10:27:20 2011 +0100
+++ b/ckanext/spatial/model.py Thu Sep 29 15:40:49 2011 +0100
@@ -1,20 +1,41 @@
+from ckan.lib.base import config
from ckan.model import Session
+from ckan.model.meta import *
+from ckan.model.domain_object import DomainObject
+from geoalchemy import *
+from geoalchemy.postgis import PGComparator
-DEFAULT_SRID = 4258
+db_srid = int(config.get('ckan.spatial.srid', '4326'))
+package_extent_table = Table('package_extent', metadata,
+ Column('package_id', types.UnicodeText, primary_key=True),
+ GeometryExtensionColumn('the_geom', Geometry(2,srid=db_srid)))
-def setup(srid=4258):
+class PackageExtent(DomainObject):
+ def __init__(self, package_id=None, the_geom=None):
+ self.package_id = package_id
+ self.the_geom = the_geom
+
+mapper(PackageExtent, package_extent_table, properties={
+ 'the_geom': GeometryColumn(package_extent_table.c.the_geom,
+ comparator=PGComparator)})
+
+# enable the DDL extension
+GeometryDDL(package_extent_table)
+
+
+
+DEFAULT_SRID = 4326
+
+def setup(srid=None):
if not srid:
srid = DEFAULT_SRID
+
srid = str(srid)
connection = Session.connection()
- connection.execute("""CREATE TABLE package_extent(
- package_id text PRIMARY KEY
- );""")
+ connection.execute('CREATE TABLE package_extent(package_id text PRIMARY KEY)')
- #connection.execute('ALTER TABLE package_extent OWNER TO ?',user_name);
-
- connection.execute('SELECT AddGeometryColumn(\'package_extent\',\'the_geom\', %s, \'POLYGON\', 2)',srid)
+ connection.execute('SELECT AddGeometryColumn(\'package_extent\',\'the_geom\', %s, \'GEOMETRY\', 2)',srid)
Session.commit()
--- a/ckanext/spatial/plugin.py Mon Sep 26 10:27:20 2011 +0100
+++ b/ckanext/spatial/plugin.py Thu Sep 29 15:40:49 2011 +0100
@@ -6,34 +6,82 @@
import ckan.lib.helpers as h
+from ckan.lib.helpers import json
+
from ckan.plugins import implements, SingletonPlugin
from ckan.plugins import IRoutes, IConfigurer
from ckan.plugins import IConfigurable, IGenshiStreamFilter
+from ckan.plugins import IPackageController
+
+from ckan.logic import ValidationError
+from ckan.logic.action.update import package_error_summary
import html
+from ckanext.spatial.lib import save_package_extent
+
log = getLogger(__name__)
class SpatialQuery(SingletonPlugin):
implements(IRoutes, inherit=True)
+ implements(IPackageController, inherit=True)
def before_map(self, map):
map.connect('api_spatial_query', '/api/2/search/package/geo',
controller='ckanext.spatial.controllers.api:ApiController',
action='spatial_query')
-
+
return map
+ def create(self, package):
+ self.check_spatial_extra(package)
+
+ def edit(self, package):
+ self.check_spatial_extra(package)
+
+ def check_spatial_extra(self,package):
+ for extra in package.extras_list:
+ if extra.key == 'spatial':
+ if extra.state == 'active':
+ try:
+ log.debug('Received: %r' % extra.value)
+ geometry = json.loads(extra.value)
+ except ValueError,e:
+ error_dict = {'spatial':[u'Error decoding JSON object: %s' % str(e)]}
+ raise ValidationError(error_dict, error_summary=package_error_summary(error_dict))
+ except TypeError,e:
+ error_dict = {'spatial':[u'Error decoding JSON object: %s' % str(e)]}
+ raise ValidationError(error_dict, error_summary=package_error_summary(error_dict))
+
+ try:
+ save_package_extent(package.id,geometry)
+
+ except ValueError,e:
+ error_dict = {'spatial':[u'Error creating geometry: %s' % str(e)]}
+ raise ValidationError(error_dict, error_summary=package_error_summary(error_dict))
+ except Exception, e:
+ error_dict = {'spatial':[u'Error: %s' % str(e)]}
+ raise ValidationError(error_dict, error_summary=package_error_summary(error_dict))
+
+ elif extra.state == 'deleted':
+ # Delete extent from table
+ save_package_extent(package.id,None)
+
+ break
+
+
+ def delete(self, package):
+ save_package_extent(package.id,None)
class WMSPreview(SingletonPlugin):
-
+
implements(IGenshiStreamFilter)
implements(IRoutes, inherit=True)
implements(IConfigurer, inherit=True)
-
+
def filter(self, stream):
from pylons import request, tmpl_context as c
routes = request.environ.get('pylons.routes_dict')
@@ -65,7 +113,7 @@
map.connect('api_spatial_query', '/api/2/search/package/geo',
controller='ckanext.spatial.controllers.api:ApiController',
action='spatial_query')
-
+
return map
def update_config(self, config):
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/pip-requirements.txt Thu Sep 29 15:40:49 2011 +0100
@@ -0,0 +1,2 @@
+GeoAlchemy>=0.6
+Shapely>=1.2.13
http://bitbucket.org/okfn/ckanext-spatial/changeset/9da75d0446e7/
changeset: 9da75d0446e7
branch: feature-1272-store-extents-from-form
user: amercader
date: 2011-09-29 16:41:45
summary: Use GeoAlchemy on the spatial query controller
affected #: 1 file (-1 bytes)
--- a/ckanext/spatial/controllers/api.py Thu Sep 29 15:40:49 2011 +0100
+++ b/ckanext/spatial/controllers/api.py Thu Sep 29 15:41:45 2011 +0100
@@ -1,64 +1,52 @@
-from ckan.lib.helpers import json
-import ckan.lib.helpers as h
-from ckan.lib.base import c, g, request, \
- response, session, render, config, abort, redirect
+from string import Template
+from ckan.lib.base import request, config, abort
from ckan.controllers.api import ApiController as BaseApiController
-
from ckan.model import Session
from ckanext.spatial.lib import get_srid
+from ckanext.spatial.model import PackageExtent
+from geoalchemy import WKTSpatialElement, functions
class ApiController(BaseApiController):
-
- db_srid = int(config.get('ckan.spatial.srid', '4258'))
+
+ db_srid = int(config.get('ckan.spatial.srid', '4326'))
+
+ bbox_template = Template('POLYGON (($minx $miny, $minx $maxy, $maxx $maxy, $maxx $miny, $minx $miny))')
def spatial_query(self):
+
+ error_400_msg = 'Please provide a suitable bbox parameter [minx,miny,maxx,maxy]'
+
if not 'bbox' in request.params:
- abort(400)
-
+ abort(400,error_400_msg)
+
bbox = request.params['bbox'].split(',')
if len(bbox) is not 4:
- abort(400)
-
- minx = float(bbox[0])
- miny = float(bbox[1])
- maxx = float(bbox[2])
- maxy = float(bbox[3])
+ abort(400,error_400_msg)
- params = {'minx':minx,'miny':miny,'maxx':maxx,'maxy':maxy,'db_srid':self.db_srid}
-
+ try:
+ minx = float(bbox[0])
+ miny = float(bbox[1])
+ maxx = float(bbox[2])
+ maxy = float(bbox[3])
+ except ValueError,e:
+ abort(400,error_400_msg)
+
+
+ wkt = self.bbox_template.substitute(minx=minx,miny=miny,maxx=maxx,maxy=maxy)
+
srid = get_srid(request.params.get('crs')) if 'crs' in request.params else None
if srid and srid != self.db_srid:
- # The input bounding box is defined in another projection, we need
- # to transform it
- statement = """SELECT package_id FROM package_extent WHERE
- ST_Intersects(
- ST_Transform(
- ST_GeomFromText('POLYGON ((%(minx)s %(miny)s,
- %(maxx)s %(miny)s,
- %(maxx)s %(maxy)s,
- %(minx)s %(maxy)s,
- %(minx)s %(miny)s))',%(srid)s),
- %(db_srid)s)
- ,the_geom)"""
- params.update({'srid': srid})
+ # Input geometry needs to be transformed to the one used on the database
+ input_geometry = functions.transform(WKTSpatialElement(wkt,srid),self.db_srid)
else:
- statement = """SELECT package_id FROM package_extent WHERE
- ST_Intersects(
- ST_GeomFromText('POLYGON ((%(minx)s %(miny)s,
- %(maxx)s %(miny)s,
- %(maxx)s %(maxy)s,
- %(minx)s %(maxy)s,
- %(minx)s %(miny)s))',%(db_srid)s),
- the_geom)"""
- conn = Session.connection()
- rows = conn.execute(statement,params)
- ids = [row['package_id'] for row in rows]
-
+ input_geometry = WKTSpatialElement(wkt,self.db_srid)
+
+ extents = Session.query(PackageExtent).filter(PackageExtent.the_geom.intersects(input_geometry))
+ ids = [extent.package_id for extent in extents]
+
output = dict(count=len(ids),results=ids)
return self._finish_ok(output)
-
-
Repository URL: https://bitbucket.org/okfn/ckanext-spatial/
--
This is a commit notification from bitbucket.org. You are receiving
this because you have the service enabled, addressing the recipient of
this email.
More information about the ckan-changes
mailing list