[ckan-dev] How to load Ckan configuration inside extension?

[Eduardo Santos]

Hi Sean,

Thank you for the answer. Yes, I've seen this extension, but I had very
specific requirements and only a few time, and it would take me more
time to know this extension than to develop a new one.

I'm going to share my code, but I don't know if it's going to be useful,
because there's a lot of other extensions to achieve the same thing. I
just can't release it now because there's a lot of client-specific
things that would not only be broken, but it would also not fit any
other environment. I'll move this code to another extension and release
the other work. It basically does this:

1 - Monitor one specific directory in the computer for changes using
pyinotify;
2 - When there's a new file detected, send it to be processed;
3 - The file is a zip data collection with self contained data records,
also zipped. A model very similar to ODF, but much more simple;
4 - Parse one RDF record inside the zip record and insert it into Ckan
as a resource;
5 - Store resource metadata on Elastic Search;
6 - Store binary files in OFS;
7 - keep a log of all transactions, even parsing errors.

As you can see, it's really a lot of operations, and the code is still a
mess, so I don't know if it's going to be useful.

Anyway, the problem persists. As the loading script run as a daemon, the
only way I found to parse the .ini file is with the following code:

import ConfigParser
config = ConfigParser.ConfigParser()

# Find config file
config_file = os.environ.get('CKAN_CONFIG')
if not config_file:
     config_file =  os.path.join(
          os.path.dirname(os.path.abspath(__file__)),
'../../../ckan/development.ini')

config.read(config_file)

# Load DB connection
from sqlalchemy import create_engine
engine = create_engine(config.get('app:main','sqlalchemy.url'))
model.metadata.bind = engine

I'm sure it's not the best implementation, and it would help me very
much if you could point me a better way to do it.


Em Seg, 2012-07-09 às 10:34 +0200, Sean Hammond escreveu:
> Hi Eduardo, are you sure you don't want to use ckanext-harvest for this?
> 
> https://github.com/okfn/ckanext-harvest
> 
> It's an officially supported extension that might do just what you want.
> 
> Alternatively, if you could put your extension as a whole on github I
> could take a look at it for you
> 
> On Thu, Jul 05, 2012 at 07:01:01PM -0300, Eduardo Santos wrote:
> > Hi everybody,
> > 
> > I'm sure this is a slilly doubt, but I'm working on it for weeks and still
> > don't know the answer. Maybe the problem is that I'm not the best Python
> > programmer myself. :)
> > 
> > The problem is: I'm creating a Ckan extension that will run as daemon to
> > import data from external sources and load it into Ckan as resources. I
> > know there are a lot of extensions to do it, but the fastest way was to
> > build my own. I have the following code:
> > 
> > 
> > <code>
> > from logging import getLogger
> > from sqlalchemy import event
> > from sqlalchemy import distinct
> > from sqlalchemy import Table
> > from sqlalchemy import Column
> > from sqlalchemy.engine.reflection import Inspector
> > from sqlalchemy.orm import backref, relation
> > 
> > from ckan.lib.base import config
> > from ckan import model
> > from ckan.model.meta import *
> > from ckan.model.domain_object import DomainObject
> > 
> > def setup():
> > 
> >     if datadaemon_table is None or error_table is None:
> >         define_tables()
> >         log.debug('Spatial tables defined in memory')
> > 
> >     if model.repo.are_tables_created():
> > 
> >         if not datadaemon_table.exists():
> >             try:
> >                 datadaemon_table.create()
> >             except Exception,e:
> >                 # Make sure the table does not remain incorrectly created
> >                 # (eg without geom column or constraints)
> >                 if datadaemon_table.exists():
> >                     Session.execute('DROP TABLE dt_files')
> >                     Session.commit()
> > 
> >                 raise e
> > 
> >             log.debug('Datadaemon tables created')
> >         else:
> >             log.debug('Datadaemon tables already exist')
> >             # Future migrations go here
> > 
> >     def __repr__(self):
> >         return "<DataRepository('%s','%s', '%s')>" % (self.hash,
> > self.original_file, self.creation_date)
> > 
> > 
> > class DataRepository(DomainObject):
> >     """
> >     The repository will hold information about every uploaded files
> >     """
> >     __tablename__ = 'dt_files'
> > 
> >     hash = Column(UnicodeText, primary_key=True,nullable=False)
> >     original_file = Column(UnicodeText,nullable=False)
> >     creation_date = Column(DateTime(timezone=True),server_default='now()')
> > 
> >     def __init__(self, hash, original_file, creation_date):
> >         self.hash = hash
> >         self.original_file = original_file
> >         self.creation_date = creation_date
> > </code>
> > 
> > 
> > I'm creating a class that will add some tables to the system. In another
> > file there's the following code:
> > 
> > 
> > <code>
> > from ckan import plugins
> > from ckanext.datadaemon.model import setup as setup_model
> > from ckan.logic import get_action
> > from ckan import model
> > 
> >  <goog_989364852>
> > from ckanext.datadaemon.model import setup as setup_model
> > from ckanext.datadaemon.model import DataRepository
> > 
> > <goog_989364840>
> > setup_model()
> > </code>
> > 
> > At that point, when I call setup_model(), I get the following error:
> > 
> > UnboundExecutionError: The MetaData is not bound to an Engine or
> > Connection.  Execution can not proceed without a database to execute
> > against.  Either execute with an explicit connection or assign the
> > MetaData's .bind to enable implicit execution.
> > 
> > It seems like the script is not loading pylons configuration containing
> > database connections.
> > 
> > Can somebody point me how I can load configuration parameters from ckan
> > development.ini file?
> > 
> > -- 
> > Eduardo Santos
> > Analista de Sistemas
> > 
> > http://www.eduardosan.com
> > http://twitter.com/eduardosan
> 
> > _______________________________________________
> > ckan-dev mailing list
> > ckan-dev at lists.okfn.org
> > http://lists.okfn.org/mailman/listinfo/ckan-dev
> 
> 
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev

-- 
Eduardo Santos
Analista de Sistemas

http://www.eduardosan.com
http://twitter.com/eduardosan