[ckan-dev] Datapusher and APScheduler errors

Wed Jul 23 16:30:31 UTC 2014

Hi Jessica,

The DataPusher documentation and deployment is a bit of a mess and
needs to be improved.

I was bitten by the same APScheduler version problem [1] today. The
second stack trace usually means that the
deployment/datapusher_settings.py file (or more precisely the database
in the config) could not be found.

As an alternative this also works for me for running the DataPusher locally:

python datapusher/main.py deployment/datapusher_settings.py

Adrià

[1] https://github.com/ckan/ckan-service-provider/issues/16

On 23 July 2014 06:59, Jessica Good <jessica.good at azgs.az.gov> wrote:
> I have been having a few issues getting the Datapusher to work correctly.
> Looks like the problem has to do with apscheduler. If anyone has any ideas
> about what could be wrong please let me know.
>
> I am running ckan release-v2.2.1. The datastore doesn't return an error when
> I do curl -X GET
> "http://127.0.0.1:5000/api/3/action/datastore_search?resource_id=_table_metadata"
> so it looks like that is working fine. I have ckan.site_url =
> http://localhost:5000 set in my development.ini file.
>
> When I try to run the Datapusher with
> JOB_CONFIG='/home/ckanenv/src/datapusher/deployment/datapusher_settings.py'
> python wsgi.py I get an error which I've pasted below as Stack Trace 1. I
> see that the version of APScheduler installed is 3.0.0rc1. Looking at the
> apscheduler site-package I see apscheduler.schedulers but not
> apscheduler.scheduler as the ckanserviceprovider package is referencing.
> There are a bunch of subsequent similar errors if I attempt to modify
> ckanserviceprovider. So at some point apscheduler was updated but
> ckanserviceprovider was not updated to work with the new version.
>
> If I roll back APScheduler to 2.1.2 then
> JOB_CONFIG='/home/ckanenv/src/datapusher/deployment/datapusher_settings.py'
> python wsgi.py executes successfully and I can see the Datapusher running at
> http://localhost:8800/. However, if I try to upload a very simple csv file,
> with no special characters, I get the error in Stack Trace 2 below. Files
> are uploaded but they are not pushed to the datastore.
>
> How can I get the Datapusher to run properly?
>
> Stack Trace 1:
> (ckanenv)ngds at ngds-VirtualBox:~/ckanenv/src/datapusher$
> JOB_CONFIG='/home/ckanv/src/datapusher/deployment/datapusher_settings.py'
> python wsgi.py
> Traceback (most recent call last):
>   File "wsgi.py", line 1, in <module>
>     import ckanserviceprovider.web as web
>   File
> "/home/ngds/ckanenv/local/lib/python2.7/site-packages/ckanserviceprovider/web.py",
> line 16, in <module>
>     import apscheduler.scheduler as apscheduler
> ImportError: No module named scheduler
>
>
> Stack Trace 2:
> (ckanenv)ngds at ngds-VirtualBox:~/ckanenv/src/datapusher$
> JOB_CONFIG='/home/ckanv/src/datapusher/deployment/datapusher_settings.py'
> python wsgi.py
> /home/ngds/ckanenv/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py:463:
> SAWarning: Unicode type received non-unicode bind param value.
>   param.append(processors[key](compiled_params[key]))
> Exception in thread APScheduler:
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
>     self.run()
>   File "/usr/lib/python2.7/threading.py", line 504, in run
>     self.__target(*self.__args, **self.__kwargs)
>   File
> "/home/ngds/ckanenv/local/lib/python2.7/site-packages/apscheduler/scheduler.py",
> line 581, in _main_loop
>     next_wakeup_time = self._process_jobs(now)
>   File
> "/home/ngds/ckanenv/local/lib/python2.7/site-packages/apscheduler/scheduler.py",
> line 560, in _process_jobs
>     self._remove_job(job, alias, jobstore)
>   File
> "/home/ngds/ckanenv/local/lib/python2.7/site-packages/apscheduler/scheduler.py",
> line 294, in _remove_job
>     jobstore.remove_job(job)
>   File
> "/home/ngds/ckanenv/local/lib/python2.7/site-packages/apscheduler/jobstores/sqlalchemy_store.py",
> line 64, in remove_job
>     self.engine.execute(delete)
>   File
> "/home/ngds/ckanenv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
> line 2447, in execute
>     return connection.execute(statement, *multiparams, **params)
>   File
> "/home/ngds/ckanenv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
> line 1449, in execute
>     params)
>   File
> "/home/ngds/ckanenv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
> line 1584, in _execute_clauseelement
>     compiled_sql, distilled_params
>   File
> "/home/ngds/ckanenv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
> line 1698, in _execute_context
>     context)
>   File
> "/home/ngds/ckanenv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
> line 1691, in _execute_context
>     context)
>   File
> "/home/ngds/ckanenv/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py",
> line 331, in do_execute
>     cursor.execute(statement, parameters)
> OperationalError: (OperationalError) no such table: apscheduler_jobs
> u'DELETE FROM apscheduler_jobs WHERE apscheduler_jobs.id = ?' (1,)
>
>
> Jessica Good
> GIS Specialist & Developer
> Arizona Geological Survey
> http://www.azgs.az.gov/
> jessica.good at azgs.az.gov
>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>