[ckan-dev] Datapusher and APScheduler errors

[Jessica Good]

Thanks Adria. All I changed was to run it with python datapusher/main.py
deployment/datapusher_settings.py and it finally worked. Thanks.

Jessica Good
GIS Specialist & Developer
Arizona Geological Survey
http://www.azgs.az.gov/
jessica.good at azgs.az.gov <jessica.alisdairi at azgs.az.gov>


On Wed, Jul 23, 2014 at 9:30 AM, Adrià Mercader <adria.mercader at okfn.org>
wrote:

> Hi Jessica,
>
> The DataPusher documentation and deployment is a bit of a mess and
> needs to be improved.
>
> I was bitten by the same APScheduler version problem [1] today. The
> second stack trace usually means that the
> deployment/datapusher_settings.py file (or more precisely the database
> in the config) could not be found.
>
> As an alternative this also works for me for running the DataPusher
> locally:
>
> python datapusher/main.py deployment/datapusher_settings.py
>
>
>
> Adrià
>
> [1] https://github.com/ckan/ckan-service-provider/issues/16
>
> On 23 July 2014 06:59, Jessica Good <jessica.good at azgs.az.gov> wrote:
> > I have been having a few issues getting the Datapusher to work correctly.
> > Looks like the problem has to do with apscheduler. If anyone has any
> ideas
> > about what could be wrong please let me know.
> >
> > I am running ckan release-v2.2.1. The datastore doesn't return an error
> when
> > I do curl -X GET
> > "
> http://127.0.0.1:5000/api/3/action/datastore_search?resource_id=_table_metadata
> "
> > so it looks like that is working fine. I have ckan.site_url =
> > http://localhost:5000 set in my development.ini file.
> >
> > When I try to run the Datapusher with
> >
> JOB_CONFIG='/home/ckanenv/src/datapusher/deployment/datapusher_settings.py'
> > python wsgi.py I get an error which I've pasted below as Stack Trace 1. I
> > see that the version of APScheduler installed is 3.0.0rc1. Looking at the
> > apscheduler site-package I see apscheduler.schedulers but not
> > apscheduler.scheduler as the ckanserviceprovider package is referencing.
> > There are a bunch of subsequent similar errors if I attempt to modify
> > ckanserviceprovider. So at some point apscheduler was updated but
> > ckanserviceprovider was not updated to work with the new version.
> >
> > If I roll back APScheduler to 2.1.2 then
> >
> JOB_CONFIG='/home/ckanenv/src/datapusher/deployment/datapusher_settings.py'
> > python wsgi.py executes successfully and I can see the Datapusher
> running at
> > http://localhost:8800/. However, if I try to upload a very simple csv
> file,
> > with no special characters, I get the error in Stack Trace 2 below. Files
> > are uploaded but they are not pushed to the datastore.
> >
> > How can I get the Datapusher to run properly?
> >
> > Stack Trace 1:
> > (ckanenv)ngds at ngds-VirtualBox:~/ckanenv/src/datapusher$
> > JOB_CONFIG='/home/ckanv/src/datapusher/deployment/datapusher_settings.py'
> > python wsgi.py
> > Traceback (most recent call last):
> >   File "wsgi.py", line 1, in <module>
> >     import ckanserviceprovider.web as web
> >   File
> >
> "/home/ngds/ckanenv/local/lib/python2.7/site-packages/ckanserviceprovider/web.py",
> > line 16, in <module>
> >     import apscheduler.scheduler as apscheduler
> > ImportError: No module named scheduler
> >
> >
> > Stack Trace 2:
> > (ckanenv)ngds at ngds-VirtualBox:~/ckanenv/src/datapusher$
> > JOB_CONFIG='/home/ckanv/src/datapusher/deployment/datapusher_settings.py'
> > python wsgi.py
> >
> /home/ngds/ckanenv/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py:463:
> > SAWarning: Unicode type received non-unicode bind param value.
> >   param.append(processors[key](compiled_params[key]))
> > Exception in thread APScheduler:
> > Traceback (most recent call last):
> >   File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
> >     self.run()
> >   File "/usr/lib/python2.7/threading.py", line 504, in run
> >     self.__target(*self.__args, **self.__kwargs)
> >   File
> >
> "/home/ngds/ckanenv/local/lib/python2.7/site-packages/apscheduler/scheduler.py",
> > line 581, in _main_loop
> >     next_wakeup_time = self._process_jobs(now)
> >   File
> >
> "/home/ngds/ckanenv/local/lib/python2.7/site-packages/apscheduler/scheduler.py",
> > line 560, in _process_jobs
> >     self._remove_job(job, alias, jobstore)
> >   File
> >
> "/home/ngds/ckanenv/local/lib/python2.7/site-packages/apscheduler/scheduler.py",
> > line 294, in _remove_job
> >     jobstore.remove_job(job)
> >   File
> >
> "/home/ngds/ckanenv/local/lib/python2.7/site-packages/apscheduler/jobstores/sqlalchemy_store.py",
> > line 64, in remove_job
> >     self.engine.execute(delete)
> >   File
> >
> "/home/ngds/ckanenv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
> > line 2447, in execute
> >     return connection.execute(statement, *multiparams, **params)
> >   File
> >
> "/home/ngds/ckanenv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
> > line 1449, in execute
> >     params)
> >   File
> >
> "/home/ngds/ckanenv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
> > line 1584, in _execute_clauseelement
> >     compiled_sql, distilled_params
> >   File
> >
> "/home/ngds/ckanenv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
> > line 1698, in _execute_context
> >     context)
> >   File
> >
> "/home/ngds/ckanenv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
> > line 1691, in _execute_context
> >     context)
> >   File
> >
> "/home/ngds/ckanenv/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py",
> > line 331, in do_execute
> >     cursor.execute(statement, parameters)
> > OperationalError: (OperationalError) no such table: apscheduler_jobs
> > u'DELETE FROM apscheduler_jobs WHERE apscheduler_jobs.id = ?' (1,)
> >
> >
> > Jessica Good
> > GIS Specialist & Developer
> > Arizona Geological Survey
> > http://www.azgs.az.gov/
> > jessica.good at azgs.az.gov
> >
> >
> > _______________________________________________
> > ckan-dev mailing list
> > ckan-dev at lists.okfn.org
> > https://lists.okfn.org/mailman/listinfo/ckan-dev
> > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
> >
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20140723/b833835c/attachment-0003.html>