[ckan-dev] Datapusher issue

Tue Nov 26 14:33:40 UTC 2013

The datapusher is active, but it's run into a problem whilst running, it
just sounds like another resource that is currently in the datastore has
the same name "census-marriage-2-csv". The alias it's mentioning is a
database view which gets created when a resource gets sent to the
datastore, there has been talk of removing the aliases as it does cause the
conflicts you have encountered.

On 26 November 2013 11:06, Athos Georgiou <ag242 at st-andrews.ac.uk> wrote:

>  Hello,
>
>
>
> After having installed the datapusher found here:
> http://datapusher.readthedocs.org/en/latest/install.html
>
> I have encountered an error upon creating a dataset by uploading a CSV
> file. The error generated within ckan is
>
>
>
>  This resource can not be previewed at the moment. Click here for more
> information.<http://researchdata.st-andrews.ac.uk/dataset/m12/resource/f4a0d5d6-526b-4419-92fa-4b4c31918ba0>
>
> Could not load preview: DataProxy returned an error (Data transformation
> failed. HTTPException: Deadline exceeded while waiting for HTTP response
> from URL:
> http://researchdata.st-andrews.ac.uk/storage/f/2013-11-26T10%3A33%3A52.558Z/census-marriage-2.csv
> )
>
>
>
> The Error log /var/log/apache2/datapusher.error.log shows:
>
>
>
>
>
> [Tue Nov 26 10:46:08 2013] [error]
> /usr/lib/ckan/default/lib/python2.7/site-packages/sqlalchemy/engine/default.py:463:
> SAWarning: Unicode type received non-unicode bind param value.
>
> [Tue Nov 26 10:46:08 2013] [error]
> param.append(processors[key](compiled_params[key]))
>
> [Tue Nov 26 10:46:14 2013] [error] Fetching from:
> http://researchdata.st-andrews.ac.uk/storage/f/2013-11-26T10%3A45%3A59.976Z/census-marriage-2.csv
>
> [Tue Nov 26 10:46:14 2013] [error] Deleting
> "1f14b9e4-a645-4f0b-bba3-446297752dc8" from datastore.
>
> [Tue Nov 26 10:46:14 2013] [error] Determined headers and types: [{'type':
> u'numeric', 'id': u'year'}, {'type': u'numeric', 'id': u'age'}, {'type':
> u'numeric', 'id': u'marst'}, {'type': u'numeric', 'id': u'sex'}, {'type':
> u'numeric', 'id': u'people'}]
>
> [Tue Nov 26 10:46:14 2013] [error] Saving chunk 0
>
> [Tue Nov 26 10:46:15 2013] [error] Job "push_to_datastore (trigger:
> RunTriggerNow, run = True, next run at: None)" raised an exception
>
> [Tue Nov 26 10:46:15 2013] [error] Traceback (most recent call last):
>
> [Tue Nov 26 10:46:15 2013] [error]   File
> "/usr/lib/ckan/default/lib/python2.7/site-packages/APScheduler-2.1.1-py2.7.egg/apscheduler/scheduler.py",
> line 512, in _run_job
>
> [Tue Nov 26 10:46:15 2013] [error]     retval = job.func(*job.args,
> **job.kwargs)
>
> [Tue Nov 26 10:46:15 2013] [error]   File
> "/usr/lib/ckan/default/src/datapusher/datapusher/jobs.py", line 318, in
> push_to_datastore
>
> [Tue Nov 26 10:46:15 2013] [error]     records, api_key, ckan_url)
>
> [Tue Nov 26 10:46:15 2013] [error]   File
> "/usr/lib/ckan/default/src/datapusher/datapusher/jobs.py", line 152, in
> send_resource_to_datastore
>
> [Tue Nov 26 10:46:15 2013] [error]     check_response(r, url, 'CKAN
> DataStore')
>
> [Tue Nov 26 10:46:15 2013] [error]   File
> "/usr/lib/ckan/default/src/datapusher/datapusher/jobs.py", line 84, in
> check_response
>
> [Tue Nov 26 10:46:15 2013] [error]     resp=pprint.pformat(json_response)))
>
> [Tue Nov 26 10:46:15 2013] [error] JobError: CKAN DataStore bad response.
> Status code: 409 Conflict. At:
> http://researchdata.st-andrews.ac.uk/api/3/action/datastore_create.
> Response: {u'error': {u'__type': u'Validation Error',
>
> [Tue Nov 26 10:46:15 2013] [error]             u'alias': [u'The alias
> "census-marriage-2-csv" already exists.']},
>
> [Tue Nov 26 10:46:15 2013] [error]  u'help': u'Adds a new table to the
> DataStore.\\n\\n    The datastore_create action allows you to post JSON
> data to be\\n    stored against a resource. This endpoint also supports
> altering tables,\\n    aliases and indexes and bulk insertion. This
> endpoint can be called multiple\\n    times to initially insert more data,
> add fields, change the aliases or indexes\\n    as well as the primary
> keys.\\n\\n    See :ref:`fields` and :ref:`records` for details on how to
> lay out records.\\n\\n    :param resource_id: resource id that the data is
> going to be stored against.\\n    :type resource_id: string\\n    :param
> aliases: names for read only aliases of the resource. (optional)\\n
> :type aliases: list or comma separated string\\n    :param fields:
> fields/columns and their extra metadata. (optional)\\n    :type fields:
> list of dictionaries\\n    :param records: the data, eg: [{"dob": "2005",
> "some_stuff": ["a", "b"]}]  (optional)\\n    :type records: list of
> dictionaries\\n    :param primary_key: fields that represent a unique key
> (optional)\\n    :type primary_key: list or comma separated string\\n
> :param indexes: indexes on table (optional)\\n    :type indexes: list or
> comma separated string\\n\\n    Please note that setting the ``aliases``,
> ``indexes`` or ``primary_key`` replaces the exising\\n    aliases or
> constraints. Setting ``records`` appends the provided records to the
> resource.\\n\\n    **Results:**\\n\\n    :returns: The newly created data
> object.\\n    :rtype: dictionary\\n\\n    See :ref:`fields` and
> :ref:`records` for details on how to lay out records.\\n\\n    ',
>
> [Tue Nov 26 10:46:15 2013] [error]  u'success': False}
>
>
>
> Is the error log above any indication why the datapusher is not activated?
>
>
>
> Thanks
>
>
>
> *Athos Georgiou*
>
> *Application Developer - Research Computing *
>
> [image: Description: Description: Description: Description: Description:
> Description: Description: cid:46E12C47-B748-4BCA-B41C-A94306C93BF6]
>
> *IT Services, University of St Andrews, St Andrews, Fife, KY16 9AL*
>
> *Tel: **01334 4644731   E-mail: ag242 at st-andrews.ac.uk
> <ag242 at st-andrews.ac.uk>*
>
>
>
> The University of St Andrews is a charity registered in Scotland (No
> SC013532)
>
>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20131126/ef8d4628/attachment-0003.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 13560 bytes
Desc: not available
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20131126/ef8d4628/attachment-0003.png>