[ckan-dev] Datapusher issue

Tue Nov 26 11:06:42 UTC 2013

Hello,

After having installed the datapusher found here: http://datapusher.readthedocs.org/en/latest/install.html
I have encountered an error upon creating a dataset by uploading a CSV file. The error generated within ckan is

 This resource can not be previewed at the moment. Click here for more information.<http://researchdata.st-andrews.ac.uk/dataset/m12/resource/f4a0d5d6-526b-4419-92fa-4b4c31918ba0>

Could not load preview: DataProxy returned an error (Data transformation failed. HTTPException: Deadline exceeded while waiting for HTTP response from URL: http://researchdata.st-andrews.ac.uk/storage/f/2013-11-26T10%3A33%3A52.558Z/census-marriage-2.csv)

The Error log /var/log/apache2/datapusher.error.log shows:

[Tue Nov 26 10:46:08 2013] [error] /usr/lib/ckan/default/lib/python2.7/site-packages/sqlalchemy/engine/default.py:463: SAWarning: Unicode type received non-unicode bind param value.
[Tue Nov 26 10:46:08 2013] [error]   param.append(processors[key](compiled_params[key]))
[Tue Nov 26 10:46:14 2013] [error] Fetching from: http://researchdata.st-andrews.ac.uk/storage/f/2013-11-26T10%3A45%3A59.976Z/census-marriage-2.csv
[Tue Nov 26 10:46:14 2013] [error] Deleting "1f14b9e4-a645-4f0b-bba3-446297752dc8" from datastore.
[Tue Nov 26 10:46:14 2013] [error] Determined headers and types: [{'type': u'numeric', 'id': u'year'}, {'type': u'numeric', 'id': u'age'}, {'type': u'numeric', 'id': u'marst'}, {'type': u'numeric', 'id': u'sex'}, {'type': u'numeric', 'id': u'people'}]
[Tue Nov 26 10:46:14 2013] [error] Saving chunk 0
[Tue Nov 26 10:46:15 2013] [error] Job "push_to_datastore (trigger: RunTriggerNow, run = True, next run at: None)" raised an exception
[Tue Nov 26 10:46:15 2013] [error] Traceback (most recent call last):
[Tue Nov 26 10:46:15 2013] [error]   File "/usr/lib/ckan/default/lib/python2.7/site-packages/APScheduler-2.1.1-py2.7.egg/apscheduler/scheduler.py", line 512, in _run_job
[Tue Nov 26 10:46:15 2013] [error]     retval = job.func(*job.args, **job.kwargs)
[Tue Nov 26 10:46:15 2013] [error]   File "/usr/lib/ckan/default/src/datapusher/datapusher/jobs.py", line 318, in push_to_datastore
[Tue Nov 26 10:46:15 2013] [error]     records, api_key, ckan_url)
[Tue Nov 26 10:46:15 2013] [error]   File "/usr/lib/ckan/default/src/datapusher/datapusher/jobs.py", line 152, in send_resource_to_datastore
[Tue Nov 26 10:46:15 2013] [error]     check_response(r, url, 'CKAN DataStore')
[Tue Nov 26 10:46:15 2013] [error]   File "/usr/lib/ckan/default/src/datapusher/datapusher/jobs.py", line 84, in check_response
[Tue Nov 26 10:46:15 2013] [error]     resp=pprint.pformat(json_response)))
[Tue Nov 26 10:46:15 2013] [error] JobError: CKAN DataStore bad response. Status code: 409 Conflict. At: http://researchdata.st-andrews.ac.uk/api/3/action/datastore_create. Response: {u'error': {u'__type': u'Validation Error',
[Tue Nov 26 10:46:15 2013] [error]             u'alias': [u'The alias "census-marriage-2-csv" already exists.']},
[Tue Nov 26 10:46:15 2013] [error]  u'help': u'Adds a new table to the DataStore.\\n\\n    The datastore_create action allows you to post JSON data to be\\n    stored against a resource. This endpoint also supports altering tables,\\n    aliases and indexes and bulk insertion. This endpoint can be called multiple\\n    times to initially insert more data, add fields, change the aliases or indexes\\n    as well as the primary keys.\\n\\n    See :ref:`fields` and :ref:`records` for details on how to lay out records.\\n\\n    :param resource_id: resource id that the data is going to be stored against.\\n    :type resource_id: string\\n    :param aliases: names for read only aliases of the resource. (optional)\\n    :type aliases: list or comma separated string\\n    :param fields: fields/columns and their extra metadata. (optional)\\n    :type fields: list of dictionaries\\n    :param records: the data, eg: [{"dob": "2005", "some_stuff": ["a", "b"]}]  (optional)\\n    :type records: list of dictionaries\\n    :param primary_key: fields that represent a unique key (optional)\\n    :type primary_key: list or comma separated string\\n    :param indexes: indexes on table (optional)\\n    :type indexes: list or comma separated string\\n\\n    Please note that setting the ``aliases``, ``indexes`` or ``primary_key`` replaces the exising\\n    aliases or constraints. Setting ``records`` appends the provided records to the resource.\\n\\n    **Results:**\\n\\n    :returns: The newly created data object.\\n    :rtype: dictionary\\n\\n    See :ref:`fields` and :ref:`records` for details on how to lay out records.\\n\\n    ',
[Tue Nov 26 10:46:15 2013] [error]  u'success': False}

Is the error log above any indication why the datapusher is not activated?

Thanks

Athos Georgiou
Application Developer - Research Computing
[Description: Description: Description: Description: Description: Description: Description: cid:46E12C47-B748-4BCA-B41C-A94306C93BF6]
IT Services, University of St Andrews, St Andrews, Fife, KY16 9AL
Tel: 01334 4644731   E-mail: ag242 at st-andrews.ac.uk<mailto:ag242 at st-andrews.ac.uk>

The University of St Andrews is a charity registered in Scotland (No SC013532)

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20131126/7323f741/attachment-0002.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 13560 bytes
Desc: image001.png
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20131126/7323f741/attachment-0002.png>