[ckan-dev] Datapusher issue

Tue Nov 26 15:28:08 UTC 2013

Hello

I have just fixed this issue, by removing the aliases from the datapusher
as this causes issues on duplicate resource names (as Joe mentioned)

You will need to refetch the latest datapusher data and restart apache.
cd /usr/lib/ckan/default/src/datapusher
git pull
sudo service apache2 restart

Thanks

David

On 26 November 2013 14:33, Joe Tsoi <joe.tsoi at okfn.org> wrote:

> The datapusher is active, but it's run into a problem whilst running, it
> just sounds like another resource that is currently in the datastore has
> the same name "census-marriage-2-csv". The alias it's mentioning is a
> database view which gets created when a resource gets sent to the
> datastore, there has been talk of removing the aliases as it does cause the
> conflicts you have encountered.
>
>
> On 26 November 2013 11:06, Athos Georgiou <ag242 at st-andrews.ac.uk> wrote:
>
>>  Hello,
>>
>>
>>
>> After having installed the datapusher found here:
>> http://datapusher.readthedocs.org/en/latest/install.html
>>
>> I have encountered an error upon creating a dataset by uploading a CSV
>> file. The error generated within ckan is
>>
>>
>>
>>  This resource can not be previewed at the moment. Click here for more
>> information.<http://researchdata.st-andrews.ac.uk/dataset/m12/resource/f4a0d5d6-526b-4419-92fa-4b4c31918ba0>
>>
>> Could not load preview: DataProxy returned an error (Data transformation
>> failed. HTTPException: Deadline exceeded while waiting for HTTP response
>> from URL:
>> http://researchdata.st-andrews.ac.uk/storage/f/2013-11-26T10%3A33%3A52.558Z/census-marriage-2.csv
>> )
>>
>>
>>
>> The Error log /var/log/apache2/datapusher.error.log shows:
>>
>>
>>
>>
>>
>> [Tue Nov 26 10:46:08 2013] [error]
>> /usr/lib/ckan/default/lib/python2.7/site-packages/sqlalchemy/engine/default.py:463:
>> SAWarning: Unicode type received non-unicode bind param value.
>>
>> [Tue Nov 26 10:46:08 2013] [error]
>> param.append(processors[key](compiled_params[key]))
>>
>> [Tue Nov 26 10:46:14 2013] [error] Fetching from:
>> http://researchdata.st-andrews.ac.uk/storage/f/2013-11-26T10%3A45%3A59.976Z/census-marriage-2.csv
>>
>> [Tue Nov 26 10:46:14 2013] [error] Deleting
>> "1f14b9e4-a645-4f0b-bba3-446297752dc8" from datastore.
>>
>> [Tue Nov 26 10:46:14 2013] [error] Determined headers and types:
>> [{'type': u'numeric', 'id': u'year'}, {'type': u'numeric', 'id': u'age'},
>> {'type': u'numeric', 'id': u'marst'}, {'type': u'numeric', 'id': u'sex'},
>> {'type': u'numeric', 'id': u'people'}]
>>
>> [Tue Nov 26 10:46:14 2013] [error] Saving chunk 0
>>
>> [Tue Nov 26 10:46:15 2013] [error] Job "push_to_datastore (trigger:
>> RunTriggerNow, run = True, next run at: None)" raised an exception
>>
>> [Tue Nov 26 10:46:15 2013] [error] Traceback (most recent call last):
>>
>> [Tue Nov 26 10:46:15 2013] [error]   File
>> "/usr/lib/ckan/default/lib/python2.7/site-packages/APScheduler-2.1.1-py2.7.egg/apscheduler/scheduler.py",
>> line 512, in _run_job
>>
>> [Tue Nov 26 10:46:15 2013] [error]     retval = job.func(*job.args,
>> **job.kwargs)
>>
>> [Tue Nov 26 10:46:15 2013] [error]   File
>> "/usr/lib/ckan/default/src/datapusher/datapusher/jobs.py", line 318, in
>> push_to_datastore
>>
>> [Tue Nov 26 10:46:15 2013] [error]     records, api_key, ckan_url)
>>
>> [Tue Nov 26 10:46:15 2013] [error]   File
>> "/usr/lib/ckan/default/src/datapusher/datapusher/jobs.py", line 152, in
>> send_resource_to_datastore
>>
>> [Tue Nov 26 10:46:15 2013] [error]     check_response(r, url, 'CKAN
>> DataStore')
>>
>> [Tue Nov 26 10:46:15 2013] [error]   File
>> "/usr/lib/ckan/default/src/datapusher/datapusher/jobs.py", line 84, in
>> check_response
>>
>> [Tue Nov 26 10:46:15 2013] [error]
>> resp=pprint.pformat(json_response)))
>>
>> [Tue Nov 26 10:46:15 2013] [error] JobError: CKAN DataStore bad response.
>> Status code: 409 Conflict. At:
>> http://researchdata.st-andrews.ac.uk/api/3/action/datastore_create.
>> Response: {u'error': {u'__type': u'Validation Error',
>>
>> [Tue Nov 26 10:46:15 2013] [error]             u'alias': [u'The alias
>> "census-marriage-2-csv" already exists.']},
>>
>> [Tue Nov 26 10:46:15 2013] [error]  u'help': u'Adds a new table to the
>> DataStore.\\n\\n    The datastore_create action allows you to post JSON
>> data to be\\n    stored against a resource. This endpoint also supports
>> altering tables,\\n    aliases and indexes and bulk insertion. This
>> endpoint can be called multiple\\n    times to initially insert more data,
>> add fields, change the aliases or indexes\\n    as well as the primary
>> keys.\\n\\n    See :ref:`fields` and :ref:`records` for details on how to
>> lay out records.\\n\\n    :param resource_id: resource id that the data is
>> going to be stored against.\\n    :type resource_id: string\\n    :param
>> aliases: names for read only aliases of the resource. (optional)\\n
>> :type aliases: list or comma separated string\\n    :param fields:
>> fields/columns and their extra metadata. (optional)\\n    :type fields:
>> list of dictionaries\\n    :param records: the data, eg: [{"dob": "2005",
>> "some_stuff": ["a", "b"]}]  (optional)\\n    :type records: list of
>> dictionaries\\n    :param primary_key: fields that represent a unique key
>> (optional)\\n    :type primary_key: list or comma separated string\\n
>> :param indexes: indexes on table (optional)\\n    :type indexes: list or
>> comma separated string\\n\\n    Please note that setting the ``aliases``,
>> ``indexes`` or ``primary_key`` replaces the exising\\n    aliases or
>> constraints. Setting ``records`` appends the provided records to the
>> resource.\\n\\n    **Results:**\\n\\n    :returns: The newly created data
>> object.\\n    :rtype: dictionary\\n\\n    See :ref:`fields` and
>> :ref:`records` for details on how to lay out records.\\n\\n    ',
>>
>> [Tue Nov 26 10:46:15 2013] [error]  u'success': False}
>>
>>
>>
>> Is the error log above any indication why the datapusher is not activated?
>>
>>
>>
>> Thanks
>>
>>
>>
>> *Athos Georgiou*
>>
>> *Application Developer - Research Computing *
>>
>> [image: Description: Description: Description: Description: Description:
>> Description: Description: cid:46E12C47-B748-4BCA-B41C-A94306C93BF6]
>>
>> *IT Services, University of St Andrews, St Andrews, Fife, KY16 9AL*
>>
>> *Tel: **01334 4644731   E-mail: ag242 at st-andrews.ac.uk
>> <ag242 at st-andrews.ac.uk>*
>>
>>
>>
>> The University of St Andrews is a charity registered in Scotland (No
>> SC013532)
>>
>>
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>
>>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20131126/26a86b7a/attachment-0003.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 13560 bytes
Desc: not available
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20131126/26a86b7a/attachment-0003.png>