[ckan-dev] Datastorer: celeryd fails on invalid resource URL

[David Raznick]

Hello

I have added a patch to ckanext-archiver which could fix the issue.  It
appends the site url to give a fully qualified that in link checkable.

We have never really tested the datastorer against local storage and so we
missed this.

Thanks

David





On Wed, May 2, 2012 at 5:18 PM, BIHANIC Laurent <laurent.bihanic at atos.net>wrote:

> Me again, the HTTP proxy configuration removed...
>
> I installed CKAN and Datastorer from the sources (latest).
> When adding a new resource to a dataset, celeryd fails with the following
> error:
>
> [2012-05-02 17:37:20,909: ERROR/MainProcess] Task
> datastorer.upload[205b2be6-3188-4694-bac1-c01492046c72] raised exception:
> LinkCheckerError('Invalid url scheme',)
>
> I added a debug trace and the URL being checked is
> "/en/storage/f/2012-05-02T153710/perf-evaluation-2.csv".
> But link_checker() expects an absolute URL with a scheme (http, https or
> ftp).
>
> Any known fix ?
>
> TIA,
>
> Laurent
>
>
> Traceback (most recent call last):
>   File
>
> "/home/lbihanic/work/ckan/test-latest/lib/python2.6/site-packages/celery/execute/trace.py",
> line 181, in trace_task
>     R = retval = fun(*args, **kwargs)
>   File
> "/home/lbihanic/work/ckan/ckanext-datastorer/ckanext/datastorer/tasks.py",
> line 73, in datastorer_upload
>     return _datastorer_upload(context, data)
>   File
> "/home/lbihanic/work/ckan/ckanext-datastorer/ckanext/datastorer/tasks.py",
> line 90, in _datastorer_upload
>     result = download(context, resource, data_formats=DATA_FORMATS)
>   File
>
> "/home/lbihanic/work/ckan/test-latest/src/ckanext-archiver/ckanext/archiver/tasks.py",
> line 78, in download
>     headers = json.loads(link_checker(link_context, link_data))
>   File
>
> "/home/lbihanic/work/ckan/test-latest/lib/python2.6/site-packages/celery/app/task/__init__.py",
> line 262, in __call__
>     return self.run(*args, **kwargs)
>   File
>
> "/home/lbihanic/work/ckan/test-latest/src/ckanext-archiver/ckanext/archiver/tasks.py",
> line 246, in link_checker
>     raise LinkCheckerError("Invalid url scheme")
> LinkCheckerError: Invalid url scheme
> ________________________________
>
>
> Ce message et les pièces jointes sont confidentiels et réservés à l'usage
> exclusif de ses destinataires. Il peut également être protégé par le secret
> professionnel. Si vous recevez ce message par erreur, merci d'en avertir
> immédiatement l'expéditeur et de le détruire. L'intégrité du message ne
> pouvant être assurée sur Internet, la responsabilité du groupe Atos ne
> pourra être engagée quant au contenu de ce message. Bien que les meilleurs
> efforts soient faits pour maintenir cette transmission exempte de tout
> virus, l'expéditeur ne donne aucune garantie à cet égard et sa
> responsabilité ne saurait être engagée pour tout dommage résultant d'un
> virus transmis.
>
> This e-mail and the documents attached are confidential and intended
> solely for the addressee; it may also be privileged. If you receive this
> e-mail in error, please notify the sender immediately and destroy it. As
> its integrity cannot be secured on the Internet, the Atos group liability
> cannot be triggered for the message content. Although the sender endeavors
> to maintain a computer virus-free network, the sender does not warrant that
> this transmission is virus-free and will not be liable for any damages
> resulting from any virus transmitted.
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20120502/f6698917/attachment-0001.html>