[ckan-dev] Datastorer: celeryd fails on invalid resource URL
David Raznick
kindly at gmail.com
Fri May 4 21:57:11 UTC 2012
Hello,
Can you pull the latest ckanext-datastorer code and try again. This was
subtle as the code that tries to generate the url relies on the url you are
currently in. I tested it without the /en in the url and it worked.
Sorry for the pain with this. We have recently moved from having to
specify a site_url in your config
i.e ckan.site_url = http://vmubuntu.local:5000
to us trying to guess the site_url based on what the webserver sends and
this has thrown up many edge cases.
Hopefully this should be last of it. If not I may decide to go with the
site_url config if it is there otherwise fall back to this method, but I
was hoping not to have to do this.
Thanks
David
On Fri, May 4, 2012 at 5:22 PM, BIHANIC Laurent <laurent.bihanic at atos.net>wrote:
> Hi,
>
> Le 03/05/12 00:18, David Raznick a écrit :
> > I have added a patch to ckanext-archiver which could fix the issue. It
> appends
> > the site url to give a fully qualified that in link checkable.
>
> Still not good: the URL I get has twice the language:
>
> link_checker - Target URL:
>
> http://vmubuntu.local:5000/en/en/storage/f/2012-05-04T162011/perf-evaluation-2.csv
>
> It now passes the URL scheme test but fails upon actual connection (stack
> trace below).
>
> Regards,
>
> Laurent
>
>
> [2012-05-04 18:20:21,255: ERROR/MainProcess] Task
> datastorer.upload[c50f7e0f-c2b2-488c-a07e-1eaf9bf5227f] raised exception:
> LinkCheckerError('URL unobtainable',)
> Traceback (most recent call last):
> File
>
> "/home/lbihanic/work/ckan/test-latest/lib/python2.6/site-packages/celery/execute/trace.py",
> line 181, in trace_task
> R = retval = fun(*args, **kwargs)
> File
> "/home/lbihanic/work/ckan/ckanext-datastorer/ckanext/datastorer/tasks.py",
> line 73, in datastorer_upload
> return _datastorer_upload(context, data)
> File
> "/home/lbihanic/work/ckan/ckanext-datastorer/ckanext/datastorer/tasks.py",
> line 90, in _datastorer_upload
> result = download(context, resource, data_formats=DATA_FORMATS)
> File
>
> "/home/lbihanic/work/ckan/test-latest/src/ckanext-archiver/ckanext/archiver/tasks.py",
> line 85, in download
> headers = json.loads(link_checker(link_context, link_data))
> File
>
> "/home/lbihanic/work/ckan/test-latest/lib/python2.6/site-packages/celery/app/task/__init__.py",
> line 262, in __call__
> return self.run(*args, **kwargs)
> File
>
> "/home/lbihanic/work/ckan/test-latest/src/ckanext-archiver/ckanext/archiver/tasks.py",
> line 272, in link_checker
> raise LinkCheckerError(error_message)
> LinkCheckerError: URL unobtainable
>
>
> > On Wed, May 2, 2012 at 5:18 PM, BIHANIC Laurent <
> laurent.bihanic at atos.net
> > <mailto:laurent.bihanic at atos.net>> wrote:
> >
> > Me again, the HTTP proxy configuration removed...
> >
> > I installed CKAN and Datastorer from the sources (latest).
> > When adding a new resource to a dataset, celeryd fails with the
> following
> > error:
> >
> > [2012-05-02 17:37:20,909: ERROR/MainProcess] Task
> > datastorer.upload[205b2be6-3188-4694-bac1-c01492046c72] raised
> exception:
> > LinkCheckerError('Invalid url scheme',)
> >
> > I added a debug trace and the URL being checked is
> > "/en/storage/f/2012-05-02T153710/perf-evaluation-2.csv".
> > But link_checker() expects an absolute URL with a scheme (http,
> https or ftp).
> >
> > Any known fix ?
> >
> > TIA,
> >
> > Laurent
> >
> >
> > Traceback (most recent call last):
> > File
> >
> "/home/lbihanic/work/ckan/test-latest/lib/python2.6/site-packages/celery/execute/trace.py",
> > line 181, in trace_task
> > R = retval = fun(*args, **kwargs)
> > File
> >
> "/home/lbihanic/work/ckan/ckanext-datastorer/ckanext/datastorer/tasks.py",
> > line 73, in datastorer_upload
> > return _datastorer_upload(context, data)
> > File
> >
> "/home/lbihanic/work/ckan/ckanext-datastorer/ckanext/datastorer/tasks.py",
> > line 90, in _datastorer_upload
> > result = download(context, resource, data_formats=DATA_FORMATS)
> > File
> >
> "/home/lbihanic/work/ckan/test-latest/src/ckanext-archiver/ckanext/archiver/tasks.py",
> > line 78, in download
> > headers = json.loads(link_checker(link_context, link_data))
> > File
> >
> "/home/lbihanic/work/ckan/test-latest/lib/python2.6/site-packages/celery/app/task/__init__.py",
> > line 262, in __call__
> > return self.run(*args, **kwargs)
> > File
> >
> "/home/lbihanic/work/ckan/test-latest/src/ckanext-archiver/ckanext/archiver/tasks.py",
> > line 246, in link_checker
> > raise LinkCheckerError("Invalid url scheme")
> > LinkCheckerError: Invalid url scheme
> > ________________________________
> >
> >
> > Ce message et les pièces jointes sont confidentiels et réservés à
> l'usage
> > exclusif de ses destinataires. Il peut également être protégé par le
> > secret professionnel. Si vous recevez ce message par erreur, merci
> d'en
> > avertir immédiatement l'expéditeur et de le détruire. L'intégrité du
> > message ne pouvant être assurée sur Internet, la responsabilité du
> groupe
> > Atos ne pourra être engagée quant au contenu de ce message. Bien que
> les
> > meilleurs efforts soient faits pour maintenir cette transmission
> exempte
> > de tout virus, l'expéditeur ne donne aucune garantie à cet égard et
> sa
> > responsabilité ne saurait être engagée pour tout dommage résultant
> d'un
> > virus transmis.
> >
> > This e-mail and the documents attached are confidential and intended
> > solely for the addressee; it may also be privileged. If you receive
> this
> > e-mail in error, please notify the sender immediately and destroy
> it. As
> > its integrity cannot be secured on the Internet, the Atos group
> liability
> > cannot be triggered for the message content. Although the sender
> endeavors
> > to maintain a computer virus-free network, the sender does not
> warrant
> > that this transmission is virus-free and will not be liable for any
> > damages resulting from any virus transmitted.
> >
> > _______________________________________________
> > ckan-dev mailing list
> > ckan-dev at lists.okfn.org <mailto:ckan-dev at lists.okfn.org>
> > http://lists.okfn.org/mailman/listinfo/ckan-dev
> >
> >
> >
> >
> > _______________________________________________
> > ckan-dev mailing list
> > ckan-dev at lists.okfn.org
> > http://lists.okfn.org/mailman/listinfo/ckan-dev
>
> --
> wWw
> (o o)
> -------------ooO-(_)-Ooo-----------------------------------------------
> Laurent Bihanic | Tel: +33 (0)1 73 26 12 92
> Atos Systems Integration | GSM: +33 (0)6 25 73 03 74
> 80 quai Voltaire |
> F-95870 Bezons | mailto: laurent.bihanic at atos.net
> -----------------------------------------------------------------------
> Very funny, Scotty. Now beam down my clothes.
>
>
> Please consider your environmental responsibility before printing
> this e-mail.
>
> DISCLAIMER:
> The opinions expressed are entirely my own and may not necessarily be
> those of my employer. Also, I am not now nor have I ever been a
> lawyer. My opinions are provided as-is with absolutely no warranty of
> merchantability or fitness for any particular use. Besides, you can't
> prove I typed this. No body saw me type this. Who says I typed this?
> ________________________________
>
>
> Ce message et les pièces jointes sont confidentiels et réservés à l'usage
> exclusif de ses destinataires. Il peut également être protégé par le secret
> professionnel. Si vous recevez ce message par erreur, merci d'en avertir
> immédiatement l'expéditeur et de le détruire. L'intégrité du message ne
> pouvant être assurée sur Internet, la responsabilité du groupe Atos ne
> pourra être engagée quant au contenu de ce message. Bien que les meilleurs
> efforts soient faits pour maintenir cette transmission exempte de tout
> virus, l'expéditeur ne donne aucune garantie à cet égard et sa
> responsabilité ne saurait être engagée pour tout dommage résultant d'un
> virus transmis.
>
> This e-mail and the documents attached are confidential and intended
> solely for the addressee; it may also be privileged. If you receive this
> e-mail in error, please notify the sender immediately and destroy it. As
> its integrity cannot be secured on the Internet, the Atos group liability
> cannot be triggered for the message content. Although the sender endeavors
> to maintain a computer virus-free network, the sender does not warrant that
> this transmission is virus-free and will not be liable for any damages
> resulting from any virus transmitted.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20120504/a842c194/attachment-0001.html>
More information about the ckan-dev
mailing list