[ckan-dev] Datastorer: celeryd fails on invalid resource URL
BIHANIC Laurent
laurent.bihanic at atos.net
Tue May 15 14:44:15 UTC 2012
Hi David,
I totally forgot to tell you it now works fine.
Thanks a lot,
Laurent
Le 04/05/12 23:57, David Raznick a écrit :
> Hello,
>
> Can you pull the latest ckanext-datastorer code and try again. This was subtle
> as the code that tries to generate the url relies on the url you are currently
> in. I tested it without the /en in the url and it worked.
>
> Sorry for the pain with this. We have recently moved from having to specify a
> site_url in your config
>
> i.e ckan.site_url = http://vmubuntu.local:5000
>
> to us trying to guess the site_url based on what the webserver sends and this
> has thrown up many edge cases.
>
> Hopefully this should be last of it. If not I may decide to go with the
> site_url config if it is there otherwise fall back to this method, but I was
> hoping not to have to do this.
>
> Thanks
>
> David
>
>
> On Fri, May 4, 2012 at 5:22 PM, BIHANIC Laurent <laurent.bihanic at atos.net
> <mailto:laurent.bihanic at atos.net>> wrote:
>
> Hi,
>
> Le 03/05/12 00:18, David Raznick a écrit :
> > I have added a patch to ckanext-archiver which could fix the issue. It
> appends
> > the site url to give a fully qualified that in link checkable.
>
> Still not good: the URL I get has twice the language:
>
> link_checker - Target URL:
> http://vmubuntu.local:5000/en/en/storage/f/2012-05-04T162011/perf-evaluation-2.csv
>
> It now passes the URL scheme test but fails upon actual connection (stack
> trace below).
>
> Regards,
>
> Laurent
>
>
> [2012-05-04 18:20:21,255: ERROR/MainProcess] Task
> datastorer.upload[c50f7e0f-c2b2-488c-a07e-1eaf9bf5227f] raised exception:
> LinkCheckerError('URL unobtainable',)
> Traceback (most recent call last):
> File
> "/home/lbihanic/work/ckan/test-latest/lib/python2.6/site-packages/celery/execute/trace.py",
> line 181, in trace_task
> R = retval = fun(*args, **kwargs)
> File
> "/home/lbihanic/work/ckan/ckanext-datastorer/ckanext/datastorer/tasks.py",
> line 73, in datastorer_upload
> return _datastorer_upload(context, data)
> File
> "/home/lbihanic/work/ckan/ckanext-datastorer/ckanext/datastorer/tasks.py",
> line 90, in _datastorer_upload
> result = download(context, resource, data_formats=DATA_FORMATS)
> File
> "/home/lbihanic/work/ckan/test-latest/src/ckanext-archiver/ckanext/archiver/tasks.py",
> line 85, in download
> headers = json.loads(link_checker(link_context, link_data))
> File
> "/home/lbihanic/work/ckan/test-latest/lib/python2.6/site-packages/celery/app/task/__init__.py",
> line 262, in __call__
> return self.run(*args, **kwargs)
> File
> "/home/lbihanic/work/ckan/test-latest/src/ckanext-archiver/ckanext/archiver/tasks.py",
> line 272, in link_checker
> raise LinkCheckerError(error_message)
> LinkCheckerError: URL unobtainable
>
>
> > On Wed, May 2, 2012 at 5:18 PM, BIHANIC Laurent
> <laurent.bihanic at atos.net <mailto:laurent.bihanic at atos.net>
> > <mailto:laurent.bihanic at atos.net <mailto:laurent.bihanic at atos.net>>> wrote:
> >
> > Me again, the HTTP proxy configuration removed...
> >
> > I installed CKAN and Datastorer from the sources (latest).
> > When adding a new resource to a dataset, celeryd fails with the following
> > error:
> >
> > [2012-05-02 17:37:20,909: ERROR/MainProcess] Task
> > datastorer.upload[205b2be6-3188-4694-bac1-c01492046c72] raised exception:
> > LinkCheckerError('Invalid url scheme',)
> >
> > I added a debug trace and the URL being checked is
> > "/en/storage/f/2012-05-02T153710/perf-evaluation-2.csv".
> > But link_checker() expects an absolute URL with a scheme (http, https
> or ftp).
> >
> > Any known fix ?
> >
> > TIA,
> >
> > Laurent
> >
> >
> > Traceback (most recent call last):
> > File
> >
> "/home/lbihanic/work/ckan/test-latest/lib/python2.6/site-packages/celery/execute/trace.py",
> > line 181, in trace_task
> > R = retval = fun(*args, **kwargs)
> > File
> > "/home/lbihanic/work/ckan/ckanext-datastorer/ckanext/datastorer/tasks.py",
> > line 73, in datastorer_upload
> > return _datastorer_upload(context, data)
> > File
> > "/home/lbihanic/work/ckan/ckanext-datastorer/ckanext/datastorer/tasks.py",
> > line 90, in _datastorer_upload
> > result = download(context, resource, data_formats=DATA_FORMATS)
> > File
> >
> "/home/lbihanic/work/ckan/test-latest/src/ckanext-archiver/ckanext/archiver/tasks.py",
> > line 78, in download
> > headers = json.loads(link_checker(link_context, link_data))
> > File
> >
> "/home/lbihanic/work/ckan/test-latest/lib/python2.6/site-packages/celery/app/task/__init__.py",
> > line 262, in __call__
> > return self.run(*args, **kwargs)
> > File
> >
> "/home/lbihanic/work/ckan/test-latest/src/ckanext-archiver/ckanext/archiver/tasks.py",
> > line 246, in link_checker
> > raise LinkCheckerError("Invalid url scheme")
> > LinkCheckerError: Invalid url scheme
> > ________________________________
> >
> >
> > Ce message et les pièces jointes sont confidentiels et réservés à l'usage
> > exclusif de ses destinataires. Il peut également être protégé par le
> > secret professionnel. Si vous recevez ce message par erreur, merci d'en
> > avertir immédiatement l'expéditeur et de le détruire. L'intégrité du
> > message ne pouvant être assurée sur Internet, la responsabilité du groupe
> > Atos ne pourra être engagée quant au contenu de ce message. Bien que les
> > meilleurs efforts soient faits pour maintenir cette transmission exempte
> > de tout virus, l'expéditeur ne donne aucune garantie à cet égard et sa
> > responsabilité ne saurait être engagée pour tout dommage résultant d'un
> > virus transmis.
> >
> > This e-mail and the documents attached are confidential and intended
> > solely for the addressee; it may also be privileged. If you receive this
> > e-mail in error, please notify the sender immediately and destroy it. As
> > its integrity cannot be secured on the Internet, the Atos group liability
> > cannot be triggered for the message content. Although the sender endeavors
> > to maintain a computer virus-free network, the sender does not warrant
> > that this transmission is virus-free and will not be liable for any
> > damages resulting from any virus transmitted.
> >
> > _______________________________________________
> > ckan-dev mailing list
> > ckan-dev at lists.okfn.org <mailto:ckan-dev at lists.okfn.org>
> <mailto:ckan-dev at lists.okfn.org <mailto:ckan-dev at lists.okfn.org>>
> > http://lists.okfn.org/mailman/listinfo/ckan-dev
> >
> >
> >
> >
> > _______________________________________________
> > ckan-dev mailing list
> > ckan-dev at lists.okfn.org <mailto:ckan-dev at lists.okfn.org>
> > http://lists.okfn.org/mailman/listinfo/ckan-dev
>
> --
> wWw
> (o o)
> -------------ooO-(_)-Ooo-----------------------------------------------
> Laurent Bihanic | Tel: +33 (0)1 73 26 12 92
> <tel:%2B33%20%280%291%2073%2026%2012%2092>
> Atos Systems Integration | GSM: +33 (0)6 25 73 03 74
> <tel:%2B33%20%280%296%2025%2073%2003%2074>
> 80 quai Voltaire |
> F-95870 Bezons | mailto: laurent.bihanic at atos.net
> <mailto:laurent.bihanic at atos.net>
> -----------------------------------------------------------------------
> Very funny, Scotty. Now beam down my clothes.
>
>
> Please consider your environmental responsibility before printing
> this e-mail.
>
> DISCLAIMER:
> The opinions expressed are entirely my own and may not necessarily be
> those of my employer. Also, I am not now nor have I ever been a
> lawyer. My opinions are provided as-is with absolutely no warranty of
> merchantability or fitness for any particular use. Besides, you can't
> prove I typed this. No body saw me type this. Who says I typed this?
> ________________________________
>
>
> Ce message et les pièces jointes sont confidentiels et réservés à l'usage
> exclusif de ses destinataires. Il peut également être protégé par le
> secret professionnel. Si vous recevez ce message par erreur, merci d'en
> avertir immédiatement l'expéditeur et de le détruire. L'intégrité du
> message ne pouvant être assurée sur Internet, la responsabilité du groupe
> Atos ne pourra être engagée quant au contenu de ce message. Bien que les
> meilleurs efforts soient faits pour maintenir cette transmission exempte
> de tout virus, l'expéditeur ne donne aucune garantie à cet égard et sa
> responsabilité ne saurait être engagée pour tout dommage résultant d'un
> virus transmis.
>
> This e-mail and the documents attached are confidential and intended
> solely for the addressee; it may also be privileged. If you receive this
> e-mail in error, please notify the sender immediately and destroy it. As
> its integrity cannot be secured on the Internet, the Atos group liability
> cannot be triggered for the message content. Although the sender endeavors
> to maintain a computer virus-free network, the sender does not warrant
> that this transmission is virus-free and will not be liable for any
> damages resulting from any virus transmitted.
>
>
--
wWw
(o o)
-------------ooO-(_)-Ooo-----------------------------------------------
Laurent Bihanic | Tel: +33 (0)1 73 26 12 92
Atos Systems Integration | GSM: +33 (0)6 25 73 03 74
80 quai Voltaire |
F-95870 Bezons | mailto: laurent.bihanic at atos.net
-----------------------------------------------------------------------
Very funny, Scotty. Now beam down my clothes.
Please consider your environmental responsibility before printing
this e-mail.
DISCLAIMER:
The opinions expressed are entirely my own and may not necessarily be
those of my employer. Also, I am not now nor have I ever been a
lawyer. My opinions are provided as-is with absolutely no warranty of
merchantability or fitness for any particular use. Besides, you can't
prove I typed this. No body saw me type this. Who says I typed this?
________________________________
Ce message et les pièces jointes sont confidentiels et réservés à l'usage exclusif de ses destinataires. Il peut également être protégé par le secret professionnel. Si vous recevez ce message par erreur, merci d'en avertir immédiatement l'expéditeur et de le détruire. L'intégrité du message ne pouvant être assurée sur Internet, la responsabilité du groupe Atos ne pourra être engagée quant au contenu de ce message. Bien que les meilleurs efforts soient faits pour maintenir cette transmission exempte de tout virus, l'expéditeur ne donne aucune garantie à cet égard et sa responsabilité ne saurait être engagée pour tout dommage résultant d'un virus transmis.
This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Atos group liability cannot be triggered for the message content. Although the sender endeavors to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted.
More information about the ckan-dev
mailing list