[ckan-dev] basic question on configuring SSL with CKAN

Mon Nov 13 18:08:17 UTC 2017

Hi Stefan,

On Tue, Nov 7, 2017 at 5:13 AM, Stefan Oderbolz <stefan.oderbolz at liip.ch>
wrote:

> Why do you suspect the python version?
>

*I suspected*, because when I force to upload a CSV file via web interface,
I eventually get the error "Process completed but unable to post to
result_url" then, when I hit "send data to datastore" multiple times, the
error changes to "File "/usr/lib/ckan/datapusher/lib/
python2.7/site-packages/apscheduler/scheduler.py", line 512, in _run_job
retval = job.func(*job.args, **job.kwargs) File
"/usr/lib/ckan/datapusher/src/datapusher/datapusher/jobs.py", line 300, in
push_to_datastore resource = get_resource(resource_id, ckan_url, api_key)
File "/usr/lib/ckan/datapusher/src/datapusher/datapusher/jobs.py", line
250, in get_resource 'Authorization': api_key} File
"/usr/lib/ckan/datapusher/lib/python2.7/site-packages/requests/api.py",
line 110, in post return request('post', url, data=data, json=json,
**kwargs) File "/usr/lib/ckan/datapusher/lib/python2.7/site-packages/requests/api.py",
line 56, in request return session.request(method=method, url=url,
**kwargs) File "/usr/lib/ckan/datapusher/lib/python2.7/site-packages/requests/sessions.py",
line 488, in request resp = self.send(prep, **send_kwargs) File
"/usr/lib/ckan/datapusher/lib/python2.7/site-packages/requests/sessions.py",
line 609, in send r = adapter.send(request, **kwargs) File
"/usr/lib/ckan/datapusher/lib/python2.7/site-packages/requests/adapters.py",
line 497, in send raise SSLError(e, request=request)
SSLError(SSLError(SSLError("bad handshake: Error([('SSL routines',
'tls_process_server_certificate', 'certificate verify failed')],)",),),)"

> If there is a problem with the SSL_VERIFY I strongly suspect datapusher.
> And as there have been some fixes for this functionality just recently, I'd
> recommend to use the latest version of datapusher.
> The SSL_VERIFY feature is available since datapusher 0.0.11 and has been
> improved in both 0.0.12 and 0.0.13.
>
> To check which version you have, open the following file:
> /usr/lib/ckan/datapusher/src/datapusher/datapusher/__init__.py
>
I found the version at file  /usr/lib/ckan/datapusher/src/datapusher/
datapusher.egg-info/PKG-INFO,* it says Version: 0.0.8*
*I will try reinstall manually the plugin*

> I quickly checked the latest 2.6 deb package on http://packaging.ckan.org/
> and they include datapusher 0.0.12, so I guess this is what you have
> installed.
>
>
> You wrote
> > But the changes looks like are not being reflected
>
> Are any values from the datapusher_settings.py reflected? Or is just the
> SSL_VERIFY not working? Note that only with datapusher 0.0.13 the
> SSL_VERIFY is applied to download the remote file. Before SSL_VERIFY is
> only used to disable the SSL check for the CKAN API calls.
>
I will try the manual installation.

>
> - Stefan
>
>
>
> On Mon, Nov 6, 2017 at 12:05 PM, gustavo w <gustavo.waku at gmail.com> wrote:
>
>> thanks for the tip.
>>
>> But I suspect it's the python version. I am using python 2.7.6
>> Does it work good on python 2.7.9 ?
>> Which python version do you recommend?
>>
>> thanks a lot,
>> gustavo
>>
>> On Tue, Oct 31, 2017 at 1:15 AM, Stefan Oderbolz <stefan.oderbolz at liip.ch
>> > wrote:
>>
>>> I'm not sure what version of datapusher is being installed when using
>>> the package installation of CKAN, as usually you need to install this
>>> seperately. But I don't have any experience with the deb package.
>>>
>>> Can you try to manually install the latest version of datapusher from
>>> https://github.com/ckan/datapusher ?
>>>
>>> On Oct 26, 2017 19:25, "gustavo w" <gustavo.waku at gmail.com> wrote:
>>>
>>>> Hello folks,
>>>>
>>>> Is there any detailed guide to disable the SSL_VERIFY in the
>>>> datapusher_settings.py?
>>>> I modified it, rebooted the server, but it does not reflect my change.
>>>> Any idea whats going on? I tried in 2.6.2 and 2.7.2
>>>>
>>>> thanks,
>>>> gustavo
>>>>
>>>>
>>>> On Wed, Oct 25, 2017 at 2:17 PM, gustavo w <gustavo.waku at gmail.com>
>>>> wrote:
>>>>
>>>>> I did install the ckan as a package install (.deb), version 2.6.2
>>>>> I do not know how to check the datapusher version.
>>>>> thx
>>>>>
>>>>> On Wed, Oct 25, 2017 at 11:44 AM, Stefan Oderbolz <
>>>>> stefan.oderbolz at liip.ch> wrote:
>>>>>
>>>>>> What version of datapusher do you use? How did you install it?
>>>>>>
>>>>>> On Wed, Oct 25, 2017 at 3:15 PM, gustavo w <gustavo.waku at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks Stefan,
>>>>>>>
>>>>>>> I just modified the files at my machine, adding the SSL_VERIFY =
>>>>>>> 'False'
>>>>>>> *./etc/ckan/datapusher_settings.py*
>>>>>>>
>>>>>>> *./usr/lib/ckan/datapusher/src/datapusher/deployment/datapusher_settings.py*
>>>>>>>
>>>>>>> *the file looks like this:*
>>>>>>> *-------*
>>>>>>> import uuid
>>>>>>>
>>>>>>> DEBUG = False
>>>>>>> TESTING = False
>>>>>>> SECRET_KEY = str(uuid.uuid4())
>>>>>>> USERNAME = str(uuid.uuid4())
>>>>>>> PASSWORD = str(uuid.uuid4())
>>>>>>>
>>>>>>> NAME = 'datapusher'
>>>>>>>
>>>>>>> # database
>>>>>>>
>>>>>>> SQLALCHEMY_DATABASE_URI = 'sqlite:////tmp/job_store.db'
>>>>>>>
>>>>>>> # webserver host and port
>>>>>>>
>>>>>>> HOST = '0.0.0.0'
>>>>>>> PORT = 8800
>>>>>>>
>>>>>>> # logging
>>>>>>>
>>>>>>> #FROM_EMAIL = 'server-error at example.com'
>>>>>>> #ADMINS = ['yourname at example.com']  # where to send emails
>>>>>>>
>>>>>>> #LOG_FILE = '/tmp/ckan_service.log'
>>>>>>> STDERR = True
>>>>>>>
>>>>>>> SSL_VERIFY = 'False'
>>>>>>> *-------*
>>>>>>>
>>>>>>> *But the changes looks like are not being reflected. I have rebooted
>>>>>>> the machine, but still nothing changes...*
>>>>>>> *Do I have to do anything else?*
>>>>>>>
>>>>>>> thanks a lot,
>>>>>>> Gustavo
>>>>>>>
>>>>>>> On Wed, Oct 25, 2017 at 9:48 AM, Stefan Oderbolz <
>>>>>>> stefan.oderbolz at liip.ch> wrote:
>>>>>>>
>>>>>>>> Hi Gustavo,
>>>>>>>>
>>>>>>>> there was just recently a bugfix for this issue:
>>>>>>>> https://github.com/ckan/datapusher/issues/149
>>>>>>>> So you could either update to the latest version of datapusher or
>>>>>>>> try to set SSL_VERIFY to 'False' (including the quotes).
>>>>>>>>
>>>>>>>> - Stefan
>>>>>>>>
>>>>>>>> On Tue, Oct 24, 2017 at 6:57 PM, gustavo w <gustavo.waku at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I installed ckan 2.6.2, and its working fine (via deb package),
>>>>>>>>> with data explorer ok.
>>>>>>>>> after I set the SSL protocol using the tutorial available at: (
>>>>>>>>> https://github.com/ckan/ckan/wiki/SSL), whenever I upload a new
>>>>>>>>> CSV file, the data is not displayed anymore into the dataexplorer.
>>>>>>>>>
>>>>>>>>> When I force the datastore to process the CSV file, I get the
>>>>>>>>> following error:
>>>>>>>>>
>>>>>>>>> "Erro: File "/usr/lib/ckan/datapusher/lib/
>>>>>>>>> python2.7/site-packages/apscheduler/scheduler.py", line 512, in
>>>>>>>>> _run_job retval = job.func(*job.args, **job.kwargs)
>>>>>>>>> File "/usr/lib/ckan/datapusher/src/datapusher/datapusher/jobs.py",
>>>>>>>>> line 300, in push_to_datastore resource = get_resource(resource_id,
>>>>>>>>> ckan_url, api_key)
>>>>>>>>> File "/usr/lib/ckan/datapusher/src/datapusher/datapusher/jobs.py",
>>>>>>>>> line 250, in get_resource 'Authorization': api_key}
>>>>>>>>> File "/usr/lib/ckan/datapusher/lib/python2.7/site-packages/requests/api.py",
>>>>>>>>> line 110, in post return request('post', url, data=data, json=json,
>>>>>>>>> **kwargs)
>>>>>>>>> File "/usr/lib/ckan/datapusher/lib/python2.7/site-packages/requests/api.py",
>>>>>>>>> line 56, in request return session.request(method=method, url=url,
>>>>>>>>> **kwargs)
>>>>>>>>> File "/usr/lib/ckan/datapusher/lib/python2.7/site-packages/requests/sessions.py",
>>>>>>>>> line 488, in request resp = self.send(prep, **send_kwargs)
>>>>>>>>> File "/usr/lib/ckan/datapusher/lib/python2.7/site-packages/requests/sessions.py",
>>>>>>>>> line 609, in send r = adapter.send(request, **kwargs)
>>>>>>>>> File "/usr/lib/ckan/datapusher/lib/python2.7/site-packages/requests/adapters.py",
>>>>>>>>> line 497, in send raise SSLError(e, request=request)
>>>>>>>>> SSLError(SSLError(SSLError(1, '_ssl.c:510: error:14090086:SSL
>>>>>>>>> routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify
>>>>>>>>> failed'),),)"
>>>>>>>>>
>>>>>>>>> Looks like it's an error with python lib SSL protocol.
>>>>>>>>> I was instructed to follow the tutorial at:
>>>>>>>>> http://docs.ckan.org/projects/datapusher/en/latest/using
>>>>>>>>> .html#configuring-ssl-verification
>>>>>>>>>
>>>>>>>>> the tutorial says:
>>>>>>>>> *"If you still have problems verifying certificates, or maybe for
>>>>>>>>> test purposes, you can switch the verification off in
>>>>>>>>> datapusher_settings.py:*
>>>>>>>>> *SSL_VERIFY = False"*
>>>>>>>>>
>>>>>>>>> *I set SSL_VERIFY = False in datapusher_settings.py*
>>>>>>>>> *but the server does not reflect this change.*
>>>>>>>>>
>>>>>>>>> *I found two files like this in my machine.*
>>>>>>>>> *./etc/ckan/datapusher_settings.py*
>>>>>>>>>
>>>>>>>>> *./usr/lib/ckan/datapusher/src/datapusher/deployment/datapusher_settings.py*
>>>>>>>>>
>>>>>>>>> *I changed both and rebooted the machine.*
>>>>>>>>> *somehow it does not work.*
>>>>>>>>> *Do I have to recompile somehow the file?*
>>>>>>>>>
>>>>>>>>> *Can anyone give me directions on how to set the SSL_VERIFY flag
>>>>>>>>> into the datapusher_settings? do I have to recompile anything else?*
>>>>>>>>>
>>>>>>>>> *Thanks a lot,*
>>>>>>>>> *Gustavo*
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> ckan-dev mailing list
>>>>>>>>> ckan-dev at lists.okfn.org
>>>>>>>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>>>>>>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Liip AG  // Limmatstrasse 183 //  CH-8005 Zürich
>>>>>>>> Tel +41 43 500 39 80 <+41%2043%20500%2039%2080> // GnuPG
>>>>>>>> 0x7B588C67 // www.liip.ch
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> ckan-dev mailing list
>>>>>>>> ckan-dev at lists.okfn.org
>>>>>>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>>>>>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> ckan-dev mailing list
>>>>>>> ckan-dev at lists.okfn.org
>>>>>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>>>>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Liip AG  // Limmatstrasse 183 //  CH-8005 Zürich
>>>>>> Tel +41 43 500 39 80 <+41%2043%20500%2039%2080> // GnuPG 0x7B588C67
>>>>>> // www.liip.ch
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> ckan-dev mailing list
>>>>>> ckan-dev at lists.okfn.org
>>>>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>>>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>>>>>
>>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> ckan-dev mailing list
>>>> ckan-dev at lists.okfn.org
>>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>>>
>>>>
>>> _______________________________________________
>>> ckan-dev mailing list
>>> ckan-dev at lists.okfn.org
>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>>
>>>
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>
>>
>
>
> --
> Liip AG  // Limmatstrasse 183 //  CH-8005 Zürich
> Tel +41 43 500 39 80 <+41%2043%20500%2039%2080> // GnuPG 0x7B588C67 //
> www.liip.ch
>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20171113/c5afab4e/attachment-0003.html>