[ckan-dev] CORS issue whe using CKAN API for creating resource
Matthew Fullerton
matthew at smartlane.de
Fri Apr 29 17:00:24 UTC 2016
Its such a coincidence that you are hitting this problem. I just two days ago learned that nginx can't add headers with non-successful HTTP responses (you should have seen my face, it was very sad). I wonder if Apache has the same issue?
Best,
Matt
---------------------------------------------------------------------
Matthew Fullerton
Smartlane UG (haftungsbeschränkt)
Metzstraße 29a
81667 München
matthew at smartlane.de
T +49.176.5789.6501
F +49.89.9041.1930.2
www.smartlane.de<http://www.smartlane.de/>
---------------------------------------------------------------------
Geschäftsführer: Dr. Mathias Baur, Florian Schimandl
Unternehmenssitz: München
Handelsregister beim Amtsgericht: München
Handelsregister-Nummer: HRB 220483
USt. ID-Nr.: DE301856148
Finanzamt: München
---------------------------------------------------------------------
________________________________
Von: ckan-dev <ckan-dev-bounces at lists.okfn.org> im Auftrag von lucia.espona at wsl.ch <lucia.espona at wsl.ch>
Gesendet: Freitag, 29. April 2016 15:26
An: CKAN Development Discussions
Betreff: Re: [ckan-dev] CORS issue whe using CKAN API for creating resource
Dear Matthew
I am still waiting for my server admin to install NGINX but I easily managed to add the CORS headers through the Apache configuration as ypu suggested. I included below the added lines to ckan_default.conf in case someone is interested.
Also, I had to set up back to 'false' the ckan.cors.origin_allow_all in the CKAN config file because otherwise they get added again by CKAN and I get the error:
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://ckan.wsl.ch/api/action/package_show?name_or_id=10-9999-184. (Reason: CORS header 'Access-Control-Allow-Origin' does not match '*, *').
For what I have read, it is not allowed to specify multiple comma-separated values in the Access-Control-Allow-Origin header, it has to be a single value or regular expression. Does CKAN add the header after the Apache or the other way around? I am doing a "SET header" instead "ADD header" so I guess that appart from setting CORS in CKAN to false there is nothing I can do.
These two files that raised the initial CORS issue, cause a system error in CKAN because now I get the error response. Something may fail in the resource creation process and the answer does not get the CORS headers added, as it properly happens with the successfully uploaded files. If someone is interested in looking into this I can provide more details.
One of the problematic files is a jpg image, the other is an xml with a typo in one of the tags, if I correct the typo the file uploads without any issue through the API.
Is this the expected behaviour of the FileStore? Can this be caused because I am not specifying properly the type of the files?
I understood the FileStore does not go in the content of the files and through the user interface I can upload all the files.
Thanks a lot,
Lucia
************* /etc/httpd/conf.d/ckan_default.conf *******************
# CORS
Header set Access-Control-Allow-Origin "*"
Header set Access-Control-Allow-Headers "X-CKAN-API-KEY, Authorization, Origin, X-Requested-With, Content-Type, Accept"
Header set Access-Control-Allow-Methods "POST, PUT, GET, DELETE, OPTIONS"
_________________________________________________________
Dr. Lucia Espona Pernas
Swiss Federal Institute for Forest, Snow and Landscape Research WSL
Hauptgebäaude Labortrakt (HL D34)
Zürcherstrasse 111
8903 Birmensdorf
Switzerland
+41 44 739 28 71 phone direct
+41 44 739 21 11 reception
www.wsl.ch<http://www.wsl.ch>
-----"ckan-dev" <ckan-dev-bounces at lists.okfn.org> wrote: -----
To: CKAN Development Discussions <ckan-dev at lists.okfn.org>
From: Matthew Fullerton
Sent by: "ckan-dev"
Date: 27.04.2016 15:56
Subject: Re: [ckan-dev] CORS issue whe using CKAN API for creating resource
The response headers show that you are using Apache, not nginx. The recommended CKAN setup is to let Apache serve things and use nginx as a proxy to (along with other things) allow setting of headers important for CORS.
Is there an unofficial document anywhere about how to get CORS working for mod_wsgi on Apache?
-Matt
On 27 Apr 2016 2:58 p.m., <lucia.espona at wsl.ch<mailto:lucia.espona at wsl.ch>> wrote:
Hi Mike
IT sounds great but I have no /etc/nginx folder in my server!! While I was waiting for your answer I modified /usr/lib/ckan/default/src/ckan/contrib/docker/nginx.conf adding the headers I saw in ckan/lib/base.py (_set_cors):
proxy_set_header 'Access-Control-Allow-Origin' '*';
proxy_set_header 'Access-Control-Allow-Methods' 'POST, PUT, GET, DELETE, OPTIONS';
proxy_set_header 'Access-Control-Allow-Headers' 'X-CKAN-API-KEY, Authorization, Content-Type';
But I guess it is not the proper file at all. Do you know if there is an alternative place where the nginx config file could be located?
I cannot do the binary data read the way you propose, I am not using Python. I am uploading javascript webapp by doing the ajax call below, in principle it should work also for binary data.
Cheers,
Lucia
var formData = new FormData();
formData.append('package_id', package_id);
....
formData.append("upload", datafile);
$.ajax({
url : ckan_url_create_resource,
type : 'POST',
async: false,
headers: {
'X-CKAN-API-Key':user_token
},
data : formData,
dataType: "json",
processData: false,
contentType: false,
success : function(response, data) {
....
},
error : function(response) {
...
}
});
_________________________________________________________
Dr. Lucia Espona Pernas
Swiss Federal Institute for Forest, Snow and Landscape Research WSL
Hauptgebäaude Labortrakt (HL D34)
Zürcherstrasse 111
8903 Birmensdorf
Switzerland
+41 44 739 28 71<tel:%2B41%2044%20739%2028%2071> phone direct
+41 44 739 21 11<tel:%2B41%2044%20739%2021%2011> reception
www.wsl.ch<http://www.wsl.ch>
-----"ckan-dev" <ckan-dev-bounces at lists.okfn.org<mailto:ckan-dev-bounces at lists.okfn.org>> wrote: -----
To: CKAN Development Discussions <ckan-dev at lists.okfn.org<mailto:ckan-dev at lists.okfn.org>>
From: Mike Sinclair
Sent by: "ckan-dev"
Date: 27.04.2016 13:50
Subject: Re: [ckan-dev] CORS issue whe using CKAN API for creating resource
If you think it is a CORS problem, just for testing purposes in nginx you can add the below to the server block in /etc/nginx/sites-enabled/ckan_default and then reload nginx.
add_header 'Access-Control-Allow-Origin' '*';
add_header 'Access-Control-Allow-Methods' '*';
If you are loading binary files from a windows environment that can cause issues. You will need to open it as a binary type.
files={'upload': open(file, "rb")}
On Wed, Apr 27, 2016 at 4:36 AM, <lucia.espona at wsl.ch<mailto:lucia.espona at wsl.ch>> wrote:
Hi Mike
My first idea was that it was because of binary data but I have done further tests and some images get successfully uploaded and now one plain text xml fails.
This xml had an "error", there was a typo that make it not compliant with the defined xsd, I fixed that in the xml and now it gets upload.
About the JPEG that fails, I cannot se the issue with it, through the user interface got properly upload and displayed (I attached it in case it is of some help).
Maybe the file content cause some issue in the FileStore and the response containing the error doesn't add the CORS headers.
I would be happy to try your suggested alternative "add the headers manually to the nginx config". Unfortunately I am not familiar with that, could you please give me some indications on how to do it?
Thanks,
Lucia
_________________________________________________________
Dr. Lucia Espona Pernas
Swiss Federal Institute for Forest, Snow and Landscape Research WSL
Hauptgebäaude Labortrakt (HL D34)
Zürcherstrasse 111
8903 Birmensdorf
Switzerland
+41 44 739 28 71<tel:%2B41%2044%20739%2028%2071> phone direct
+41 44 739 21 11<tel:%2B41%2044%20739%2021%2011> reception
www.wsl.ch<http://www.wsl.ch>
-----"ckan-dev" <ckan-dev-bounces at lists.okfn.org<mailto:ckan-dev-bounces at lists.okfn.org>> wrote: -----
To: CKAN Development Discussions <ckan-dev at lists.okfn.org<mailto:ckan-dev at lists.okfn.org>>
From: Mike Sinclair
Sent by: "ckan-dev"
Date: 27.04.2016 12:58
Subject: Re: [ckan-dev] CORS issue whe using CKAN API for creating resource
Hi Lucia,
Does this happen to fail on only binary type data? Also, have you tried to manually add the headers to the nginx config?
Mike
[https://mailfoogae.appspot.com/t?sender=abW0uc2luY2xhaXIwOEBnbWFpbC5jb20%3D&type=zerocontent&guid=9b01325e-d4eb-4cb5-9392-910da3c22876]ᐧ
On Wed, Apr 27, 2016 at 3:47 AM, <lucia.espona at wsl.ch<mailto:lucia.espona at wsl.ch>> wrote:
Dear all
I am new to CKAN and I have an issue I am not able to solve.
In the configuration file (.ini) of my CKAN instance (2.5.2) I have set the CORS configuration as follows:
ckan.cors.origin_allow_all = true
I am using the FileStore API for creating resources and SOMETIMES the CORS headers are missing in the response and the request fails:
"Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://ckan.wsl.ch/api/action/resource_create. (Reason: CORS header 'Access-Control-Allow-Origin' missing)."
You can find below two example request I performed, one successful and the other failed, I haven't been able to identify the reason why. Could someone give me a hint how to fix this?
Many thanks in advance and best regards,
Lucia
*********************************************************************
****** Successful Request (file uploaded is a "plain text" xml)******
Request: http://ckan.wsl.ch/api/action/resource_create
Header
Accept application/json, text/javascript, */*; q=0.01
Accept-Encoding gzip, deflate
Accept-Language en-GB,en;q=0.5
Content-Length 2487139
Content-Type multipart/form-data; boundary=---------------------------32069285045071
Host ckan.wsl.ch<http://ckan.wsl.ch>
Origin http://envidat01.wsl.ch:8080
Referer http://envidat01.wsl.ch:8080/MetadataUpload/
User-Agent Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0
X-CKAN-API-Key f1083246-********b3b7bd16
form-data;
name="package_id" 6672445a-3fd7-40af-a47e-690c9b8c6b39
name="id" logo
name="url"
name="format" PNG
name="mimetype" image/png
name="name" logo
name="description" Resource Created from MetadataUpload Webapp using the FileStore API
name="upload";
filename="logo.png" Content-Type: image/png [....]
Response Headers:
Access-Control-Allow-Head... X-CKAN-API-KEY, Authorization, Content-Type
Access-Control-Allow-Meth... POST, PUT, GET, DELETE, OPTIONS
Access-Control-Allow-Orig... *
Cache-Control no-cache
Connection Keep-Alive
Content-Length 836
Content-Type application/json;charset=utf-8
Date Wed, 27 Apr 2016 08:00:51 GMT
Keep-Alive timeout=5, max=96
Pragma no-cache
Server Apache/2.4.6 (CentOS) mod_wsgi/3.4 Python/2.7.5
*********************************************************************
********** FAILED request (file uploaded is a "binary" png) *********
Request: http://ckan.wsl.ch/api/action/resource_create
Header
Accept application/json, text/javascript, */*; q=0.01
Accept-Encoding gzip, deflate
Accept-Language en-GB,en;q=0.5
Content-Length 27875
Content-Type multipart/form-data; boundary=---------------------------16263549323323
Host ckan.wsl.ch<http://ckan.wsl.ch>
Origin http://envidat01.wsl.ch:8080
Referer http://envidat01.wsl.ch:8080/MetadataUpload/
User-Agent Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0
X-CKAN-API-Key f108324**************3b7bd16
form-data;
name="package_id" 6672445a-3fd7-40af-a47e-690c9b8c6b39
name="id" logo
name="url"
name="format" PNG
name="mimetype" image/png
name="name" logo
name="description" Resource Created from MetadataUpload Webapp using the FileStore API
name="upload";
filename="logo.png" Content-Type: image/png [....]
Response Headers:
Connection close
Content-Length 175
Content-Type text/html; charset=utf8
Date Wed, 27 Apr 2016 08:02:48 GMT
Server Apache/2.4.6 (CentOS) mod_wsgi/3.4 Python/2.7.5
_________________________________________________________
Dr. Lucia Espona Pernas
Swiss Federal Institute for Forest, Snow and Landscape Research WSL
Hauptgebäaude Labortrakt (HL D34)
Zürcherstrasse 111
8903 Birmensdorf
Switzerland
+41 44 739 28 71<tel:%2B41%2044%20739%2028%2071> phone direct
+41 44 739 21 11<tel:%2B41%2044%20739%2021%2011> reception
www.wsl.ch<http://www.wsl.ch>
_______________________________________________
ckan-dev mailing list
ckan-dev at lists.okfn.org<mailto:ckan-dev at lists.okfn.org>
https://lists.okfn.org/mailman/listinfo/ckan-dev
Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
_______________________________________________
ckan-dev mailing list
ckan-dev at lists.okfn.org<mailto:ckan-dev at lists.okfn.org>
https://lists.okfn.org/mailman/listinfo/ckan-dev
Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
_______________________________________________
ckan-dev mailing list
ckan-dev at lists.okfn.org<mailto:ckan-dev at lists.okfn.org>
https://lists.okfn.org/mailman/listinfo/ckan-dev
Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
_______________________________________________
ckan-dev mailing list
ckan-dev at lists.okfn.org<mailto:ckan-dev at lists.okfn.org>
https://lists.okfn.org/mailman/listinfo/ckan-dev
Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
_______________________________________________
ckan-dev mailing list
ckan-dev at lists.okfn.org<mailto:ckan-dev at lists.okfn.org>
https://lists.okfn.org/mailman/listinfo/ckan-dev
Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
_______________________________________________
ckan-dev mailing list
ckan-dev at lists.okfn.org
https://lists.okfn.org/mailman/listinfo/ckan-dev
Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20160429/b9e8c6c7/attachment-0003.html>
More information about the ckan-dev
mailing list