[ckan-dev] CORS issue whe using CKAN API for creating resource

lucia.espona at wsl.ch lucia.espona at wsl.ch
Fri Apr 29 13:26:48 UTC 2016 

 Dear Matthew

I am still waiting for my server admin to install NGINX but I easily managed to add the CORS headers through the Apache configuration as ypu suggested. I included below  the added lines to ckan_default.conf in case someone is interested. 

Also, I had to set up back to 'false' the ckan.cors.origin_allow_all in the CKAN config file because otherwise they get added again by CKAN and I get the error:
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://ckan.wsl.ch/api/action/package_show?name_or_id=10-9999-184. (Reason: CORS header 'Access-Control-Allow-Origin' does not match '*, *'). 
For what I have read, it is not allowed to specify multiple comma-separated values in the Access-Control-Allow-Origin header, it has to be a single value or regular expression. Does CKAN add the header after the Apache or the other way around? I am doing a "SET header" instead "ADD header" so I guess that appart from setting CORS in CKAN to false there is nothing I can do. 

These two files that raised the initial CORS issue, cause a system error in CKAN because now I get the error response. Something may fail in the resource creation process and the answer does not get the CORS headers added, as it properly happens with the successfully uploaded files. If someone is interested in looking into this I can provide more details.

One of the problematic files is a jpg image, the other is an xml with a typo in one of the tags, if I correct the typo the file uploads without any issue through the API.
Is this the expected behaviour of the FileStore? Can this be caused because I am not specifying properly the type of the files?
I understood the FileStore does not go in the content of the files and through the user interface I can upload all the files.


Thanks a lot,
Lucia

 
************* /etc/httpd/conf.d/ckan_default.conf *******************
    # CORS 
    Header set Access-Control-Allow-Origin "*"
    Header set Access-Control-Allow-Headers "X-CKAN-API-KEY, Authorization, Origin, X-Requested-With, Content-Type, Accept"
    Header set Access-Control-Allow-Methods "POST, PUT, GET, DELETE, OPTIONS"

_________________________________________________________
Dr. Lucia Espona Pernas

Swiss Federal Institute for Forest, Snow and Landscape Research WSL
Hauptgebäaude Labortrakt (HL D34)
Zürcherstrasse 111
8903 Birmensdorf
Switzerland

+41 44 739 28 71 phone direct
+41 44 739 21 11 reception

www.wsl.ch

-----"ckan-dev" <ckan-dev-bounces at lists.okfn.org> wrote: -----
To: CKAN Development Discussions <ckan-dev at lists.okfn.org>
From: Matthew Fullerton 
Sent by: "ckan-dev" 
Date: 27.04.2016 15:56
Subject: Re: [ckan-dev] CORS issue whe using CKAN API for creating resource

The response headers show that you are using Apache, not nginx. The recommended CKAN setup is to let Apache serve things and use nginx as a proxy to (along with other things) allow setting of headers important for CORS.
 Is there an unofficial document anywhere about how to get CORS working for mod_wsgi on Apache?
 -Matt
 
On 27 Apr 2016 2:58 p.m.,  <lucia.espona at wsl.ch> wrote:
 Hi Mike

IT sounds great but I have no /etc/nginx folder in my server!! While I was waiting for your answer I modified /usr/lib/ckan/default/src/ckan/contrib/docker/nginx.conf adding the headers I saw in ckan/lib/base.py (_set_cors):

      proxy_set_header 'Access-Control-Allow-Origin' '*';
      proxy_set_header 'Access-Control-Allow-Methods' 'POST, PUT, GET, DELETE, OPTIONS';
      proxy_set_header 'Access-Control-Allow-Headers' 'X-CKAN-API-KEY, Authorization, Content-Type';

But I guess it is not the proper file at all. Do you know if there is an alternative place where the nginx config file could be located?

I cannot do the binary data read the way you propose, I am not using Python. I am uploading javascript webapp by doing the ajax call below, in principle it should work also for binary data.


Cheers,
Lucia




var formData = new FormData();
        formData.append('package_id',  package_id);
        ....
        formData.append("upload", datafile);
        
        $.ajax({
              url : ckan_url_create_resource,
              type : 'POST',
              async: false,
              headers: {
                  'X-CKAN-API-Key':user_token
              },
              data : formData,
              dataType: "json",
              processData: false,
              contentType: false,   
              success : function(response, data) {
                    ....
              },
              error : function(response) {
                  ...
                }
            });
 
_________________________________________________________
Dr. Lucia Espona Pernas

Swiss Federal Institute for Forest, Snow and Landscape Research WSL
Hauptgebäaude Labortrakt (HL D34)
Zürcherstrasse 111
8903 Birmensdorf
Switzerland

+41 44 739 28 71 phone direct
+41 44 739 21 11 reception

www.wsl.ch

-----"ckan-dev" <ckan-dev-bounces at lists.okfn.org> wrote: -----
To: CKAN Development Discussions <ckan-dev at lists.okfn.org>
From: Mike Sinclair 
Sent by: "ckan-dev" 
Date: 27.04.2016 13:50
Subject: Re: [ckan-dev] CORS issue whe using CKAN API for creating resource

If you think it is a CORS problem, just for testing purposes in nginx you can add the below to the server block in /etc/nginx/sites-enabled/ckan_default and then reload nginx.

    add_header 'Access-Control-Allow-Origin' '*';
    add_header 'Access-Control-Allow-Methods' '*';

If you are loading binary files from a windows environment that can cause issues. You will need to open it as a binary type.
files={'upload': open(file, "rb")}




On Wed, Apr 27, 2016 at 4:36 AM,  <lucia.espona at wsl.ch> wrote:
Hi Mike

My first idea was that it was because of binary data but I have done further tests and some images get successfully uploaded and now one plain text xml fails. 
This xml had an "error", there was a typo that make it not compliant with the defined xsd, I fixed that in the xml and now it gets upload.
About the JPEG that fails, I cannot se the issue with it, through the user interface got properly upload and displayed (I attached it in case it is of some help). 
Maybe the file content cause some issue in the FileStore and the response containing the error doesn't add the CORS headers.

I would be happy to try your suggested alternative "add the headers manually to the nginx config". Unfortunately I am not familiar with that,ÿ could you please give me some indications on how to do it? 

Thanks,
Lucia

_________________________________________________________
Dr. Lucia Espona Pernas

Swiss Federal Institute for Forest, Snow and Landscape Research WSL
Hauptgeb„aude Labortrakt (HL D34)
Zrcherstrasse 111
8903 Birmensdorf
Switzerland

+41 44 739 28 71 phone direct
+41 44 739 21 11 reception

www.wsl.ch

-----"ckan-dev" <ckan-dev-bounces at lists.okfn.org> wrote: -----
To: CKAN Development Discussions <ckan-dev at lists.okfn.org>
From: Mike Sinclair 
Sent by: "ckan-dev" 
Date: 27.04.2016 12:58
Subject: Re: [ckan-dev] CORS issue whe using CKAN API for creating resource


Hi Lucia,

Does this happen to fail on only binary type data? Also, have you tried to manually add the headers to the nginx config?

Mike
'

On Wed, Apr 27, 2016 at 3:47 AM,  <lucia.espona at wsl.ch> wrote:
 Dear all

I am new to CKAN and I have an issue I am not able to solve.

In the configuration file (.ini) of my CKAN instance (2.5.2) I have set the CORS configuration as follows:
ckan.cors.origin_allow_all = trueI am using the FileStore API for creating resources and SOMETIMES the CORS headers are missing in the response and the request fails: 
"Cross-Origin  Request Blocked: The Same Origin Policy disallows reading the remote  resource at http://ckan.wsl.ch/api/action/resource_create. (Reason: CORS  header 'Access-Control-Allow-Origin' missing)."
You can find below two example request I performed, one successful and the other failed, I haven't been able to identify the reason why. Could someone give me a hint how to fix this?

Many thanks in advance and best regards,
Lucia

*********************************************************************
****** Successful Request (file uploaded is a "plain text" xml)******

Request: http://ckan.wsl.ch/api/action/resource_create
ÿÿ Header
ÿÿÿ Acceptÿÿÿ application/json, text/javascript, */*; q=0.01
ÿÿÿ Accept-Encodingÿÿÿ gzip, deflate
ÿÿÿ Accept-Languageÿÿÿ en-GB,en;q=0.5
ÿÿÿ Content-Lengthÿÿÿ 2487139ÿÿÿ 
ÿÿÿ Content-Typeÿÿÿ multipart/form-data; boundary=---------------------------32069285045071
ÿÿÿ Hostÿÿÿ ckan.wsl.ch
ÿÿÿ Originÿÿÿ http://envidat01.wsl.ch:8080
ÿÿÿ Refererÿÿÿ http://envidat01.wsl.ch:8080/MetadataUpload/
ÿÿÿ User-Agentÿÿÿ Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0
ÿÿÿ X-CKAN-API-Keyÿÿÿ f1083246-********b3b7bd16

ÿÿ form-data; 
ÿÿÿÿÿÿÿÿÿÿ name="package_id"  6672445a-3fd7-40af-a47e-690c9b8c6b39
ÿÿÿÿÿÿÿÿÿÿ name="id"  logo 
ÿÿÿÿÿÿÿÿÿÿ name="url"   
ÿÿÿÿÿÿÿÿÿÿ name="format"  PNG
ÿÿÿÿÿÿÿÿÿÿ name="mimetype"  image/png 
ÿÿÿÿÿÿÿÿÿÿ name="name"  logo 
ÿÿÿÿÿÿÿÿÿÿ name="description"  Resource Created from MetadataUpload Webapp using the FileStore API 
ÿÿÿÿÿÿÿÿÿÿ name="upload"; 
ÿ
ÿÿ filename="logo.png" Content-Type: image/png [....]

Response Headers:
ÿÿÿ Access-Control-Allow-Head...ÿÿÿ X-CKAN-API-KEY, Authorization, Content-Type
ÿÿÿ Access-Control-Allow-Meth...ÿÿÿ POST, PUT, GET, DELETE, OPTIONS
ÿÿÿ Access-Control-Allow-Orig...ÿÿÿ *
ÿÿÿ Cache-Controlÿÿÿ no-cache
ÿÿÿ Connectionÿÿÿ Keep-Alive
ÿÿÿ Content-Lengthÿÿÿ 836
ÿÿÿ Content-Typeÿÿÿ application/json;charset=utf-8
ÿÿÿ Dateÿÿÿ Wed, 27 Apr 2016 08:00:51 GMT
ÿÿÿ Keep-Aliveÿÿÿ timeout=5, max=96
ÿÿÿ Pragmaÿÿÿ no-cache
ÿÿÿ Serverÿÿÿ Apache/2.4.6 (CentOS) mod_wsgi/3.4 Python/2.7.5



*********************************************************************
********** FAILED request (file uploaded is a "binary" png) *********
Request: http://ckan.wsl.ch/api/action/resource_create
ÿÿ Header
ÿÿÿÿÿAcceptÿÿÿ application/json, text/javascript, */*; q=0.01
ÿÿÿÿÿ Accept-Encoding gzip, deflate
ÿÿÿÿÿ Accept-Languageÿ en-GB,en;q=0.5
ÿÿÿÿÿ Content-Lengthÿÿ 27875
ÿÿÿÿÿ Content-Typeÿÿÿ multipart/form-data; boundary=---------------------------16263549323323
ÿÿÿÿÿ Hostÿÿÿ ckan.wsl.ch
ÿÿÿÿÿ Originÿ http://envidat01.wsl.ch:8080
ÿÿÿÿÿ Referer http://envidat01.wsl.ch:8080/MetadataUpload/
ÿÿÿÿÿ User-Agent Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0
ÿÿÿÿÿ X-CKAN-API-Key f108324**************3b7bd16

ÿÿ form-data; 
ÿÿÿÿÿÿÿÿÿÿ name="package_id"  6672445a-3fd7-40af-a47e-690c9b8c6b39
ÿÿÿÿÿÿÿÿÿÿ name="id"  logo 
ÿÿÿÿÿÿÿÿÿÿ name="url"   
ÿÿÿÿÿÿÿÿÿÿ name="format"  PNG
ÿÿÿÿÿÿÿÿÿÿ name="mimetype"  image/png 
ÿÿÿÿÿÿÿÿÿÿ name="name"  logo 
ÿÿÿÿÿÿÿÿÿÿ name="description"  Resource Created from MetadataUpload Webapp using the FileStore API 
ÿÿÿÿÿÿÿÿÿÿ name="upload"; 
ÿ
ÿÿ filename="logo.png" Content-Type: image/png [....]

Response Headers:
Connectionÿÿÿ close
Content-Length 175
Content-Typeÿÿ text/html; charset=utf8
Dateÿÿÿ Wed, 27 Apr 2016 08:02:48 GMT
Serverÿ Apache/2.4.6 (CentOS) mod_wsgi/3.4 Python/2.7.5





_________________________________________________________
Dr. Lucia Espona Pernas

Swiss Federal Institute for Forest, Snow and Landscape Research WSL
Hauptgeb„aude Labortrakt (HL D34)
Zrcherstrasse 111
8903 Birmensdorf
Switzerland

+41 44 739 28 71 phone direct
+41 44 739 21 11 reception

www.wsl.ch
_______________________________________________
 ckan-dev mailing list
 ckan-dev at lists.okfn.org
 https://lists.okfn.org/mailman/listinfo/ckan-dev
 Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
 

 
_______________________________________________
ckan-dev mailing list
ckan-dev at lists.okfn.org
https://lists.okfn.org/mailman/listinfo/ckan-dev
Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev

_______________________________________________
 ckan-dev mailing list
 ckan-dev at lists.okfn.org
 https://lists.okfn.org/mailman/listinfo/ckan-dev
 Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
 

 
_______________________________________________
ckan-dev mailing list
ckan-dev at lists.okfn.org
https://lists.okfn.org/mailman/listinfo/ckan-dev
Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev

_______________________________________________
 ckan-dev mailing list
 ckan-dev at lists.okfn.org
 https://lists.okfn.org/mailman/listinfo/ckan-dev
 Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
 
 
_______________________________________________
ckan-dev mailing list
ckan-dev at lists.okfn.org
https://lists.okfn.org/mailman/listinfo/ckan-dev
Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20160429/ace27dda/attachment-0003.html>

More information about the ckan-dev mailing list