[ckan-dev] CORS issue whe using CKAN API for creating resource
lucia.espona at wsl.ch
lucia.espona at wsl.ch
Fri Apr 29 19:32:39 UTC 2016
Hi Matthew
Sorry about Nginx limitations. With Apache there is no problem so far, after adding the "set header" lines to the .conf file all responses have the CORS headers, even the "system error 500" ones.
I found out that the issue with the image file is the name. If you name a jpeg image just with numbers, ex. 157.jpg, the resource creation through the API fails. I does not happen with the text files, I am not sure if it is an issue with other binary files. This issue does not appear in the user interface uploads. I cannot understand from the log error messages what happens, it says 'directory doesn't exist'. It is not very important for me because I will be mainly uploading text files but I guess it is good to know :)
Cheers,
Lucía.
> On 29.04.2016, at 20:35, Matthew Fullerton <matthew at smartlane.de> wrote:
>
> Its such a coincidence that you are hitting this problem. I just two days ago learned that nginx can't add headers with non-successful HTTP responses (you should have seen my face, it was very sad). I wonder if Apache has the same issue?
>
>
> Best,
>
> Matt
>
> ---------------------------------------------------------------------
>
> Matthew Fullerton
>
>
>
> Smartlane UG (haftungsbeschränkt)
>
> Metzstraße 29a
>
> 81667 München
>
>
>
> matthew at smartlane.de
>
> T +49.176.5789.6501
> F +49.89.9041.1930.2
>
>
>
> www.smartlane.de
>
>
>
> ---------------------------------------------------------------------
>
> Geschäftsführer: Dr. Mathias Baur, Florian Schimandl
> Unternehmenssitz: München
> Handelsregister beim Amtsgericht: München
> Handelsregister-Nummer: HRB 220483
> USt. ID-Nr.: DE301856148
>
> Finanzamt: München
>
> ---------------------------------------------------------------------
>
> Von: ckan-dev <ckan-dev-bounces at lists.okfn.org> im Auftrag von lucia.espona at wsl.ch <lucia.espona at wsl.ch>
> Gesendet: Freitag, 29. April 2016 15:26
> An: CKAN Development Discussions
> Betreff: Re: [ckan-dev] CORS issue whe using CKAN API for creating resource
>
> Dear Matthew
>
> I am still waiting for my server admin to install NGINX but I easily managed to add the CORS headers through the Apache configuration as ypu suggested. I included below the added lines to ckan_default.conf in case someone is interested.
>
> Also, I had to set up back to 'false' the ckan.cors.origin_allow_all in the CKAN config file because otherwise they get added again by CKAN and I get the error:
> Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://ckan.wsl.ch/api/action/package_show?name_or_id=10-9999-184. (Reason: CORS header 'Access-Control-Allow-Origin' does not match '*, *').
> For what I have read, it is not allowed to specify multiple comma-separated values in the Access-Control-Allow-Origin header, it has to be a single value or regular expression. Does CKAN add the header after the Apache or the other way around? I am doing a "SET header" instead "ADD header" so I guess that appart from setting CORS in CKAN to false there is nothing I can do.
>
> These two files that raised the initial CORS issue, cause a system error in CKAN because now I get the error response. Something may fail in the resource creation process and the answer does not get the CORS headers added, as it properly happens with the successfully uploaded files. If someone is interested in looking into this I can provide more details.
>
> One of the problematic files is a jpg image, the other is an xml with a typo in one of the tags, if I correct the typo the file uploads without any issue through the API.
> Is this the expected behaviour of the FileStore? Can this be caused because I am not specifying properly the type of the files?
> I understood the FileStore does not go in the content of the files and through the user interface I can upload all the files.
>
>
> Thanks a lot,
> Lucia
>
>
> ************* /etc/httpd/conf.d/ckan_default.conf *******************
> # CORS
> Header set Access-Control-Allow-Origin "*"
> Header set Access-Control-Allow-Headers "X-CKAN-API-KEY, Authorization, Origin, X-Requested-With, Content-Type, Accept"
> Header set Access-Control-Allow-Methods "POST, PUT, GET, DELETE, OPTIONS"
>
> _________________________________________________________
> Dr. Lucia Espona Pernas
>
> Swiss Federal Institute for Forest, Snow and Landscape Research WSL
> Hauptgebäaude Labortrakt (HL D34)
> Zürcherstrasse 111
> 8903 Birmensdorf
> Switzerland
>
> +41 44 739 28 71 phone direct
> +41 44 739 21 11 reception
>
> www.wsl.ch
>
> -----"ckan-dev" <ckan-dev-bounces at lists.okfn.org> wrote: -----
> To: CKAN Development Discussions <ckan-dev at lists.okfn.org>
> From: Matthew Fullerton
> Sent by: "ckan-dev"
> Date: 27.04.2016 15:56
> Subject: Re: [ckan-dev] CORS issue whe using CKAN API for creating resource
>
> The response headers show that you are using Apache, not nginx. The recommended CKAN setup is to let Apache serve things and use nginx as a proxy to (along with other things) allow setting of headers important for CORS.
>
> Is there an unofficial document anywhere about how to get CORS working for mod_wsgi on Apache?
>
> -Matt
>
>> On 27 Apr 2016 2:58 p.m., <lucia.espona at wsl.ch> wrote:
>> Hi Mike
>>
>> IT sounds great but I have no /etc/nginx folder in my server!! While I was waiting for your answer I modified /usr/lib/ckan/default/src/ckan/contrib/docker/nginx.conf adding the headers I saw in ckan/lib/base.py (_set_cors):
>>
>> proxy_set_header 'Access-Control-Allow-Origin' '*';
>> proxy_set_header 'Access-Control-Allow-Methods' 'POST, PUT, GET, DELETE, OPTIONS';
>> proxy_set_header 'Access-Control-Allow-Headers' 'X-CKAN-API-KEY, Authorization, Content-Type';
>>
>> But I guess it is not the proper file at all. Do you know if there is an alternative place where the nginx config file could be located?
>>
>> I cannot do the binary data read the way you propose, I am not using Python. I am uploading javascript webapp by doing the ajax call below, in principle it should work also for binary data.
>>
>>
>> Cheers,
>> Lucia
>>
>>
>>
>>
>> var formData = new FormData();
>> formData.append('package_id', package_id);
>> ....
>> formData.append("upload", datafile);
>>
>> $.ajax({
>> url : ckan_url_create_resource,
>> type : 'POST',
>> async: false,
>> headers: {
>> 'X-CKAN-API-Key':user_token
>> },
>> data : formData,
>> dataType: "json",
>> processData: false,
>> contentType: false,
>> success : function(response, data) {
>> ....
>> },
>> error : function(response) {
>> ...
>> }
>> });
>>
>> _________________________________________________________
>> Dr. Lucia Espona Pernas
>>
>> Swiss Federal Institute for Forest, Snow and Landscape Research WSL
>> Hauptgebäaude Labortrakt (HL D34)
>> Zürcherstrasse 111
>> 8903 Birmensdorf
>> Switzerland
>>
>> +41 44 739 28 71 phone direct
>> +41 44 739 21 11 reception
>>
>> www.wsl.ch
>>
>> -----"ckan-dev" <ckan-dev-bounces at lists.okfn.org> wrote: -----
>> To: CKAN Development Discussions <ckan-dev at lists.okfn.org>
>> From: Mike Sinclair
>> Sent by: "ckan-dev"
>> Date: 27.04.2016 13:50
>> Subject: Re: [ckan-dev] CORS issue whe using CKAN API for creating resource
>>
>> If you think it is a CORS problem, just for testing purposes in nginx you can add the below to the server block in /etc/nginx/sites-enabled/ckan_default and then reload nginx.
>>
>> add_header 'Access-Control-Allow-Origin' '*';
>> add_header 'Access-Control-Allow-Methods' '*';
>>
>> If you are loading binary files from a windows environment that can cause issues. You will need to open it as a binary type.
>> files={'upload': open(file, "rb")}
>>
>>
>>
>>
>>> On Wed, Apr 27, 2016 at 4:36 AM, <lucia.espona at wsl.ch> wrote:
>>> Hi Mike
>>>
>>> My first idea was that it was because of binary data but I have done further tests and some images get successfully uploaded and now one plain text xml fails.
>>> This xml had an "error", there was a typo that make it not compliant with the defined xsd, I fixed that in the xml and now it gets upload.
>>> About the JPEG that fails, I cannot se the issue with it, through the user interface got properly upload and displayed (I attached it in case it is of some help).
>>> Maybe the file content cause some issue in the FileStore and the response containing the error doesn't add the CORS headers.
>>>
>>> I would be happy to try your suggested alternative "add the headers manually to the nginx config". Unfortunately I am not familiar with that, could you please give me some indications on how to do it?
>>>
>>> Thanks,
>>> Lucia
>>>
>>> _________________________________________________________
>>> Dr. Lucia Espona Pernas
>>>
>>> Swiss Federal Institute for Forest, Snow and Landscape Research WSL
>>> Hauptgebäaude Labortrakt (HL D34)
>>> Zürcherstrasse 111
>>> 8903 Birmensdorf
>>> Switzerland
>>>
>>> +41 44 739 28 71 phone direct
>>> +41 44 739 21 11 reception
>>>
>>> www.wsl.ch
>>>
>>> -----"ckan-dev" <ckan-dev-bounces at lists.okfn.org> wrote: -----
>>> To: CKAN Development Discussions <ckan-dev at lists.okfn.org>
>>> From: Mike Sinclair
>>> Sent by: "ckan-dev"
>>> Date: 27.04.2016 12:58
>>> Subject: Re: [ckan-dev] CORS issue whe using CKAN API for creating resource
>>>
>>>
>>> Hi Lucia,
>>>
>>> Does this happen to fail on only binary type data? Also, have you tried to manually add the headers to the nginx config?
>>>
>>> Mike
>>> ᐧ
>>>
>>>> On Wed, Apr 27, 2016 at 3:47 AM, <lucia.espona at wsl.ch> wrote:
>>>> Dear all
>>>>
>>>> I am new to CKAN and I have an issue I am not able to solve.
>>>>
>>>> In the configuration file (.ini) of my CKAN instance (2.5.2) I have set the CORS configuration as follows:
>>>> ckan.cors.origin_allow_all = true
>>>> I am using the FileStore API for creating resources and SOMETIMES the CORS headers are missing in the response and the request fails:
>>>> "Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://ckan.wsl.ch/api/action/resource_create. (Reason: CORS header 'Access-Control-Allow-Origin' missing)."
>>>> You can find below two example request I performed, one successful and the other failed, I haven't been able to identify the reason why. Could someone give me a hint how to fix this?
>>>>
>>>> Many thanks in advance and best regards,
>>>> Lucia
>>>>
>>>> *********************************************************************
>>>> ****** Successful Request (file uploaded is a "plain text" xml)******
>>>>
>>>> Request: http://ckan.wsl.ch/api/action/resource_create
>>>> Header
>>>> Accept application/json, text/javascript, */*; q=0.01
>>>> Accept-Encoding gzip, deflate
>>>> Accept-Language en-GB,en;q=0.5
>>>> Content-Length 2487139
>>>> Content-Type multipart/form-data; boundary=---------------------------32069285045071
>>>> Host ckan.wsl.ch
>>>> Origin http://envidat01.wsl.ch:8080
>>>> Referer http://envidat01.wsl.ch:8080/MetadataUpload/
>>>> User-Agent Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0
>>>> X-CKAN-API-Key f1083246-********b3b7bd16
>>>>
>>>> form-data;
>>>> name="package_id" 6672445a-3fd7-40af-a47e-690c9b8c6b39
>>>> name="id" logo
>>>> name="url"
>>>> name="format" PNG
>>>> name="mimetype" image/png
>>>> name="name" logo
>>>> name="description" Resource Created from MetadataUpload Webapp using the FileStore API
>>>> name="upload";
>>>>
>>>> filename="logo.png" Content-Type: image/png [....]
>>>>
>>>> Response Headers:
>>>> Access-Control-Allow-Head... X-CKAN-API-KEY, Authorization, Content-Type
>>>> Access-Control-Allow-Meth... POST, PUT, GET, DELETE, OPTIONS
>>>> Access-Control-Allow-Orig... *
>>>> Cache-Control no-cache
>>>> Connection Keep-Alive
>>>> Content-Length 836
>>>> Content-Type application/json;charset=utf-8
>>>> Date Wed, 27 Apr 2016 08:00:51 GMT
>>>> Keep-Alive timeout=5, max=96
>>>> Pragma no-cache
>>>> Server Apache/2.4.6 (CentOS) mod_wsgi/3.4 Python/2.7.5
>>>>
>>>>
>>>>
>>>> *********************************************************************
>>>> ********** FAILED request (file uploaded is a "binary" png) *********
>>>> Request: http://ckan.wsl.ch/api/action/resource_create
>>>> Header
>>>> Accept application/json, text/javascript, */*; q=0.01
>>>> Accept-Encoding gzip, deflate
>>>> Accept-Language en-GB,en;q=0.5
>>>> Content-Length 27875
>>>> Content-Type multipart/form-data; boundary=---------------------------16263549323323
>>>> Host ckan.wsl.ch
>>>> Origin http://envidat01.wsl.ch:8080
>>>> Referer http://envidat01.wsl.ch:8080/MetadataUpload/
>>>> User-Agent Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0
>>>> X-CKAN-API-Key f108324**************3b7bd16
>>>>
>>>> form-data;
>>>> name="package_id" 6672445a-3fd7-40af-a47e-690c9b8c6b39
>>>> name="id" logo
>>>> name="url"
>>>> name="format" PNG
>>>> name="mimetype" image/png
>>>> name="name" logo
>>>> name="description" Resource Created from MetadataUpload Webapp using the FileStore API
>>>> name="upload";
>>>>
>>>> filename="logo.png" Content-Type: image/png [....]
>>>>
>>>> Response Headers:
>>>> Connection close
>>>> Content-Length 175
>>>> Content-Type text/html; charset=utf8
>>>> Date Wed, 27 Apr 2016 08:02:48 GMT
>>>> Server Apache/2.4.6 (CentOS) mod_wsgi/3.4 Python/2.7.5
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> _________________________________________________________
>>>> Dr. Lucia Espona Pernas
>>>>
>>>> Swiss Federal Institute for Forest, Snow and Landscape Research WSL
>>>> Hauptgebäaude Labortrakt (HL D34)
>>>> Zürcherstrasse 111
>>>> 8903 Birmensdorf
>>>> Switzerland
>>>>
>>>> +41 44 739 28 71 phone direct
>>>> +41 44 739 21 11 reception
>>>>
>>>> www.wsl.ch
>>>> _______________________________________________
>>>> ckan-dev mailing list
>>>> ckan-dev at lists.okfn.org
>>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>>
>>> _______________________________________________
>>> ckan-dev mailing list
>>> ckan-dev at lists.okfn.org
>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>>
>>> _______________________________________________
>>> ckan-dev mailing list
>>> ckan-dev at lists.okfn.org
>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20160429/8ca398bc/attachment-0003.html>
More information about the ckan-dev
mailing list