No subject

Sun Oct 20 16:01:18 UTC 2013

not taken into account. We have updated rdf2ckan to put is also in the
header and will do a test with the new version.

kind regards,

Bert

2013/11/12 John Glover <john.glover at okfn.org>

> Hi Bert,
>
> Yes I tested both create and update (both of which go through the same RDF
> validator).
>
> Ok thanks for the info.
>
> Regards,
> John
>
>
> On 12 November 2013 11:34, Bert Van Nuffelen <
> bert.van.nuffelen at tenforce.com> wrote:
>
>> Hi John,
>>
>> strange. I will create a full dump of the request to see the content. Did
>> you do a create and an update?
>>
>> best,
>>
>> Bert
>>
>> ps. we have an power interrupt from 12h (in half an hour) till 13h. So
>> there might be communication glitches.
>>
>> Bert
>>
>>
>>
>> 2013/11/12 John Glover <john.glover at okfn.org>
>>
>>> Hi Bert,
>>>
>>> I am unable to reproduce your error here. Both of those JSON files are
>>> imported fine for me using a simple Python client to make an API request
>>> (release 01), neither produce the error. Could you therefore please double
>>> check that your API request is correct, and let me know when the test
>>> server is back to release 01 so that we can test it again there.
>>>
>>> I have pushed a change to ckanext-ecportal so it will now raise a
>>> ValidationError if it cannot parse the RDF in the dataset JSON as requested.
>>>
>>> Regards,
>>> John
>>>
>>>
>>> On 12 November 2013 09:44, John Glover <john.glover at okfn.org> wrote:
>>>
>>>> Hi Bert,
>>>>
>>>> Thanks, I will have a look at that JSON file and get back to you.
>>>>
>>>> Regards,
>>>> John
>>>>
>>>>
>>>> On 12 November 2013 09:08, Bert Van Nuffelen <
>>>> bert.van.nuffelen at tenforce.com> wrote:
>>>>
>>>>> Hi John,
>>>>>
>>>>> I forgot to add the message that I attached example json's  to the
>>>>> previous respons.
>>>>>
>>>>> Bert
>>>>>
>>>>>
>>>>> 2013/11/12 Bert Van Nuffelen <bert.van.nuffelen at tenforce.com>
>>>>>
>>>>>> Hi John,
>>>>>>
>>>>>>
>>>>>> 2013/11/11 John Glover <john.glover at okfn.org>
>>>>>>
>>>>>>> Hi Bert,
>>>>>>>
>>>>>>> The last time this issue came up, the only way that we could
>>>>>>> recreate the problem was when making a request without setting the content
>>>>>>> type to 'application/json'. As you are now doing this, could you please
>>>>>>> send us an example of the JSON that you are sending to CKAN in a failing
>>>>>>> request so that we can investigate further.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>> > When I do my tests, it seems that in case the error occurs the
>>>>>>> submitted RDF is not saved, and hence the RDF in CKAN is
>>>>>>> > that one of the CKAN RDF generation process. Is that true?
>>>>>>>
>>>>>>> Yes this is correct.
>>>>>>>
>>>>>>> > In that case the level of the message ERROR and the result by the
>>>>>>> API call return to it are not in sync. On an ERROR I
>>>>>>> > expect that the api call return an error code. If that was a warn
>>>>>>> I can understand that the api call returns success.
>>>>>>>
>>>>>>> We could of course return a validation error here (although this has
>>>>>>> not been requested before). I believe the thinking was that it was best to
>>>>>>> save the data and ignore this error as CKAN can generate RDF, but this
>>>>>>> logic could be changed.
>>>>>>>
>>>>>>
>>>>>> I understand the reasoning, but it results in an incorrect
>>>>>> end-result. In this case it even has hidden a communication error between
>>>>>> RDF2CKAN and CKAN. Since the api provides a correct feedback mechanism I
>>>>>> would like to rely on that.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Regards,
>>>>>>> John
>>>>>>>
>>>>>>>
>>>>>>> On 8 November 2013 11:47, Bert Van Nuffelen <
>>>>>>> bert.van.nuffelen at tenforce.com> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> one addition, I run overnight with the same RDF2CKAN and the same
>>>>>>>> ESTAT package an upload on the release 09.00.0x version and there in the
>>>>>>>> CKAN logs the above message is not  present.
>>>>>>>>
>>>>>>>> But when I do the synchronisation with virtuoso then I get
>>>>>>>> exceptions on parsing the RDF files as they have been converted by CKAN.
>>>>>>>>
>>>>>>>> So it seems that
>>>>>>>>    * in release 09.00.0x the check on this issue was not present at
>>>>>>>> all,
>>>>>>>>    * in release 01.00.0x the check is there, but is silently
>>>>>>>> replaces it with an empty value
>>>>>>>>
>>>>>>>> In both releases the accept-header is not bound to the desired
>>>>>>>> execution.
>>>>>>>>
>>>>>>>> kind regards,
>>>>>>>>
>>>>>>>> Bert
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> 2013/11/7 Bert Van Nuffelen <bert.van.nuffelen at tenforce.com>
>>>>>>>>
>>>>>>>>> Dear all,
>>>>>>>>>
>>>>>>>>> We have again this problem. This is related to the issue of ODP-294
>>>>>>>>>
>>>>>>>>> in the apache logs:
>>>>>>>>>
>>>>>>>>> [Tue Nov 05 21:16:57 2013] [error] 2013-11-05 21:16:57,306 ERROR
>>>>>>>>> [ckanext.ecportal.rdfutil] EntityRef: expecting ';', line 39, column 103
>>>>>>>>>
>>>>>>>>> in the CKAN logs
>>>>>>>>>
>>>>>>>>> 2013-11-07 22:14:04,975 ERROR [ckanext.ecportal.rdfutil]
>>>>>>>>> EntityRef: expecting ';', line 39, column 102
>>>>>>>>>
>>>>>>>>> As this is silently in the logs instead of a _proper_ validation
>>>>>>>>> error like:
>>>>>>>>>
>>>>>>>>> 2013-11-07 22:09:21,996 ERROR [ckan.controllers.api] Validation
>>>>>>>>> error: '{\'keyword_string\': [u\'Tag "inter
>>>>>>>>> national trade,trade statistics" must be alphanumeric characters
>>>>>>>>> or symbols: -_.\'], \'__type\': \'Validati
>>>>>>>>> on Error\'}'
>>>>>>>>>
>>>>>>>>> rdf2ckan can only conclude it has inserted the data correctly
>>>>>>>>> while it is not.
>>>>>>>>> So the outcome of rdf2ckan is hence untrust whorthy. For that
>>>>>>>>> reason this issue remained unseen by TF development.
>>>>>>>>>
>>>>>>>>> Secondly I am concerned about the fact if the above configuration
>>>>>>>>> solution described by David is working.
>>>>>>>>> It seems that the setting of the header to "application/json" does
>>>>>>>>> not result in the correct handling by CKAN.
>>>>>>>>> Here under is the extract of the java code showing that we sent
>>>>>>>>> the content-type as requested.
>>>>>>>>>
>>>>>>>>> private static final String CONTENT_JSON = "application/json";
>>>>>>>>>
>>>>>>>>> public Map<String, Object> post(String json, String method) {
>>>>>>>>>         if (json.isEmpty() || method.isEmpty())
>>>>>>>>>             return null;
>>>>>>>>>         DefaultHttpClient httpClient = new DefaultHttpClient();
>>>>>>>>>
>>>>>>>>>         HttpPost httpPost = requestCredentials(method);
>>>>>>>>>         json = json.replaceAll(";", "");
>>>>>>>>>         httpPost.setEntity(streamify(json, CONTENT_JSON));
>>>>>>>>>        ...
>>>>>>>>>
>>>>>>>>>  public InputStreamEntity streamify(String json, String
>>>>>>>>> contentType) {
>>>>>>>>>         json = json.replaceAll(";", "");
>>>>>>>>>         InputStreamEntity inputStreamEntity = new
>>>>>>>>> InputStreamEntity(new
>>>>>>>>> ByteArrayInputStream(json.getBytes(Charset.forName("UTF-8"))),
>>>>>>>>> json.getBytes().length);
>>>>>>>>>         inputStreamEntity.setContentType(contentType);
>>>>>>>>>         return inputStreamEntity;
>>>>>>>>>     }
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Note that the submitted RDF is all valid rdf and XML in the
>>>>>>>>> request body: it passes through all XML & RDF validations (e.g. rapper).
>>>>>>>>>
>>>>>>>>> When I do my tests, it seems that in case the error occurs the
>>>>>>>>> submitted RDF is not saved, and hence the RDF in CKAN is that one of the
>>>>>>>>> CKAN RDF generation process. Is that true? In that case the level of the
>>>>>>>>> message ERROR and the result by the API call return to it are not in sync.
>>>>>>>>> On an ERROR I expect that the api call return an error code. If that was a
>>>>>>>>> warn I can understand that the api call returns success.
>>>>>>>>>
>>>>>>>>> So the next actions should be taken:
>>>>>>>>>
>>>>>>>>> 1) make the above error message a proper validation result.
>>>>>>>>>     Note this solves also the problem of the meaningless error
>>>>>>>>> message. There is no reference at all in the message to know about which
>>>>>>>>> call/object it is. So the problem is unresolvable with this message. And
>>>>>>>>> putting CKAN in debug mode logging for that is not very sensible.
>>>>>>>>> 2) ensure that the handling of RDF (in rdfxml-format) inside the
>>>>>>>>> json is treated properly and not interpreted as an html encoding.
>>>>>>>>>
>>>>>>>>> best regards,
>>>>>>>>>
>>>>>>>>> Bert
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2013/9/18 David Raznick <david.raznick at okfn.org>
>>>>>>>>>
>>>>>>>>>> Hello
>>>>>>>>>>
>>>>>>>>>> It looks like the error is because the wrong content type is sent
>>>>>>>>>> when posting to ckan in rdf2ckan.   Content-Type:application/json should be
>>>>>>>>>> sent otherwise CKAN will think it is urlencoded and therefore make the
>>>>>>>>>> final document encoded wrongly.
>>>>>>>>>>
>>>>>>>>>> This must have changed at somepoint, as the live db does not have
>>>>>>>>>> this issue and is likely what caused the unicode issues from happening too.
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>>
>>>>>>>>>> David
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 18 September 2013 16:16, Dimitrios Mexis <
>>>>>>>>>> dimitrios.mexis at tenforce.com> wrote:
>>>>>>>>>>
>>>>>>>>>>>  Hello David,
>>>>>>>>>>>
>>>>>>>>>>> we don't do any postprocessing for the data we send.
>>>>>>>>>>>
>>>>>>>>>>> It's pure rdf content as read from the file. So, we depend on
>>>>>>>>>>> CKAN how it will digest it.
>>>>>>>>>>> If we need to do some postprocessing, can you give clarification
>>>>>>>>>>> on the matter ?
>>>>>>>>>>>
>>>>>>>>>>> Also, does that have to do with the problem we faced with
>>>>>>>>>>> unicode exceptions from CKAN as well ?
>>>>>>>>>>>
>>>>>>>>>>> Kind regards
>>>>>>>>>>> Dimitrios
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 18/09/2013 17:12, David Raznick wrote:
>>>>>>>>>>>
>>>>>>>>>>>  Hello Bert
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>  There appears to be a lot of malformed rdf documents in the
>>>>>>>>>>> test system e.g
>>>>>>>>>>>
>>>>>>>>>>> http://212.71.25.148/en/data/dataset/BrvXA5sZQ1AFKgE4Pktw.rdf
>>>>>>>>>>>
>>>>>>>>>>>  After some investigation this seems to due to the data send by
>>>>>>>>>>> rdf2ckan.  It appears that xml "&" are not being escaped correctly in some
>>>>>>>>>>> places.
>>>>>>>>>>>
>>>>>>>>>>>  These errors do not appear on the live db as far as I can see,
>>>>>>>>>>> could you please look into this.
>>>>>>>>>>>
>>>>>>>>>>>  Thanks
>>>>>>>>>>>
>>>>>>>>>>> David
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Ecodp-dev mailing listEcodp-dev at lists.okfn.orghttp://lists.okfn.org/mailman/listinfo/ecodp-dev
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Ecodp-dev mailing list
>>>>>>>>>>> Ecodp-dev at lists.okfn.org
>>>>>>>>>>> http://lists.okfn.org/mailman/listinfo/ecodp-dev
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Ecodp-dev mailing list
>>>>>>>>>> Ecodp-dev at lists.okfn.org
>>>>>>>>>> http://lists.okfn.org/mailman/listinfo/ecodp-dev
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Bert Van Nuffelen
>>>>>>>>>
>>>>>>>>> Semantic Technologies Software Architect at TenForce
>>>>>>>>> www.tenforce.be
>>>>>>>>>
>>>>>>>>> Bert.Van.Nuffelen at tenforce.com
>>>>>>>>> Office: +32 (0)16 31 48 60
>>>>>>>>> Mobile:+32 479 06 24 26
>>>>>>>>> skype: bert.van.nuffelen
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Bert Van Nuffelen
>>>>>>>>
>>>>>>>> Semantic Technologies Software Architect at TenForce
>>>>>>>> www.tenforce.be
>>>>>>>>
>>>>>>>> Bert.Van.Nuffelen at tenforce.com
>>>>>>>> Office: +32 (0)16 31 48 60
>>>>>>>> Mobile:+32 479 06 24 26
>>>>>>>> skype: bert.van.nuffelen
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Ecodp-dev mailing list
>>>>>>>> Ecodp-dev at lists.okfn.org
>>>>>>>> http://lists.okfn.org/mailman/listinfo/ecodp-dev
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Ecodp-dev mailing list
>>>>>>> Ecodp-dev at lists.okfn.org
>>>>>>> http://lists.okfn.org/mailman/listinfo/ecodp-dev
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Bert Van Nuffelen
>>>>>>
>>>>>> Semantic Technologies Software Architect at TenForce
>>>>>> www.tenforce.be
>>>>>>
>>>>>> Bert.Van.Nuffelen at tenforce.com
>>>>>> Office: +32 (0)16 31 48 60
>>>>>> Mobile:+32 479 06 24 26
>>>>>> skype: bert.van.nuffelen
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Bert Van Nuffelen
>>>>>
>>>>> Semantic Technologies Software Architect at TenForce
>>>>> www.tenforce.be
>>>>>
>>>>> Bert.Van.Nuffelen at tenforce.com
>>>>> Office: +32 (0)16 31 48 60
>>>>> Mobile:+32 479 06 24 26
>>>>> skype: bert.van.nuffelen
>>>>>
>>>>> _______________________________________________
>>>>> Ecodp-dev mailing list
>>>>> Ecodp-dev at lists.okfn.org
>>>>> http://lists.okfn.org/mailman/listinfo/ecodp-dev
>>>>>
>>>>>
>>>>
>>>
>>> _______________________________________________
>>> Ecodp-dev mailing list
>>> Ecodp-dev at lists.okfn.org
>>> http://lists.okfn.org/mailman/listinfo/ecodp-dev
>>>
>>>
>>
>>
>> --
>> Bert Van Nuffelen
>>
>> Semantic Technologies Software Architect at TenForce
>> www.tenforce.be
>>
>> Bert.Van.Nuffelen at tenforce.com
>> Office: +32 (0)16 31 48 60
>> Mobile:+32 479 06 24 26
>> skype: bert.van.nuffelen
>>
>> _______________________________________________
>> Ecodp-dev mailing list
>> Ecodp-dev at lists.okfn.org
>> http://lists.okfn.org/mailman/listinfo/ecodp-dev
>>
>>
>
> _______________________________________________
> Ecodp-dev mailing list
> Ecodp-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ecodp-dev
>
>

-- 
Bert Van Nuffelen

Semantic Technologies Software Architect at TenForce
www.tenforce.be

Bert.Van.Nuffelen at tenforce.com
Office: +32 (0)16 31 48 60
Mobile:+32 479 06 24 26
skype: bert.van.nuffelen

--001a11335194b5839504eafc6c8f
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><div><div><div>Hi John,<br><br></div>=C2=A0- I am bus=
y resurrecting the machines after the power cut this noon. - <br><br></div>=
<div>On this topic: we see by doing a detailed analysis that the apache Jav=
a library is setting the content-type in the body of the request. <br>
</div><div>From what I have seen so far that might be the case why this par=
ameter is not taken into account. We have updated rdf2ckan to put is also i=
n the header and will do a test with the new version.<br></div><br></div>
kind regards,<br><br></div>Bert<br><div><div><br></div></div></div><div cla=
ss=3D"gmail_extra"><br><br><div class=3D"gmail_quote">2013/11/12 John Glove=
r <span dir=3D"ltr">&lt;<a href=3D"mailto:john.glover at okfn.org" target=3D"_=
blank">john.glover at okfn.org</a>&gt;</span><br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div dir=3D"ltr">Hi Bert,<div><br></div><div=
>Yes I tested both create and update (both of which go through the same RDF=
 validator).</div>
<div><br></div><div>Ok thanks for the info.</div><div><br></div><div>Regard=
s,</div><div>

John</div></div><div class=3D"HOEnZb"><div class=3D"h5"><div class=3D"gmail=
_extra"><br><br><div class=3D"gmail_quote">On 12 November 2013 11:34, Bert =
Van Nuffelen <span dir=3D"ltr">&lt;<a href=3D"mailto:bert.van.nuffelen at tenf=
orce.com" target=3D"_blank">bert.van.nuffelen at tenforce.com</a>&gt;</span> w=
rote:<br>

<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div dir=3D"ltr"><div><div><div><div><div>Hi=
 John,<br><br></div>strange. I will create a full dump of the request to se=
e the content. Did you do a create and an update?<br>

<br></div>best,<br><br></div>Bert<br><br></div>ps. we have an power interru=
pt from 12h (in half an hour) till 13h. So there might be communication gli=
tches.<br>
<br></div>Bert<br><div><div><div><div><br></div></div></div></div></div><di=
v><div><div class=3D"gmail_extra"><br><br><div class=3D"gmail_quote">2013/1=
1/12 John Glover <span dir=3D"ltr">&lt;<a href=3D"mailto:john.glover at okfn.o=
rg" target=3D"_blank">john.glover at okfn.org</a>&gt;</span><br>

<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div dir=3D"ltr">Hi Bert,<div><br></div><div=
>I am unable to reproduce your error here. Both of those JSON files are imp=
orted fine for me using a simple Python client to make an API request (rele=
ase 01), neither produce the error. Could you therefore please double check=
 that your API request is correct, and let me know when the test server is =
back to release 01 so that we can test it again there.</div>

<div><br></div><div>I have pushed a change to ckanext-ecportal so it will n=
ow raise a ValidationError if it cannot parse the RDF in the dataset JSON a=
s requested.</div><div><br></div><div>Regards,</div><div>John</div></div>

<div><div>

<div class=3D"gmail_extra"><br><br><div class=3D"gmail_quote">On 12 Novembe=
r 2013 09:44, John Glover <span dir=3D"ltr">&lt;<a href=3D"mailto:john.glov=
er at okfn.org" target=3D"_blank">john.glover at okfn.org</a>&gt;</span> wrote:<b=
r><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:=
1px #ccc solid;padding-left:1ex">

<div dir=3D"ltr">Hi Bert,<div><br></div><div>Thanks, I will have a look at =
that JSON file and get back to you.</div><div><br></div><div>Regards,</div>=
<div>John</div></div><div><div><div class=3D"gmail_extra">

<br><br><div class=3D"gmail_quote">
On 12 November 2013 09:08, Bert Van Nuffelen <span dir=3D"ltr">&lt;<a href=
=3D"mailto:bert.van.nuffelen at tenforce.com" target=3D"_blank">bert.van.nuffe=
len at tenforce.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote"=
 style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">

<div dir=3D"ltr"><div><div>Hi John,<br><br></div>I forgot to add the messag=
e that I attached example json&#39;s=C2=A0 to the previous respons.<span><f=
ont color=3D"#888888"><br><br></font></span></div><span><font color=3D"#888=
888">Bert<br>

</font></span></div><div><div><div class=3D"gmail_extra"><br><br><div class=
=3D"gmail_quote">2013/11/12 Bert Van Nuffelen <span dir=3D"ltr">&lt;<a href=
=3D"mailto:bert.van.nuffelen at tenforce.com" target=3D"_blank">bert.van.nuffe=
len at tenforce.com</a>&gt;</span><br>

<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div dir=3D"ltr">Hi John,<br><br><div class=
=3D"gmail_extra"><br><div class=3D"gmail_quote"><div>2013/11/11 John Glover=
 <span dir=3D"ltr">&lt;<a href=3D"mailto:john.glover at okfn.org" target=3D"_b=
lank">john.glover at okfn.org</a>&gt;</span><br>

<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">
<div dir=3D"ltr">Hi Bert,<div><br></div><div>The last time this issue came =
up, the only way that we could recreate the problem was when making a reque=
st without setting the content type to &#39;application/json&#39;. As you a=
re now doing this, could you please send us an example of the JSON that you=
 are sending to CKAN in a failing request so that we can investigate furthe=
r.</div>

<div>

<div><br></div></div></div></blockquote><div><br>=C2=A0</div><blockquote cl=
ass=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;p=
adding-left:1ex"><div dir=3D"ltr"><div><div></div><div>&gt; When I do my te=
sts, it seems that in case the error occurs the submitted RDF is not saved,=
 and hence the RDF in CKAN is=C2=A0</div>

<div>&gt; that one of the CKAN RDF generation process. Is that true?</div>

<div><br></div></div><div>Yes this is correct.</div><div><div><br></div><di=
v>&gt; In that case the level of the message ERROR and the result by the AP=
I call return to it are not in sync. On an ERROR I=C2=A0</div><div>
&gt; expect that the api call return an error code. If that was a warn I ca=
n understand that the api call returns success.<br>

</div><div><br></div></div><div>We could of course return a validation erro=
r here (although this has not been requested before). I believe the thinkin=
g was that it was best to save the data and ignore this error as CKAN can g=
enerate RDF, but this logic could be changed.</div>

</div></blockquote><div><br></div></div><div>I understand the reasoning, bu=
t it results in an incorrect end-result. In this case it even has hidden a =
communication error between RDF2CKAN and CKAN. Since the api provides a cor=
rect feedback mechanism I would like to rely on that.<br>

<br><br><br><br></div><div><div><div>=C2=A0</div><blockquote class=3D"gmail=
_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:=
1ex"><div dir=3D"ltr">

<div><br></div><div>Regards,</div><div>John</div></div><div><div><div class=
=3D"gmail_extra"><br><br><div class=3D"gmail_quote">On 8 November 2013 11:4=
7, Bert Van Nuffelen <span dir=3D"ltr">&lt;<a href=3D"mailto:bert.van.nuffe=
len at tenforce.com" target=3D"_blank">bert.van.nuffelen at tenforce.com</a>&gt;<=
/span> wrote:<br>

<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div dir=3D"ltr"><div><div>Hi,<br><br></div>=
one addition, I run overnight with the same RDF2CKAN and the same ESTAT pac=
kage an upload on the release 09.00.0x version and there in the CKAN logs t=
he above message is not=C2=A0 present.<br>

<br></div><div>But when I do the synchronisation with virtuoso then I get e=
xceptions on parsing the RDF files as they have been converted by CKAN.<br>=
</div><div><br>So it seems that <br>=C2=A0=C2=A0 * in release 09.00.0x the =
check on this issue was not present at all,<br>

</div><div>=C2=A0=C2=A0 * in release 01.00.0x the check is there, but is si=
lently replaces it with an empty value<br><br></div><div>In both releases t=
he accept-header is not bound to the desired execution.<br><br></div><div>k=
ind regards,<br>

<br></div><div>Bert<br></div><div><br></div><div><br></div></div><div><div>=
<div class=3D"gmail_extra"><br><br><div class=3D"gmail_quote">2013/11/7 Ber=
t Van Nuffelen <span dir=3D"ltr">&lt;<a href=3D"mailto:bert.van.nuffelen at te=
nforce.com" target=3D"_blank">bert.van.nuffelen at tenforce.com</a>&gt;</span>=
<br>

<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div dir=3D"ltr"><div><div><div><div>Dear al=
l,<br><br></div>We have again this problem. This is related to the issue of=
 ODP-294<br>

<br>in the apache logs:<br><br>[Tue Nov 05 21:16:57 2013] [error] 2013-11-0=
5 21:16:57,306 ERROR [ckanext.ecportal.rdfutil] EntityRef: expecting &#39;;=
&#39;, line 39, column 103<br>
<br></div><div>in the CKAN logs<br><br>2013-11-07 22:14:04,975 ERROR [ckane=
xt.ecportal.rdfutil] EntityRef: expecting &#39;;&#39;, line 39, column 102<=
br><br></div><div></div>As this is silently in the logs instead of a _prope=
r_ validation error like:<br>

<br>2013-11-07 22:09:21,996 ERROR [ckan.controllers.api] Validation error: =
&#39;{\&#39;keyword_string\&#39;: [u\&#39;Tag &quot;inter<br>national trade=
,trade statistics&quot; must be alphanumeric characters or symbols: -_.\&#3=
9;], \&#39;__type\&#39;: \&#39;Validati<br>

on Error\&#39;}&#39;<br><br></div>rdf2ckan can only conclude it has inserte=
d the data correctly while it is not. <br></div>So the outcome of rdf2ckan =
is hence untrust whorthy. For that reason this issue remained unseen by TF =
development.<br>

<div><br></div><div>Secondly I am concerned about the fact if the above con=
figuration solution described by David is working. <br></div><div>It seems =
that the setting of the header to &quot;application/json&quot; does not res=
ult in the correct handling by CKAN.<br>

</div><div>Here under is the extract of the java code showing that we sent =
the content-type as requested.<br></div><div><div><div><br>private static f=
inal String CONTENT_JSON =3D &quot;application/json&quot;;<br><br>public Ma=
p&lt;String, Object&gt; post(String json, String method) {<br>

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 if (json.isEmpty() || method.isE=
mpty())<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0 return null;<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 DefaultHttpC=
lient httpClient =3D new DefaultHttpClient();<br><br>=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0 HttpPost httpPost =3D requestCredentials(method);<br>=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 json =3D json.replaceAll(&quot;;=
&quot;, &quot;&quot;);<br>

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 httpPost.setEntity(streamify(jso=
n, CONTENT_JSON));<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ...<br><br>=C2=
=A0public InputStreamEntity streamify(String json, String contentType) {<br=
>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 json =3D json.replaceAll(&quot;=
;&quot;, &quot;&quot;);<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Input=
StreamEntity inputStreamEntity =3D new InputStreamEntity(new ByteArrayInput=
Stream(json.getBytes(Charset.forName(&quot;UTF-8&quot;))), json.getBytes().=
length);<br>

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 inputStreamEntity.setContentType=
(contentType);<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 return inputSt=
reamEntity;<br>=C2=A0=C2=A0=C2=A0 }<br><br><br></div><div>Note that the sub=
mitted RDF is all valid rdf and XML in the request body: it passes through =
all XML &amp; RDF validations (e.g. rapper).<br>

<br></div><div>When I do my tests, it seems that in case the error occurs t=
he submitted RDF is not saved, and hence the RDF in CKAN is that one of the=
 CKAN RDF generation process. Is that true? In that case the level of the m=
essage ERROR and the result by the API call return to it are not in sync. O=
n an ERROR I expect that the api call return an error code. If that was a w=
arn I can understand that the api call returns success.<br>

<br></div><div>So the next actions should be taken:<br><br></div><div>1) ma=
ke the above error message a proper validation result. <br></div><div>=C2=
=A0=C2=A0=C2=A0 Note this solves also the problem of the meaningless error =
message. There is no reference at all in the message to know about which ca=
ll/object it is. So the problem is unresolvable with this message. And putt=
ing CKAN in debug mode logging for that is not very sensible.<br>

</div><div>2) ensure that the handling of RDF (in rdfxml-format) inside the=
 json is treated properly and not interpreted as an html encoding.<br></div=
><div><br></div><div>best regards,<br><br></div><div>Bert<br></div><div>

<br></div><div><div><div><div><br></div></div></div></div></div></div></div=
><div class=3D"gmail_extra"><div><div><br><br><div class=3D"gmail_quote">20=
13/9/18 David Raznick <span dir=3D"ltr">&lt;<a href=3D"mailto:david.raznick=
@okfn.org" target=3D"_blank">david.raznick at okfn.org</a>&gt;</span><br>

<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div dir=3D"ltr"><div><div><div><div>Hello<b=
r><br></div>It looks like the error is because the wrong content type is se=
nt when posting to ckan in rdf2ckan.=C2=A0=C2=A0 Content-Type:application/j=
son should be sent otherwise CKAN will think it is urlencoded and therefore=
 make the final document encoded wrongly.<br>

<br></div>This must have changed at somepoint, as the live db does not have=
 this issue and is likely what caused the unicode issues from happening too=
.<br><br></div>Thanks<span><font color=3D"#888888"><br><br>
</font></span></div><span><font color=3D"#888888">David<br></font></span></=
div><div><div><div class=3D"gmail_extra">
<br><br><div class=3D"gmail_quote">On 18 September 2013 16:16, Dimitrios Me=
xis <span dir=3D"ltr">&lt;<a href=3D"mailto:dimitrios.mexis at tenforce.com" t=
arget=3D"_blank">dimitrios.mexis at tenforce.com</a>&gt;</span> wrote:<br><blo=
ckquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #c=
cc solid;padding-left:1ex">

 =20
   =20
 =20
  <div text=3D"#000000" bgcolor=3D"#FFFFFF">
    <div>Hello David,<br>
      <br>
      we don&#39;t do any postprocessing for the data we send.<br>
      <br>
      It&#39;s pure rdf content as read from the file. So, we depend on CKA=
N
      how it will digest it.<br>
      If we need to do some postprocessing, can you give clarification
      on the matter ?<br>
      <br>
      Also, does that have to do with the problem we faced with unicode
      exceptions from CKAN as well ?<br>
      <br>
      Kind regards<br>
      Dimitrios<div><div><br>
      <br>
      On 18/09/2013 17:12, David Raznick wrote:<br>
    </div></div></div>
    <blockquote type=3D"cite"><div><div>
      <div dir=3D"ltr">
        <div>Hello Bert<br>
          <br>
          <br>
        </div>
        <div>There appears to be a lot of malformed rdf documents in the
          test system e.g<br>
          <br>
          <a href=3D"http://212.71.25.148/en/data/dataset/BrvXA5sZQ1AFKgE4P=
ktw.rdf" target=3D"_blank">http://212.71.25.148/en/data/dataset/BrvXA5sZQ1A=
FKgE4Pktw.rdf</a><br>
          <br>
        </div>
        <div>After some investigation this seems to due to the data send
          by rdf2ckan.=C2=A0 It appears that xml &quot;&amp;&quot; are not =
being
          escaped correctly in some places. <br>
          <br>
        </div>
        <div>These errors do not appear on the live db as far as I can
          see, could you please look into this.<br>
          <br>
        </div>
        <div>Thanks<br>
          <br>
          David<br>
        </div>
      </div>
      <br>
      <fieldset></fieldset>
      <br>
      </div></div><pre>_______________________________________________
Ecodp-dev mailing list
<a href=3D"mailto:Ecodp-dev at lists.okfn.org" target=3D"_blank">Ecodp-dev at lis=
ts.okfn.org</a>
<a href=3D"http://lists.okfn.org/mailman/listinfo/ecodp-dev" target=3D"_bla=
nk">http://lists.okfn.org/mailman/listinfo/ecodp-dev</a>
</pre>
    </blockquote>
    <br>
  </div>

<br>_______________________________________________<br>
Ecodp-dev mailing list<br>
<a href=3D"mailto:Ecodp-dev at lists.okfn.org" target=3D"_blank">Ecodp-dev at lis=
ts.okfn.org</a><br>
<a href=3D"http://lists.okfn.org/mailman/listinfo/ecodp-dev" target=3D"_bla=
nk">http://lists.okfn.org/mailman/listinfo/ecodp-dev</a><br>
<br></blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
Ecodp-dev mailing list<br>
<a href=3D"mailto:Ecodp-dev at lists.okfn.org" target=3D"_blank">Ecodp-dev at lis=
ts.okfn.org</a><br>
<a href=3D"http://lists.okfn.org/mailman/listinfo/ecodp-dev" target=3D"_bla=
nk">http://lists.okfn.org/mailman/listinfo/ecodp-dev</a><br>
<br></blockquote></div><br><br clear=3D"all"><br></div></div><span><font co=
lor=3D"#888888">-- <br>Bert Van Nuffelen<br><br>Semantic Technologies Softw=
are Architect at TenForce<br><a href=3D"http://www.tenforce.be" target=3D"_=
blank">www.tenforce.be</a><br>

<br><a href=3D"mailto:Bert.Van.Nuffelen at tenforce.com" target=3D"_blank">Ber=
t.Van.Nuffelen at tenforce.com</a><br>
Office: <a href=3D"tel:%2B32%20%280%2916%2031%2048%2060" value=3D"+32163148=
60" target=3D"_blank">+32 (0)16 31 48 60</a><br>Mobile:<a href=3D"tel:%2B32=
%20479%2006%2024%2026" value=3D"+32479062426" target=3D"_blank">+32 479 06 =
24 26</a><br>

skype: bert.van.nuffelen
</font></span></div>
</blockquote></div><br><br clear=3D"all"><br>-- <br>Bert Van Nuffelen<br><b=
r>Semantic Technologies Software Architect at TenForce<br><a href=3D"http:/=
/www.tenforce.be" target=3D"_blank">www.tenforce.be</a><br><br><a href=3D"m=
ailto:Bert.Van.Nuffelen at tenforce.com" target=3D"_blank">Bert.Van.Nuffelen at t=
enforce.com</a><br>

Office: <a href=3D"tel:%2B32%20%280%2916%2031%2048%2060" value=3D"+32163148=
60" target=3D"_blank">+32 (0)16 31 48 60</a><br>Mobile:<a href=3D"tel:%2B32=
%20479%2006%2024%2026" value=3D"+32479062426" target=3D"_blank">+32 479 06 =
24 26</a><br>

skype: bert.van.nuffelen
</div>
</div></div><br>_______________________________________________<br>
Ecodp-dev mailing list<br>
<a href=3D"mailto:Ecodp-dev at lists.okfn.org" target=3D"_blank">Ecodp-dev at lis=
ts.okfn.org</a><br>
<a href=3D"http://lists.okfn.org/mailman/listinfo/ecodp-dev" target=3D"_bla=
nk">http://lists.okfn.org/mailman/listinfo/ecodp-dev</a><br>
<br></blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
Ecodp-dev mailing list<br>
<a href=3D"mailto:Ecodp-dev at lists.okfn.org" target=3D"_blank">Ecodp-dev at lis=
ts.okfn.org</a><br>
<a href=3D"http://lists.okfn.org/mailman/listinfo/ecodp-dev" target=3D"_bla=
nk">http://lists.okfn.org/mailman/listinfo/ecodp-dev</a><br>
<br></blockquote></div></div></div><div><div><br><br clear=3D"all"><br>-- <=
br>Bert Van Nuffelen<br><br>Semantic Technologies Software Architect at Ten=
Force<br><a href=3D"http://www.tenforce.be" target=3D"_blank">www.tenforce.=
be</a><br>

<br><a href=3D"mailto:Bert.Van.Nuffelen at tenforce.com" target=3D"_blank">Ber=
t.Van.Nuffelen at tenforce.com</a><br>
Office: <a href=3D"tel:%2B32%20%280%2916%2031%2048%2060" value=3D"+32163148=
60" target=3D"_blank">+32 (0)16 31 48 60</a><br>Mobile:<a href=3D"tel:%2B32=
%20479%2006%2024%2026" value=3D"+32479062426" target=3D"_blank">+32 479 06 =
24 26</a><br>

skype: bert.van.nuffelen
</div></div></div></div>
</blockquote></div><br><br clear=3D"all"><br>-- <br>Bert Van Nuffelen<br><b=
r>Semantic Technologies Software Architect at TenForce<br><a href=3D"http:/=
/www.tenforce.be" target=3D"_blank">www.tenforce.be</a><br><br><a href=3D"m=
ailto:Bert.Van.Nuffelen at tenforce.com" target=3D"_blank">Bert.Van.Nuffelen at t=
enforce.com</a><br>

Office: <a href=3D"tel:%2B32%20%280%2916%2031%2048%2060" value=3D"+32163148=
60" target=3D"_blank">+32 (0)16 31 48 60</a><br>Mobile:<a href=3D"tel:%2B32=
%20479%2006%2024%2026" value=3D"+32479062426" target=3D"_blank">+32 479 06 =
24 26</a><br>

skype: bert.van.nuffelen
</div>
</div></div><br>_______________________________________________<br>
Ecodp-dev mailing list<br>
<a href=3D"mailto:Ecodp-dev at lists.okfn.org" target=3D"_blank">Ecodp-dev at lis=
ts.okfn.org</a><br>
<a href=3D"http://lists.okfn.org/mailman/listinfo/ecodp-dev" target=3D"_bla=
nk">http://lists.okfn.org/mailman/listinfo/ecodp-dev</a><br>
<br></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
Ecodp-dev mailing list<br>
<a href=3D"mailto:Ecodp-dev at lists.okfn.org" target=3D"_blank">Ecodp-dev at lis=
ts.okfn.org</a><br>
<a href=3D"http://lists.okfn.org/mailman/listinfo/ecodp-dev" target=3D"_bla=
nk">http://lists.okfn.org/mailman/listinfo/ecodp-dev</a><br>
<br></blockquote></div><br><br clear=3D"all"><br>-- <br>Bert Van Nuffelen<b=
r><br>Semantic Technologies Software Architect at TenForce<br><a href=3D"ht=
tp://www.tenforce.be" target=3D"_blank">www.tenforce.be</a><br><br><a href=
=3D"mailto:Bert.Van.Nuffelen at tenforce.com" target=3D"_blank">Bert.Van.Nuffe=
len at tenforce.com</a><br>

Office: <a href=3D"tel:%2B32%20%280%2916%2031%2048%2060" value=3D"+32163148=
60" target=3D"_blank">+32 (0)16 31 48 60</a><br>Mobile:<a href=3D"tel:%2B32=
%20479%2006%2024%2026" value=3D"+32479062426" target=3D"_blank">+32 479 06 =
24 26</a><br>

skype: bert.van.nuffelen
</div>
</div></div><br>_______________________________________________<br>
Ecodp-dev mailing list<br>
<a href=3D"mailto:Ecodp-dev at lists.okfn.org" target=3D"_blank">Ecodp-dev at lis=
ts.okfn.org</a><br>
<a href=3D"http://lists.okfn.org/mailman/listinfo/ecodp-dev" target=3D"_bla=
nk">http://lists.okfn.org/mailman/listinfo/ecodp-dev</a><br>
<br></blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
Ecodp-dev mailing list<br>
<a href=3D"mailto:Ecodp-dev at lists.okfn.org">Ecodp-dev at lists.okfn.org</a><br=
>
<a href=3D"http://lists.okfn.org/mailman/listinfo/ecodp-dev" target=3D"_bla=
nk">http://lists.okfn.org/mailman/listinfo/ecodp-dev</a><br>
<br></blockquote></div><br><br clear=3D"all"><br>-- <br>Bert Van Nuffelen<b=
r><br>Semantic Technologies Software Architect at TenForce<br><a href=3D"ht=
tp://www.tenforce.be" target=3D"_blank">www.tenforce.be</a><br><br><a href=
=3D"mailto:Bert.Van.Nuffelen at tenforce.com" target=3D"_blank">Bert.Van.Nuffe=
len at tenforce.com</a><br>
Office: +32 (0)16 31 48 60<br>Mobile:+32 479 06 24 26<br>skype: bert.van.nu=
ffelen
</div>

--001a11335194b5839504eafc6c8f--