[CKAN-support] IATI: CSV upload not working

Tue Feb 25 17:01:21 UTC 2014

OK, that took longer than expected to solve...

For some reason some datasets were missing resources, and this caused
issues on the database that eventually triggered the exception on the
CSV import. I suspect this may have been caused during the migration
to CKAN 2 but I haven't been able to reproduce it on the staging site
or locally. I've manually fixed the offending datasets and the CSV
import worked fine:

http://i.imgur.com/Zdtk7sv.png

Feel free to repeat it if you want to make sure.

To prevent this kind of issues, I added another validator that ensures
that every dataset created or edited has a resource (file URL).

I'll keep an eye on the logs and alerts to see if this comes up again.
Sorry for the hassle.

Adrià

On 25 February 2014 11:05, Steven Flower <Steven.Flower at devinit.org> wrote:
> Thanks Adria
>
> Only think I could spot was that Georgia didnt have a file URL prior to
> the CSV upload
>
> Thanks
> Steven
>
> On 25/02/14 10:45, Adrià Mercader wrote:
>> Hi,
>>
>> I see, I've replicated it too. It is unrelated to the previous
>> encoding issues and it only happens on the production site. I'll
>> investigate what's wrong with the Georgia file
>>
>> Adrià
>>
>> On 25 February 2014 08:18, Steven Flower <Steven.Flower at devinit.org> wrote:
>>> Hi Adria
>>>
>>> Thanks
>>>
>>> But - the issue seems to be back / remain
>>>
>>> Attached is a CSV file that UNOPS have tried to upload.  It looks like
>>> the CSV file updates up to Gambia, but then stops/ provides a server
>>> error.
>>>
>>> I've replicated this.  I've also tried to make smaller CSV files, but
>>> still get a server error eventually.  Georgia was the record where the
>>> original problem seemed to occur, but I cant understand why.
>>>
>>> Thanks
>>>
>>> Steven
>>>
>>> On Mon 24 Feb 2014 14:23:31 GMT, AdriÃ  Mercader wrote:
>>>> Hi,
>>>>
>>>> There was an issue with the fix I deployed a while ago. I've now
>>>> double-checked and pushed the relevant patch. The import worked fine
>>>> for me on the staging site, let me know if you still have problems.
>>>>
>>>> Best,
>>>>
>>>> Adrià
>>>>
>>>> On 24 February 2014 12:49, Adam McGreggor <adam.mcgreggor at okfn.org> wrote:
>>>>> Thanks...
>>>>>
>>>>> Begin forwarded message:
>>>>>
>>>>> From: Steven Flower <Steven.Flower at devinit.org>
>>>>> Subject: [CKAN-support] IATI: CSV upload not working
>>>>> Date: 21 February 2014 17:54:38 GMT
>>>>> To: "support at ckan.org" <support at ckan.org>
>>>>> Cc: Matt Bartlett <Matt.Bartlett at devinit.org>
>>>>> Message-Id: <7C344E2C1520744787BC41F5934FACAA12F10B38 at DIPR-MX01.DIPR.local>
>>>>> X-Beenthere: ckan-support at lists.okfn.org
>>>>>
>>>>> Hi
>>>>>
>>>>> I tried to upload a CSV on behalf of UNDP
>>>>>
>>>>> Some records were populated
>>>>> (http://iatiregistry.org/publisher/activity/undp) but the remainder
>>>>> (attached) were not.  On trying this again, I now get a immediate
>>>>>
>>>>> Server Error
>>>>>
>>>>> An internal server error occurred
>>>>>
>>>>> Thanks
>>>>>
>>>>> Steven
>>>>>
>>>>> ______________________________________________________________________
>>>>> This email has been scanned by the Symantec Email Security.cloud service.
>>>>> For more information please visit http://www.symanteccloud.com
>>>>> ______________________________________________________________________
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> ckan-support mailing list
>>>>> ckan-support at lists.okfn.org
>>>>> https://lists.okfn.org/mailman/listinfo/ckan-support
>>>>>
>>>>>
>>>>>
>>>>>
>>>> ______________________________________________________________________
>>>> This email has been scanned by the Symantec Email Security.cloud service.
>>>> For more information please visit http://www.symanteccloud.com
>>>> ______________________________________________________________________
>>>>
>>> ______________________________________________________________________
>>> This email has been scanned by the Symantec Email Security.cloud service.
>>> For more information please visit http://www.symanteccloud.com
>>> ______________________________________________________________________
>> ______________________________________________________________________
>> This email has been scanned by the Symantec Email Security.cloud service.
>> For more information please visit http://www.symanteccloud.com
>> ______________________________________________________________________
>>
>
>
> ______________________________________________________________________
> This email has been scanned by the Symantec Email Security.cloud service.
> For more information please visit http://www.symanteccloud.com
> ______________________________________________________________________