[ckan-dev] Not all records loading to datastore

Mon Nov 23 03:22:14 UTC 2015

Hi, I have a csv file which has 31,989 rows of data.

The columns are all numeric data, but some of the cells have missing values, which the author has coded as NA (which is common when using R for analysis).  CKAN seems to be having trouble with the NA.

If I load up the original file, it begins pushing the data to datastore, but stops part way through, and the preview only shows 3750 records.  If you edit (manage) the resource and go to the DataStore tab, it shows this error:

Error: The data was invalid (for example: a numeric value is out of range or was inserted into a text field).
HTTP status code: 409
Response: {"help": "https://datastore.landcareresearch.co.nz/api/3/action/help_show?name=datastore_create", "success": false, "error": {"message": "The data was invalid (for example: a numeric value is out of r...
Requested URL: https://datastore.landcareresearch.co.nz/api/3/action/datastore_create

However, if I recode the NA as -9999, then the file uploads fine, and all 31,989 records are previewed.

Is there any way around this so we can retain NA for missing values (as used in R)?

Thanks
Aaron

________________________________

Please consider the environment before printing this email
Warning: This electronic message together with any attachments is confidential. If you receive it in error: (i) you must not read, use, disclose, copy or retain it; (ii) please contact the sender immediately by reply email and then delete the emails.
The views expressed in this email may not be those of Landcare Research New Zealand Limited. http://www.landcareresearch.co.nz