[OpenSpending-discuss] Anyone willing to help with getting basic data into openspending?

Thu Nov 3 15:38:12 UTC 2011

On 2011-11-03 12:27, Lucy Chambers wrote:
> Dear Michal,
>
> We would be very happy to help indeed. We actually regularly run
> drop-in sessions to help people out, the next one will be next
> Thursday, 10th November at 6pm UK time (via Skype) - would you be able
> to make it?
>
> If you are able to make it, could you add your name and Skype ID to
> the pad below?:
>
>   http://wdmmg.okfnpad.org/community-2011-11-10
>
> In the meantime, perhaps we can kill two birds with one stone? i.e.
> help you and improve the documentation...
Ok,
I will go through my process:

http://wiki.openspending.org/Data_Format
Section Required data says about Date: "This is generally expected to be 
a single year.", but the next section Column formatting says: "Dates 
have to be in the format YYYY-MM-DD" So, is it "2010" or "2010-12-31", 
or both?

http://wiki.openspending.org/Model_Format
(that's the part where I begin to be lost)
Section Model format
"The model must be a valid JSON file, which contains a JSON object with 
"dataset" at its root "
But the example shows 'dataset', 'mapping' and 'views' at the same level.

Section Dataset metadata: I have realized just now, during 3rd reading, 
that there is an array required for unique_keys. A simple example below 
the definition would be great. (It might be a trouble that I am not used 
to json-s (I generate them from php), so one overlooks something easily.)

Section View definitions: I do not get this at all. Paragraph 'To 
explain' is an example, however the sample code is different - includes 
names 'function', 'subfunction', which I cannot imagine what they shall 
mean in this context; breakdown and filters - no idea how to construct 
them from my data.

http://wiki.openspending.org/Mapping_Format
(that's the part I get really lost)
Section Mappings
"There exist four mandatory dimensions", but in 
http://wiki.openspending.org/Data_Format, section Required data, it says 
only two columns are required: Date and Amount (how can it be that 'to' 
and 'from' are not required in data format/csv file?)
What are required information about any column in the db? The example 
shows 6 of them (type, description, label, datatype, default_value, column).
What is "default_value" for, if the 
http://wiki.openspending.org/Data_Format, section Columns says "each 
cell below the heading must be non-empty" ?

Section Types->Column types: what is the type for "year" (e.g. value 
"2010") - float or string (or date?) ? As there is no "integer" there.
What is 'currency' for, if the currency (e.g. 'CZK') is specified in 
"dataset" part of the model.

Section Field descriptions
What are fields? They come from nowhere, there are no fields in the 
first main example (="Overall, a mapping resembles the following") What 
is the difference between 'column' and 'fields', there are 'column's in 
some examples and 'fields' in others.

This is only partly about the documentation, but the process continues:
http://etl.sandbox.openspending.org/load/preflight/aris-test-2010
Why are there 3 URLs? Are all three required? 
http://wiki.openspending.org/Preparing_Datasets section Upload the data 
and the model to CKAN says "OpenSpending expects your CKAN package to 
reference (at least) two files", the whole documentation speak only 
about these two files (with 'mapping' required in the json file, see 
http://wiki.openspending.org/Model_Format section Model format). What 
about the "model:mapping URL" ? It is just confusing.

What I am really missing:
One single simple example which would be used consistently through the 
whole documentation 
(e.g.,http://sandbox.openspending.org/dataset/openspending-example, I'd 
prefer to have two years, not one)

And for my own dataset:
I get: These errors were found when attempting to validate your model:
   - 'model.mapping.time' field had error 'Required'
And the views (their definition) make no sense to me so far (how to 
define the hierarchy 'chapter'->'organization' as in  
http://test.kohovolit.sk/m/chart_3.html)

Best,
Michal

> We have some brilliant data-wranglers on this list, could you outline
> how far you have got and where the sticking point is?
>
> A quick glance at your data suggests it is in the right format and
> pretty clean, so this shouldn't take too much effort!
>
> All the best,
>
> Lucy
>
>
> On Thu, Nov 3, 2011 at 3:34 AM, M.Skop KohoVolit.eu
> <michal.skop at kohovolit.eu>  wrote:
>> Hi,
>>
>> we would like to use OS for very detailed Czech budget data, but I am
>> fighting with the simple task of getting basic trial data into OS. ( I must
>> confess that the documentation is often unclear to me, this was my 2nd
>> attempt, different data, and still no results.)
>>
>> Isn't there anybody wiling to help us with the task? I believe if I can get
>> a simple dataset running, it shall be the same for the more complex one.
>>
>> What I want to achieve by this trial is shown here (the same data):
>> http://test.kohovolit.sk/m/chart_3.html
>> One year, only a simple hierarchy 'chapter'(=group of organizations) ->
>> 'organization', nothing more for the beginning.
>>
>> The data is here:
>> http://thedatahub.org/dataset/aris-test-2010
>> https://raw.github.com/michalskop/BudovaniStatu.cz/master/dev/os1.csv
>> https://raw.github.com/michalskop/BudovaniStatu.cz/master/dev/os1.json
>> (which is probably the trouble)
>>
>> Thanks a lot,
>> Michal
>>
>> --
>> Mgr. Michal Škop, Ph.D.
>> KohoVolit.eu
>> michal.skop at kohovolit.eu
>> +420 775 187 021
>>
>>
>> _______________________________________________
>> wdmmg-discuss mailing list
>> wdmmg-discuss at lists.okfn.org
>> http://lists.okfn.org/mailman/listinfo/wdmmg-discuss
>>
>
>

-- 
Mgr. Michal Škop, Ph.D.
KohoVolit.eu
michal.skop at kohovolit.eu
+420 775 187 021