[openspending-dev] [okfn-labs] Data validation reporting

Thu Nov 20 16:31:41 UTC 2014

Hey, 

> The point is not Node-based ETL or not - you mentioned Python in your post, and I didn’t want to commit to a Python solution (or any language) while we are talking about needs and use cases. Python and Node were mentioned as as they are the most obvious candidates for myself.

Sure, I was just ranting a bit. I guess my use case starts with “As a Python developer, I want to…” ;)

> Can you point me to any examples of Stiivi’s work on “expected result detection” as you describe?

Check out: 

https://github.com/Stiivi/brewery/blob/master/examples/audit_unknown_csv.py
https://github.com/Stiivi/brewery/blob/master/brewery/probes.py

Cheers, 

- Friedrich 

> 
> Thanks
> 
>> On 20 Nov 2014, at 12:44, Friedrich Lindenberg <friedrich at pudo.org> wrote:
>> 
>> Hey, 
>> 
>>> On 19 Nov 2014, at 19:46, Paul Walsh <paulywalsh at gmail.com> wrote:
>>>> Something I would really need is a stand-alone, light-weight structured logging system for ETL processes in Python.
>>> 
>>> That is def. in line with what we are aiming for here - something modular/pluggable rather than something tied to a specific validator. Not sure yet if Python or Node.
>> 
>> I struggle to see the point in Node-based ETL: the callbacks thing means it’s easy to just cram all the data down your pipeline at once if you don’t explicitly sequence it; and the programming language and libraries just aren’t good for data processing. No proper handling of dates and time, shitty NLP tools, even file handling is awkward. In short: I want to meet the people who would actually do this. And possibly bring them before the ICC.
>> 
>>>> That's the more general version of what we were doing with the UK25k stuff a few years ago, and it would not overlap with CSVLint. Features: 
>>>> 
>>>> * Be able to log structured data from ETL processes
>>>> * Generate custom, jinja2-based reports from them
>>>> * Have a set of pre-defined gauges to test for stuff like null values, extreme values etc. 
>>>> * Have an emailer for certain really nasty events 
>>> 
>>> By gauges do you mean something beyond a schema, and more like:
>>> 
>>> *expect values in col Y to be in range X
>>> * if more than X% of NULL in col Y, do something (warning after validation, etc)
>>> 
>>> Let's be explicit here as as it sounds interesting and quite useful for large datasets.
>> 
>> Yep, that sort of stuff. It’s a notion that Stiivi had been exploring in earlier versions of this ETL framework, and it seems really cool to me. 
>> 
>> - Friedrich
> 