[openspending-dev] Currency normalisation

Thu May 30 23:47:14 UTC 2013

On 27 May 2013 22:40, Tryggvi Björgvinsson <tryggvi.bjorgvinsson at okfn.org>wrote:

>  On þri 14.maí 2013 23:04, Tryggvi Björgvinsson wrote:
>
> On a related note. We probably have to collect some CPI (consumer price
> index) data (or similar data) for all countries (at least all supported
> currencies) in order to find value across time.
>
>
> I've made some headway with consumer price indexes. Rufus told me how to
> get data into data.okfn.org (using data packages) and I've now created a
> data package with the annual CPI collected by the World Bank:
> https://github.com/datasets/cpi
>
> We can now use this data package to make use of this data in OpenSpending.
> CPI helps us compute inflation (and thus compute the present value of an
> amount) so we can start making historical comparisons (and we're one big
> step closer to international comparisons since this allows us to normalise
> the currency). The reason I chose CPI data and not inflation data (which
> the World Bank also collects) is that inflation is measured with last year
> as reference while with CPI we can compute it irrespective of years using:
> (CPI_year - CPI_base_year)*100/CPI_base_year
>
> Now this brings up the question of how to use the data in OpenSpending. We
> can either import it directly into OpenSpending or use it via an external
> service (create an API).
>
> I don't like importing the data into OpenSpending since although
> OpenSpending will use this data, it's not the core data so imo we should go
> with the API approach.
>

I'm guessing this is  part of the ETL process rather than the display
process. As such the API versus local is less of a big deal and pulling
down locally at the start of the ETL script makes some sense.

> What I would want is a simple lookup API, e.g.
> http://api.com/cpi?year=2005&country=Ukraine
>

I'd just write this in code frankly as it will be much faster - turning the
CSV into a look up dictionary with 2 keys (year and country) would be
trivial.

>
> The problem this API needs to solve is what to return when the data isn't
> available. For example Zimbabwe doesn't have any CPI data since 2007. If
> I'd call:
> http://api.com/cpi?year=2011&country=Zimbabwe
> it should either give me a projected value (computed based on other years)
> or the closest year (2007). We could solve this by filling either of these
> values into the data and leave no gaps (and hope nobody will try to get
> data older than 1960).
>

Yes interpolation is important and mostly trivial (but sometimes not).

> Another possibility for the API would be to have an inflation service, e.g.
> http://api.com/inflation?base=2000&year=2010&country=Mexico
> which would simply compute and return the inflation based on the formula
> above. This isn't solvable by filling in some values (except if we set the
> base year as fixed or something as dirty or dirtier).
>
> The API service doesn't have to be something fancy. We could build a
> lightweight webapp for it, something that would read in the data (e.g. into
> Redis - since it's persisted in the data package) and host it somewhere to
> make the service available to others. We could then extend this if we
> collect currency exchange rates or purchasing power parity (PPP) when we
> start looking into international comparisons.
>

Just pull down the CSV and do it in code atm IMO.

BTW: as discussed I think ETL should move out of OpenSpending.org core code
asap but that's a different discussion.

> We might also just use datahub.io. It's running on CKAN so we can use the
> CKAN API to do datastore searches. We could even use the datastore sql
> search and perform the inflation computation via the sql query. This
> wouldn't exactly make the service reusable (well the module and it's
> configs or however we'd implement it could be copied and be reused since
> we're a free/open source project). The benefit of the CKAN API is that we
> don't have to host anything, just use it a build a wrapper.
>
> I'm interested in hearing your opinions. How would you like to make this
> data available to OpenSpending?
>

Rufus
*

*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/openspending-dev/attachments/20130531/e67b77a0/attachment.html>