[wdmmg-dev] dfig loader and dataset (DFIDv3.xls)

Fri Mar 11 18:10:19 UTC 2011

Hi Carsten!

On Fri, Mar 11, 2011 at 6:47 PM, Carsten Senger <senger at rehfisch.de> wrote:
> My current task is to update the dfig loader. wdmmgext.load.dfig tries to
> get the data from DFIDv3.xls. There is a dfid package in ckan
> (<http://ckan.net/package/ukgov-dfid-project-db>), but that contains
> different data. dataDFIDv3.xls does not seem to be available online, e.g.
> from the dfig site.

Yes, that file does not seem to be online as of yet - I found it on
the old WDMMG production server, eu2.okfn.org. Perhaps you could
upload it to http://storage.ckan.net/openspending ("new label") and
reference it from the relevant CKAN package?

> Friedrich mentioned that Nick (?) is the brain behind
> it and working on a bigger dataset atm. Is that right?

Sorry, that must have been a misunderstanding, Nick is doing the EPSRC
thing (UK research funding) - DFID was an earlier project and I'm not
exactly sure who was behind it.

> wdmmgext.load.utils
> This package contains helper functions that still contain sqlalchemy
> related code:
> * get_or_create_dataset
> * get_or_create_key
> * get_or_create_value
>
> These functions are used in the gla and ordnance loaders which fail. Should
> they be updated too.

Over time, I guess so. GLA, as far as I understand, is part of the UK
local spending data for which there is a larger scheme involving
OpenlyLocal in the making :-) But for now, I think its OK not to
re-produce these two. Nevertheless, cleaning out util.py would be a
great thing to do!

- Friedrich