[wdmmg-discuss] Request for scraping and keeping in closer contact

Donovan Hide donovanhide at gmail.com
Fri May 28 22:10:31 UTC 2010


Hi Rufus, Alistair,

I've attached a patch which might help with the integration of
Scraperwiki with CKAN. It basically adds to the CKAN API:

http://ckan.net/api/rest/resource

which returns something like:

{
    "http://scraperwiki.com/scrapers/show/welsh-primary-schools-performance-data/":
[
        "http://ckan.net/package/income_tax",
        "http://ckan.net/package/schools"
    ],
    "http://scraperwiki.com/scrapers/export/nottingham-council-schools/": [
        "http://ckan.net/package/schools"
    ],
    "http://scraperwiki.com/scrapers/export/income-tax-rates/": [
        "http://ckan.net/package/income_tax"
    ],
    ......
}

This would enable some jQuery on ScraperWiki to run live queries
against CKAN to check if a scraper is referenced on the site. If it
seems like a good idea, it might make sense to make a POST version
which only returns a single record. Let me know any thoughts.

Cheers,
Donny,

On 28 May 2010 11:13, Alistair Turnbull <apt1002 at goose.minworks.co.uk> wrote:
> On Thu, 27 May 2010, Rufus Pollock wrote:
>
>> On 27 May 2010 17:40, Donovan Hide <donovanhide at gmail.com> wrote:
>>>
>>> Hi Rufus,
>>>
>>> this scraper has united the income tax rates from these two spreadsheets:
>>>
>>> http://www.hmrc.gov.uk/stats/tax_structure/incometaxrates_1974to1990.xls
>>>
>>> http://www.hmrc.gov.uk/stats/tax_structure/table-a2a.xls
>>
>> Great work. For others (just cc'd this to wdmmg-discuss), scraped data is
>> here:
>>
>> <http://scraperwiki.com/scrapers/show/income-tax-rates/>
>>
>>> and should be useful for visualisations. Have also done some analysis
>>> of the income tax personal and married allowances and come to the
>>> conclusion that the data is almost useless without all the explanatory
>>> notes. It seems that no useful function would be served by flattening
>>> the data into a key value store, although I could be wrong, if you had
>>> a particular plan for that time series.
>>
>> That sounds probable. My guess is also that taking those allowances is
>> very second-order ....
>>
>>> Perhaps just scraping the personal allowance and ignoring the married
>>> couple's allowance would be useful? They are two different things
>>> completely, but both depend on the same age criteria.
>>
>> I think think we can hold off on this just for the moment -- there may
>> be more important things from a tax calculator perspective to work
>> out.
>
>> @Alistair: do you have any view on this?
>
> I agree with you both: hold off on this.
>
>        Alistair
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: donny.patch
Type: text/x-patch
Size: 11463 bytes
Desc: not available
URL: <http://lists.okfn.org/pipermail/openspending/attachments/20100528/51db662a/attachment-0001.bin>


More information about the openspending mailing list