[openspending-dev] data.gov.uk spend browser

[Friedrich Lindenberg]

Hey,

(cc'ing list, this is innocent and relevant enough ;)

On Thu, Mar 28, 2013 at 9:32 PM, David Read
<david.read at hackneyworkshop.com>wrote:

> I've written a script to do the upload, but I get 403 "Sorry, you're
> not permitted to do this." error. I'm supplying my apikey in the
> header, but no luck. Can I get some help with this?
>

Hm, I'm not quite sure what is happening here, but for what its worth you
can also do HTTP Basic which may be helpful to find out whether the issue
is with permissions or the API key.

I couldn't actually find any docs about loading via the API, although
> the authorization stuff is mentioned here:
> http://openspending.org/help/api.html


That's because it's not so much an API as "emulating user input". This is
really requested a lot though, would be amazing to have a clean API for it
in core!

Here's how I run my script:
>
> jenkins at s030:~/jobs/openspending-uk25k-etl/workspace$ venv/bin/python
> dpkg-uk25k/transfer.py /home/okfn/public/uk25k/spending-20130312.csv
>
> This takes a while to load the 3Gb CSV, so to try and solve the 403
> issue I've been trying any old file:
>
> $ venv/bin/python dpkg-uk25k/transfer.py /tmp/tmpzzZJ1wxls
>
> The script just POSTs the file contents to the URL mentioned by Pudo
> with this header:
>
> Authorization: ApiKey 9983a9d9-(etc)
>
> What am I doing wrong?
>

OpenSpending doesn't really allow you to POST files, they're expected to be
at some publicly available URL and then slurped in by OS's backend. So my
proposal was to either

1) create a new source spec in the OpenSpending UI which points to
http://data.etl.openspending.org/uk25k/spending-latest.csv. Then, create a
symlink to that after each load. Finally, use the ID of the source spec to
POST to the source/NNN/load url, which basically emulates pressing the
"Load" button in the OS UI.

2) the nicer way: find out how to make diff emit the only lines that have
been added and use that to generate incremental spending source files.
Then, create a new source spec for each load and have a much nicer audit
trail.

Does that make sense?

- Friedrich



>
> Dave
>
> On 28 March 2013 17:19, David Read <david.read at hackneyworkshop.com> wrote:
> > Thanks Pudo, I'll give that a shot now,
> >
> > David
> >
> > On 28 March 2013 15:52, Friedrich Lindenberg <friedrich at pudo.org> wrote:
> >> You're now an admin on the dataset, the URL you want to POST to is
> something
> >> like http://openspending.org/ukgov-25k-spending/sources/740/load, your
> API
> >> key is in your user settings on the site. Basically I'd make a symlink
> to
> >> load from spending-latest.csv....
> >>
> >> - Friedrich
> >>
> >> p.s. OpenSpending people: shout if I need to stop handing out admin
> access
> >> to your site ;)
> >>
> >>
> >> On Thu, Mar 28, 2013 at 4:48 PM, David Read <
> david.read at hackneyworkshop.com>
> >> wrote:
> >>>
> >>> Sounds good. If someone can enable my user or give me an API key then
> >>> I'll have a bash at this.
> >>>
> >>> D
> >>>
> >>> On 28 March 2013 14:56, Friedrich Lindenberg <friedrich at pudo.org>
> wrote:
> >>> > Hey all,
> >>> >
> >>> > it's not incredibly hard set up auto-loading, but I'd propose making
> a
> >>> > small
> >>> > python script to do it cleanly. Basically, it needs to POST to the
> >>> > source
> >>> > load URL using an API key. This could be triggered at the end of the
> >>> > UK25k
> >>> > ETL run.
> >>> >
> >>> > Does this make sense? Is there someone who'd set it up - I'm keen to
> >>> > assist!
> >>> >
> >>> >  - Friedrich
> >>> >
> >>> >
> >>> >
> >>> > On Thu, Mar 28, 2013 at 11:48 AM, David Read
> >>> > <david.read at hackneyworkshop.com> wrote:
> >>> >>
> >>> >> Lucy,
> >>> >>
> >>> >> Great, yes let me know if manual or automatic data loading is easily
> >>> >> possible.
> >>> >>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/openspending-dev/attachments/20130330/96150abf/attachment.html>