[ddj] Query on integrating scraperwiki & Google fusion tables

Srinivasan Ramani srini at epw.in
Sat May 17 03:57:50 UTC 2014


Dear Phuoc Do,

Many thanks.

I managed to get the script written with the help of a friend who used the
Google Fusion Tables API and utilised python libraries (lxml, simplejson)
to update the tables automatically.

We shall host the python script (after some cleaning up) in github soon.

Many thanks again.

Best Regards,
Srinivasan V. Ramani ,
Senior Assistant Editor,
Economic and Political Weekly ,
New Delhi: 110 067
09650855669
http://epw.in


On Fri, May 16, 2014 at 11:29 PM, Phuoc Do <phuocd at vida.io> wrote:

> Hi Srinivan,
>
> Have you tried this API?
>
> https://classic.scraperwiki.com/docs/api.html
>
> Alternatively, you can send your data to S3 which has well established
> API. You can write this code in your scraperwiki script.
>
> Phuoc Do
> https://vida.io/dnprock
>
> On Thu, May 15, 2014 at 3:04 AM, <
> data-driven-journalism-request at lists.okfn.org> wrote:
>
>> Send data-driven-journalism mailing list submissions to
>>         data-driven-journalism at lists.okfn.org
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>>         https://lists.okfn.org/mailman/listinfo/data-driven-journalism
>> or, via email, send a message with subject or body 'help' to
>>         data-driven-journalism-request at lists.okfn.org
>>
>> You can reach the person managing the list at
>>         data-driven-journalism-owner at lists.okfn.org
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of data-driven-journalism digest..."
>>
>>
>> Today's Topics:
>>
>>    1. media uses for open contracting data? workshop by WWW
>>       Foundation (Mason Inman)
>>    2. Query on integrating scraperwiki & Google fusion tables
>>       (Srinivasan Ramani)
>>    3. Re: Query on integrating scraperwiki & Google fusion tables
>>       (Tim McNamara)
>>    4. Re: Query on integrating scraperwiki & Google fusion tables
>>       (Srinivasan Ramani)
>>
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Wed, 14 May 2014 11:45:14 -0700
>> From: Mason Inman <masoninman at gmail.com>
>> To: data-driven-journalism at lists.okfn.org
>> Subject: [ddj] media uses for open contracting data? workshop by WWW
>>         Foundation
>> Message-ID:
>>         <
>> CABOMNLHf_CvYV0FjXsZU8cUBVcpCs1RcXBB_PzkqDnjLi+E6JA at mail.gmail.com>
>> Content-Type: text/plain; charset="utf-8"
>>
>> If you're interested in using open contracting data?that is, data on the
>> contracts for governments' buying and selling of goods and services, for
>> anything from pencils to coal?there's a web meeting coming up on Friday,
>> organized by the World Wide Web Foundation.
>> http://open-contracting.github.io/pages/usecases/events.html
>>
>> They're working on a standard for this open contracting data, so that the
>> information will be easier to make sense of, and easier to compare across
>> countries. If you want to weigh in about what would be useful for
>> journalists, and how journalists might put this data to use, check it out.
>> Below is a message from the project leader.
>>
>> ?Mason
>>
>>
>> Hi,
>>
>> Over May and June 2014 we are hosting a series of Web Meetings to hear
>> about, and discuss, the needs that different stakeholders have for
>> information on the public contracting process.
>> We hope that many of you will be able to join us for this open discussion
>> taking place via GoToMeeting (you can connect via dial-in or computer
>> audio).
>>
>> The first kick-off informal meeting will take place this Friday May 16,
>> 2014 at 8am EST for 1 hour to explore media interest in open contracting
>> data.
>> We have two presenters who will help set the stage for the discussion:
>>
>> Roberto Rocha, Interactives and data editor, The Montreal Gazette - Using
>> open data for investigative journalism; and
>> Fredrich Lindenberg, Open TED - Digging Deeper with Data Journalism.
>>
>> There will then be space for you to share your ideas, questions and
>> reflections on media use cases for open contracting data.
>>
>> A summary of the discussion will be posted to this list after the web
>> meeting so that there is a further opportunity to contribute suggestions
>> and ideas through this mailing list.  A final summary of the discussion on
>> the mailing list and from the webinar will shared before the next web
>> meeting.
>>
>> If you are interested to join this web meeting, please register here which
>> includes the goto meeting link:
>> http://open-contracting.github.io/pages/usecases/events.html
>> We looking forward to an engaging discussion.
>>
>> Cheers,
>> Michael
>>
>> ---
>> Michael Roberts
>> Open Data Technical Manager
>> +1 514 802 9528
>> @michaeloroberts | michaelr at webfoundation.org
>> World Wide Web Foundation | 1110 Vermont Ave NW, Suite 500, Washington DC,
>> 20005, USA | www.webfoundation.org | Twitter: @webfoundation
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: <
>> https://lists.okfn.org/pipermail/data-driven-journalism/attachments/20140514/c57e527b/attachment-0001.html
>> >
>>
>> ------------------------------
>>
>> Message: 2
>> Date: Thu, 15 May 2014 03:45:21 +0530
>> From: Srinivasan Ramani <srini at epw.in>
>> To: data-driven-journalism at lists.okfn.org
>> Subject: [ddj] Query on integrating scraperwiki & Google fusion tables
>> Message-ID:
>>         <CADvvm359ZkEMa7i5RV8gzztm=
>> XZDTtfzb6qpP4zzcVH-NkNYCg at mail.gmail.com>
>> Content-Type: text/plain; charset="utf-8"
>>
>>
>> Dear all,
>>
>> I have written a scraperwiki that downloads electoral data from my
>> country's election commission website.
>>
>> https://classic.scraperwiki.com/profiles/vrsrini/index.html
>>
>> I have a Google Fusion Table that has constituency data and is rendered in
>> a map.
>>
>> I need to update the Google Fusion Table dynamically with data from the
>> scraperwiki tables, so that this data is shown dynamically on the rendered
>> map.
>>
>> I understand that Google Fusion Table API helps in this purpose. But I am
>> unable (as a tyro) to figure as to how to use this API and combine it with
>> my scraperwiki.
>>
>> Please send me pointers if any of you have expertise in this. Your help
>> will be much appreciated!
>>
>> Thanks and Best Regards,
>> Srinivasan V. Ramani ,
>> Senior Assistant Editor,
>> Economic and Political Weekly ,
>> New Delhi: 110 067
>> 09650855669
>> http://epw.in
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: <
>> https://lists.okfn.org/pipermail/data-driven-journalism/attachments/20140515/84a75d4d/attachment-0001.html
>> >
>>
>> ------------------------------
>>
>> Message: 3
>> Date: Thu, 15 May 2014 18:24:24 +1200
>> From: Tim McNamara <paperless at timmcnamara.co.nz>
>> To: "List about Data Driven Journalism and Open Data in Journalism."
>>         <data-driven-journalism at lists.okfn.org>
>> Subject: Re: [ddj] Query on integrating scraperwiki & Google fusion
>>         tables
>> Message-ID:
>>         <
>> CA+YLJLtoTv+eMJz0nhsqoKuj4j12bnLv29o8284u8pXYaapvFQ at mail.gmail.com>
>> Content-Type: text/plain; charset="utf-8"
>>
>>
>> Srinivasan, what an excellent project!
>>
>> Is the code you have written so far available anywhere for others to
>> comment on?
>>
>> Tim McNamara
>> @timClicks <http://twitter.com/timClicks> | timmcnamara.co.nz
>>
>> <http://timmcnamara.co.nz/>
>>
>>
>>
>> On 15 May 2014 10:15, Srinivasan Ramani <srini at epw.in> wrote:
>>
>> > Dear all,
>> >
>> > I have written a scraperwiki that downloads electoral data from my
>> > country's election commission website.
>> >
>> > https://classic.scraperwiki.com/profiles/vrsrini/index.html
>> >
>> > I have a Google Fusion Table that has constituency data and is rendered
>> in
>> > a map.
>> >
>> > I need to update the Google Fusion Table dynamically with data from the
>> > scraperwiki tables, so that this data is shown dynamically on the
>> rendered
>> > map.
>> >
>> > I understand that Google Fusion Table API helps in this purpose. But I
>> am
>> > unable (as a tyro) to figure as to how to use this API and combine it
>> with
>> > my scraperwiki.
>> >
>> > Please send me pointers if any of you have expertise in this. Your help
>> > will be much appreciated!
>> >
>> > Thanks and Best Regards,
>> > Srinivasan V. Ramani ,
>> > Senior Assistant Editor,
>> > Economic and Political Weekly ,
>> > New Delhi: 110 067
>> > 09650855669
>> > http://epw.in
>> >
>> > _______________________________________________
>> > data-driven-journalism mailing list
>> > data-driven-journalism at lists.okfn.org
>> > https://lists.okfn.org/mailman/listinfo/data-driven-journalism
>> > Unsubscribe:
>> https://lists.okfn.org/mailman/options/data-driven-journalism
>> >
>> >
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: <
>> https://lists.okfn.org/pipermail/data-driven-journalism/attachments/20140515/793d45a9/attachment-0001.html
>> >
>>
>> ------------------------------
>>
>> Message: 4
>> Date: Thu, 15 May 2014 12:04:42 +0530
>> From: Srinivasan Ramani <srini at epw.in>
>> To: "List about Data Driven Journalism and Open Data in Journalism."
>>         <data-driven-journalism at lists.okfn.org>
>> Subject: Re: [ddj] Query on integrating scraperwiki & Google fusion
>>         tables
>> Message-ID:
>>         <
>> CADvvm35X_L1Z7GKq5+NedNU5VTP5Sw9xCtTgEcuCV0YzAjbfkg at mail.gmail.com>
>> Content-Type: text/plain; charset="utf-8"
>>
>>
>> Dear Tim,
>>
>> The code for scraping data from India's Election Commission website is
>> available here already:
>>
>> https://classic.scraperwiki.com/profiles/vrsrini/index.html
>>
>> The fusion table that I need to use to integrate data from the scraperwiki
>> tables is available here (as of now it carries turnout information on the
>> elections and it will be updated with party vote share information as data
>> comes in):
>>
>>
>> https://www.google.com/fusiontables/DataSource?docid=1gMha069AC-bkJjv7TZg5dzWbbbqq1ducBBMF_Sy7&pli=1#map:id=3
>>
>> (Data above has been organised on the basis of constituency/district.
>> India
>> has a simple majority electoral system).
>>
>> Where I need help from you and others is creating an API call to query the
>> data tables from scraperwiki and populate the google fusion table above. I
>> intend to keep this table viewable and downloadable for all (media houses
>> in India unfortunately do not do so and are very proprietary about this).
>>
>> If you can give me pointers about using the Google Fusion Tables API -
>> suitably an example that is already out there which can be modified and
>> used- it would be very useful and I would be very grateful!
>>
>> Many thanks,
>> Srinivasan
>>
>>
>>
>> Best Regards,
>> Srinivasan V. Ramani ,
>> Senior Assistant Editor,
>> Economic and Political Weekly ,
>> New Delhi: 110 067
>> 09650855669
>> http://epw.in
>>
>>
>> On Thu, May 15, 2014 at 11:54 AM, Tim McNamara
>> <paperless at timmcnamara.co.nz>wrote:
>>
>> > Srinivasan, what an excellent project!
>> >
>> > Is the code you have written so far available anywhere for others to
>> > comment on?
>> >
>> > Tim McNamara
>> > @timClicks <http://twitter.com/timClicks> | timmcnamara.co.nz
>> >
>> > <http://timmcnamara.co.nz/>
>>
>> >
>> >
>> > On 15 May 2014 10:15, Srinivasan Ramani <srini at epw.in> wrote:
>> >
>> >> Dear all,
>> >>
>> >> I have written a scraperwiki that downloads electoral data from my
>> >> country's election commission website.
>> >>
>> >> https://classic.scraperwiki.com/profiles/vrsrini/index.html
>> >>
>> >> I have a Google Fusion Table that has constituency data and is rendered
>> >> in a map.
>> >>
>> >> I need to update the Google Fusion Table dynamically with data from the
>> >> scraperwiki tables, so that this data is shown dynamically on the
>> rendered
>> >> map.
>> >>
>> >> I understand that Google Fusion Table API helps in this purpose. But I
>> am
>> >> unable (as a tyro) to figure as to how to use this API and combine it
>> with
>> >> my scraperwiki.
>> >>
>> >> Please send me pointers if any of you have expertise in this. Your help
>> >> will be much appreciated!
>> >>
>> >> Thanks and Best Regards,
>> >> Srinivasan V. Ramani ,
>> >> Senior Assistant Editor,
>> >> Economic and Political Weekly ,
>> >> New Delhi: 110 067
>> >> 09650855669
>> >> http://epw.in
>> >>
>> >> _______________________________________________
>> >> data-driven-journalism mailing list
>> >> data-driven-journalism at lists.okfn.org
>> >> https://lists.okfn.org/mailman/listinfo/data-driven-journalism
>> >> Unsubscribe:
>> >> https://lists.okfn.org/mailman/options/data-driven-journalism
>> >>
>> >>
>> >
>> > _______________________________________________
>> > data-driven-journalism mailing list
>> > data-driven-journalism at lists.okfn.org
>> > https://lists.okfn.org/mailman/listinfo/data-driven-journalism
>> > Unsubscribe:
>> https://lists.okfn.org/mailman/options/data-driven-journalism
>> >
>> >
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: <
>> https://lists.okfn.org/pipermail/data-driven-journalism/attachments/20140515/f0981fa2/attachment.html
>> >
>>
>> ------------------------------
>>
>> Subject: Digest Footer
>>
>>
>> _______________________________________________
>> data-driven-journalism mailing list
>> data-driven-journalism at lists.okfn.org
>> https://lists.okfn.org/mailman/listinfo/data-driven-journalism
>> Unsubscribe:
>> https://lists.okfn.org/mailman/optionss/data-driven-journalism
>>
>>
>> ------------------------------
>>
>> End of data-driven-journalism Digest, Vol 38, Issue 23
>> ******************************************************
>>
>
>
>
> --
> Phuoc Do
> https://vida.io/dnprock
>
> _______________________________________________
> data-driven-journalism mailing list
> data-driven-journalism at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/data-driven-journalism
> Unsubscribe: https://lists.okfn.org/mailman/options/data-driven-journalism
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/data-driven-journalism/attachments/20140517/ec2dda1c/attachment-0003.html>


More information about the data-driven-journalism mailing list