[okfn-labs] [OpenScienceFramework] Re: Github webhooks and automating creating Data APIs for data packages in Github

Wed Mar 19 17:40:19 UTC 2014

Hi, Rufus, OKFN,

Sheila cued us into this conversation (and I have since signed up for
the mailing list), but Github webhooks require no code,
necessarily--just an endpoint that does what you want knowing that an
update has been made. I'd be happy to chat about it.

Beyond that, however, that's sort of what the Open Science Framework
(OSF) offers as a free service--and I think there's a lot that we at
the Center for Open Science can do to help with some of the
initiatives I see linked in this email.

The OSF has a Github add-on, for instance. We also have a modular file
rendering infrastructure
(https://github.com/CenterForOpenScience/modular-file-renderer) that
allows open source contributors to, basically, do stuff with files
(render them in html, convert between formats, make changes, etc.) in
a very simple "player" environment (so even those with minimal
programming experience can contribute without needing to understand
the full stack). These two things together allow for the following:

https://osf.io/hks4q/files/

Under the Github Add-on, click on the excel, spss, or csv data file to
see it rendered as a rich table (direct links below):

https://osf.io/cz8si/?sha=b907e5dc0ec4be15bb6c8365a9570de2db545625&branch=master
https://osf.io/yd8bh/?sha=b907e5dc0ec4be15bb6c8365a9570de2db545625&branch=master
https://osf.io/uvrj6/?sha=b907e5dc0ec4be15bb6c8365a9570de2db545625&branch=master

If you have tools or suggestions for other things--other ways of
rendering data formats (perhaps with tools you've created),
automatically creating a data api for every file, etc.--please let us
know. We have many great interns, will be doing an open source
developer push in a few weeks at PyCon, and welcome contribution from
any OKFNers. We're also working on a similar player for add-ons to
make the add-ons that are developed as modular and reusable as the
file rendering modules. We also have a beta release of an R library
that allows for api-based interaction with any files on the OSF or
add-on repos that the OSF can connect to.

For the scientist, they wouldn't need to do anything other than login
with github (or figshare, s3, or any of the other many add-ons we have
in the queue)--the OSF deals with the webhooks, collaboration
features, etc--even at the level of the add-ons interacting (e.g.,
coming soon, drag and drop moving of files between repos or analytics
add-ons recognizing the existence of code and data add-ons and
automatically preparing the analytics environment with minimal user
intervention).

I look forward to future OKFN Mailing List discussion,

Jeff.

On Wed, Mar 19, 2014 at 12:59 PM, sheila miguez <shekay at pobox.com> wrote:
> This is an intriguing idea, so I answer yes to A!
>
> For B. I believe that some of the people in the Open Science Framework group
> have created a github webhook, at least in the case for their hosting of the
> Open Science Collaboration blog, and perhaps that code is available
> somewhere.
>
>
> On Wed, Mar 19, 2014 at 7:35 AM, Rufus Pollock <rufus.pollock at okfn.org>
> wrote:
>>
>> Hi All,
>>
>> I have a User Story as follows:
>>
>> As a Data Wrangler (Rufus!) I want to push my tabular data package to
>> github and have it automatically have a Data API for that data package
>> created/updated so that I have a nice API I can use in my app (or JS
>> visualization or ...)
>>
>> Context
>>
>> I'm currently really liking the work flow of pushing Data Packaged
>> Datasets to Github (see growing set of "Core" datasets at
>> https://github.com/datasets plus personal ones like hard drive prices).
>> However, it would be really nice to get a proper Data API. CKAN, and hence
>> the DataHub, now has a nice DataStore with a rich Data API which we could
>> use. There is even a nice standalone webapp for automatically pulling data
>> into the DataStore - https://github.com/ckan/datapusher
>>
>> (Note: this would also address the Data API issue in the Frictionless Data
>> roadmap)
>>
>> Plan
>>
>> Rather than create a new Data API webapp I thought the best thing would be
>> create a small webapp that a github webhook can connect to. The app will
>> take care of then pushing the updated data to the DataStore.
>>
>> Question
>>
>> A. Does this sound sensible
>>
>> B. Do folks have any thoughs and experience on build github webhook apps
>> (especially in nodejs or python)
>>
>> All the best,
>>
>> Rufus
>>
>>
>> _______________________________________________
>> okfn-labs mailing list
>> okfn-labs at lists.okfn.org
>> https://lists.okfn.org/mailman/listinfo/okfn-labs
>> Unsubscribe: https://lists.okfn.org/mailman/options/okfn-labs
>>
>
>
>
> --
> sheila
>
> --
> You received this message because you are subscribed to the Google Groups
> "Open Science Framework" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to openscienceframework+unsubscribe at googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.