[datacatalogs] Proposal: migrating datacatalogs.org to a new simpler setup

Thomas Levine _ at thomaslevine.com
Tue Jul 22 11:32:21 UTC 2014 

The only thing I'd like with datacatalogs.org
is for the data catalogs to be correct. But I
don't care that much.

I could set OpenPrism and pluplusch
http://openprism.thomaslevine.com
https://pypi.python.org/pypi/pluplusch

to load their catalogs lists from some
magic CSV file or whatever. Would this
increase anyone's motivation to keep the
information up to date?

On 22 Jul 01:10, Philip Ashlock wrote:
> So it seems that we (myself included) probably don't have enough time or
> motivation for building or maintaining something with extra attention to
> usability and maybe that's not actually a huge problem, we'll see. Where
> are we in terms of just using git or google docs? Looks like the CSV in
> github hasn't been updated for about a year. As far as google docs, has
> anyone tried any of the more complex collaborative arrangements with google
> fusion tables? Seems like one option could be segmenting editing rights per
> region or something (eg canada), if that adds any value -
> https://support.google.com/fusiontables/answer/2584135?hl=en
> 
> 
> On Sun, Jun 22, 2014 at 9:20 PM, James McKinney <james at opennorth.ca> wrote:
> 
> > Yeah, it sounds like a fair bit of work, especially for anyone unfamiliar
> > with the current deployment.
> >
> > For my own selfish needs, I just need to easily edit and export the
> > Canadian catalogs, which the new solution promises to do.
> >
> > James
> >
> > On Jun 9, 2014, at 7:30 AM, Rufus Pollock <rufus.pollock at okfn.org> wrote:
> >
> > On 6 June 2014 21:05, James McKinney <james at opennorth.ca> wrote:
> >
> >> Can we make a list of what’s wrong with the current datacatalogs.org,
> >> and what would need to change for it to be satisfactory? I don’t think
> >> we’ve yet described explicitly what’s bad about the current website.
> >>
> >
> > Here's a couple of examples
> >
> > A. Easily adding new fields to catalog entries e.g.
> >
> > - Lon/Lat
> > - Size
> > - Start Date
> > - Official status
> >
> > Then using these in faceting.
> >
> > B. Modifying the theme or specific pages (e.g. adding a map on the front
> > page)
> >
> > C. Moving to free or close to free hosting (the new app as a very
> > light-weight nodejs app can run on heroku free tier).
> >
> > To give a bit of context: current site is built on CKAN with a custom
> > extension / theme in 2011 and has seen marginal updates. CKAN is awesome
> > but is much more sophisticated than we probably need here. Moreover,
> > because the CKAN extension is so old, quite a bit of work would be be need
> > to get it upgraded (ready for doing further modifications)
> >
> > Rufus
> >
> > Once we have a list, it will be easier to commit to saying “I will help
> >> close issue X”.
> >>
> >> James
> >>
> >> On Jun 6, 2014, at 3:56 PM, Andrew Ferlitsch <aferlitsch at gmail.com>
> >> wrote:
> >>
> >> This is my first timer responding to a thread for this mailing list.
> >> First, I like to say thanks for the plug on the pretty simple user
> >> interface I have on my catalog of open data portals. I upgraded it a little
> >> to make it easier viewing (more tabular) and sortable (radio selection
> >> boxes). One problem I do have with no-login requirement for submission is
> >> that I get a least one spam submission a day (argh). The user (or bot) is
> >> using obfuscated urls so I can't detect them automatically by keywords. I
> >> will need to start tracking IP and put in place protection from MySQL
> >> injection.
> >>
> >> For this purpose, a simple CSV file for both submission collection and
> >> for the catalog work well for me. My whole user interface is
> >> auto-constructed from a PHP script that reads the CSV file.
> >>
> >> I can understand your concerns about editing. I took a similar tack here.
> >> I put together a generic editing form, which is then populated by a PHP
> >> script from the same CSV file. The text and dropdown boxes allow me to
> >> modify values from the form and then resave back to the CSV file. I press
> >> the MAKE INDEX button and the catalog is fully reconstructed. Below is a
> >> screenshot of the editing form.
> >>
> >> Andrew Ferlitsch
> >> Co-Founder, opengeocode.org
> >>
> >>
> >> On Fri, Jun 6, 2014 at 2:50 AM, Rufus Pollock <rufus.pollock at okfn.org>
> >> wrote:
> >>
> >>> On 2 June 2014 21:52, James McKinney <james at opennorth.ca> wrote:
> >>>
> >>>> The features you describe are more-or-less all on the current
> >>>> datacatalogs.org. It’s just that datacatalogs.org has accumulated
> >>>> technical debt.
> >>>>
> >>>> It seems that Rufus wants to solve the technical debt by rewriting it
> >>>> as a thin git-based system that throws away all the features you mention.
> >>>>
> >>>
> >>> I feel we would get a lot of mileage with the google spreadsheets option
> >>> and deliver most of the other features
> >>>
> >>>
> >>>> I would agree that it’s better to keep the features, and to just pay
> >>>> the price of the debt…
> >>>>
> >>>
> >>> Are the folks out there willing to help manage that debt (along with me)?
> >>>
> >>> Rufus
> >>>
> >>>
> >>>>
> >>>> James
> >>>>
> >>>>
> >>>> On Jun 2, 2014, at 4:43 PM, Philip Ashlock <phil at civicagency.org>
> >>>> wrote:
> >>>>
> >>>> For most users I don't think git or Google spreadsheets would be
> >>>> simpler or very useful, but maybe the "simpler" was only referring to
> >>>> maintaining the site. I agree with James' criteria although I might rank
> >>>> good search/API interfaces higher than versioning. Github certainly doesn't
> >>>> provide interfaces for adding or editing that are very user friendly for
> >>>> managing CSV data and Google Spreadsheets (or any spreadsheet interface)
> >>>> isn't very useable either. I guess a Google spreadsheet form would provide
> >>>> a minimal level of useability, but that would only work for submissions,
> >>>> not edits.
> >>>>
> >>>> I think we'd be better off with a traditional CRUD app with well
> >>>> designed UI for submissions and edits than either of those options, but if
> >>>> you wanted git functionality you could provide bi-directional sync to
> >>>> github and treat the github copy as canonical. I'd still want a basic API
> >>>> though.
> >>>>
> >>>> For some recent precedents for doing bidirectional github sync with a
> >>>> CMS see:
> >>>> https://konklone.com/post/writing-in-public-syncing-with-github
> >>>> https://github.com/benbalter/wordpress-github-sync
> >>>>
> >>>> For me the ideal would be:
> >>>>
> >>>>
> >>>>    - Submissions could be made without a user account but they get
> >>>>    moderated. First via Akismet for spam filtering and then by human
> >>>>    verification. Unmoderated submissions could still be public but with
> >>>>    mechanisms to reduce abuse (eg on a separate URL blocked by search indexes
> >>>>    with robots.txt and without any URLs being linked)
> >>>>    - Edits could be made through a similar process or directly with
> >>>>    approved user accounts
> >>>>    - Everything would be accessible via full text search as well as an
> >>>>    API with basic filtering options
> >>>>    - Github syncing could be an optional alternative way to make
> >>>>    submissions/edits
> >>>>
> >>>> For what it's worth, it looks like http://www.opengeocode.org/opendata/
> >>>> provides a pretty simple interface and currently appears more comprehensive
> >>>> than datacatalogs.org. There's also a list of other precedents at
> >>>> http://wiki.civiccommons.org/Initiatives#Comprehensive_Lists_of_Open_Government_Data_Catalogs
> >>>> though many have been abandoned
> >>>>
> >>>>
> >>>>
> >>>> On Fri, May 9, 2014 at 1:19 PM, Rufus Pollock <rufus.pollock at okfn.org>
> >>>> wrote:
> >>>>
> >>>>> On 9 May 2014 18:02, Ross Jones <ross at servercode.co.uk> wrote:
> >>>>>
> >>>>>>
> >>>>>> On 9 May 2014, at 15:05, Rufus Pollock <rufus.pollock at okfn.org>
> >>>>>> wrote:
> >>>>>>
> >>>>>> *Running Code*
> >>>>>>
> >>>>>> I'm able to put my money where my mouth is here :-) I have a running
> >>>>>> demo:
> >>>>>>
> >>>>>> http://new.datacatalogs.org/
> >>>>>>
> >>>>>>
> >>>>>> http://new.datacatalogs.org/catalog/caib_es has an error
> >>>>>> https://github.com/okfn/datacatalogs.org/pull/20 fixes. Needs more
> >>>>>> meta-data on the detail page.
> >>>>>>
> >>>>>
> >>>>> thanks for the fix and now deployed.
> >>>>>
> >>>>>  Should also root out dead portals, there are one or two, should be
> >>>>>> marked as dead rather than removed I guess.
> >>>>>>
> >>>>>
> >>>>> Nice to get version control in before we do that properly ... (plus i
> >>>>> need to pull latest set from live datacatalogs.org)
> >>>>>
> >>>>> Rufus
> >>>>>
> >>>>>>
> >>>>>> Ross
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> * Rufus Pollock Founder and President | skype: rufuspollock |
> >>>>> @rufuspollock <https://twitter.com/rufuspollock> Open Knowledge
> >>>>> <http://okfn.org/> - see how data can change the world **http://okfn.org/
> >>>>> <http://okfn.org/> | @okfn <http://twitter.com/OKFN> | Open Knowledge on
> >>>>> Facebook <https://www.facebook.com/OKFNetwork> |  Blog
> >>>>> <http://blog.okfn.org/>*
> >>>>>
> >>>>> The Open Knowledge Foundation is a not-for-profit organisation.  It is
> >>>>> incorporated in England & Wales as a company limited by guarantee, with
> >>>>> company number 05133759.  VAT Registration № GB 984404989. Registered
> >>>>> office address: Open Knowledge Foundation, St John’s Innovation Centre,
> >>>>> Cowley Road, Cambridge, CB4 0WS, UK.
> >>>>>
> >>>>> _______________________________________________
> >>>>> data-catalogs mailing list
> >>>>> data-catalogs at lists.okfn.org
> >>>>> https://lists.okfn.org/mailman/listinfo/data-catalogs
> >>>>> Unsubscribe: https://lists.okfn.org/mailman/options/data-catalogs
> >>>>>
> >>>>>
> >>>> _______________________________________________
> >>>> data-catalogs mailing list
> >>>> data-catalogs at lists.okfn.org
> >>>> https://lists.okfn.org/mailman/listinfo/data-catalogs
> >>>> Unsubscribe: https://lists.okfn.org/mailman/options/data-catalogs
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> * Rufus Pollock Founder and President | skype: rufuspollock |
> >>> @rufuspollock <https://twitter.com/rufuspollock> Open Knowledge
> >>> <http://okfn.org/> - see how data can change the world **http://okfn.org/
> >>> <http://okfn.org/> | @okfn <http://twitter.com/OKFN> | Open Knowledge on
> >>> Facebook <https://www.facebook.com/OKFNetwork> |  Blog
> >>> <http://blog.okfn.org/>*
> >>>
> >>> The Open Knowledge Foundation is a not-for-profit organisation.  It is
> >>> incorporated in England & Wales as a company limited by guarantee, with
> >>> company number 05133759.  VAT Registration № GB 984404989. Registered
> >>> office address: Open Knowledge Foundation, St John’s Innovation Centre,
> >>> Cowley Road, Cambridge, CB4 0WS, UK.
> >>>
> >>> _______________________________________________
> >>> data-catalogs mailing list
> >>> data-catalogs at lists.okfn.org
> >>> https://lists.okfn.org/mailman/listinfo/data-catalogs
> >>> Unsubscribe: https://lists.okfn.org/mailman/options/data-catalogs
> >>>
> >>>
> >> <edit.jpg>
> >>
> >>
> >>
> >
> >
> > --
> > * Rufus Pollock Founder and President | skype: rufuspollock |
> > @rufuspollock <https://twitter.com/rufuspollock> Open Knowledge
> > <http://okfn.org/> - see how data can change the world **http://okfn.org/
> > <http://okfn.org/> | @okfn <http://twitter.com/OKFN> | Open Knowledge on
> > Facebook <https://www.facebook.com/OKFNetwork> |  Blog
> > <http://blog.okfn.org/>*
> >
> > The Open Knowledge Foundation is a not-for-profit organisation.  It is
> > incorporated in England & Wales as a company limited by guarantee, with
> > company number 05133759.  VAT Registration № GB 984404989. Registered
> > office address: Open Knowledge Foundation, St John’s Innovation Centre,
> > Cowley Road, Cambridge, CB4 0WS, UK.
> >
> >
> >
> > _______________________________________________
> > data-catalogs mailing list
> > data-catalogs at lists.okfn.org
> > https://lists.okfn.org/mailman/listinfo/data-catalogs
> > Unsubscribe: https://lists.okfn.org/mailman/options/data-catalogs
> >
> >

> _______________________________________________
> data-catalogs mailing list
> data-catalogs at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/data-catalogs
> Unsubscribe: https://lists.okfn.org/mailman/options/data-catalogs

More information about the data-catalogs mailing list