[datacatalogs] Proposal: migrating datacatalogs.org to a new simpler setup

Mon Jul 28 10:21:14 UTC 2014

Agreed re usability. I think we do simplest think possible to start with.

We already have a basic working site at http://new.datacatalogs.org/

In terms of submissions I've quite inclined to try the github csv route
perhaps with a google form for new submissions which we take turns
monitoring.

Immediate first step would be to resync with the latest data from
datacatalogs.org (basically a repeat of
https://github.com/okfn/datacatalogs.org/issues/2)

Rufus

On 22 July 2014 06:10, Philip Ashlock <phil at civicagency.org> wrote:

> So it seems that we (myself included) probably don't have enough time or
> motivation for building or maintaining something with extra attention to
> usability and maybe that's not actually a huge problem, we'll see. Where
> are we in terms of just using git or google docs? Looks like the CSV in
> github hasn't been updated for about a year. As far as google docs, has
> anyone tried any of the more complex collaborative arrangements with google
> fusion tables? Seems like one option could be segmenting editing rights per
> region or something (eg canada), if that adds any value -
> https://support.google.com/fusiontables/answer/2584135?hl=en
>
>
>
> On Sun, Jun 22, 2014 at 9:20 PM, James McKinney <james at opennorth.ca>
> wrote:
>
>> Yeah, it sounds like a fair bit of work, especially for anyone unfamiliar
>> with the current deployment.
>>
>> For my own selfish needs, I just need to easily edit and export the
>> Canadian catalogs, which the new solution promises to do.
>>
>> James
>>
>> On Jun 9, 2014, at 7:30 AM, Rufus Pollock <rufus.pollock at okfn.org> wrote:
>>
>> On 6 June 2014 21:05, James McKinney <james at opennorth.ca> wrote:
>>
>>> Can we make a list of what’s wrong with the current datacatalogs.org,
>>> and what would need to change for it to be satisfactory? I don’t think
>>> we’ve yet described explicitly what’s bad about the current website.
>>>
>>
>> Here's a couple of examples
>>
>> A. Easily adding new fields to catalog entries e.g.
>>
>> - Lon/Lat
>> - Size
>> - Start Date
>> - Official status
>>
>> Then using these in faceting.
>>
>> B. Modifying the theme or specific pages (e.g. adding a map on the front
>> page)
>>
>> C. Moving to free or close to free hosting (the new app as a very
>> light-weight nodejs app can run on heroku free tier).
>>
>> To give a bit of context: current site is built on CKAN with a custom
>> extension / theme in 2011 and has seen marginal updates. CKAN is awesome
>> but is much more sophisticated than we probably need here. Moreover,
>> because the CKAN extension is so old, quite a bit of work would be be need
>> to get it upgraded (ready for doing further modifications)
>>
>> Rufus
>>
>> Once we have a list, it will be easier to commit to saying “I will help
>>> close issue X”.
>>>
>>> James
>>>
>>> On Jun 6, 2014, at 3:56 PM, Andrew Ferlitsch <aferlitsch at gmail.com>
>>> wrote:
>>>
>>> This is my first timer responding to a thread for this mailing list.
>>> First, I like to say thanks for the plug on the pretty simple user
>>> interface I have on my catalog of open data portals. I upgraded it a little
>>> to make it easier viewing (more tabular) and sortable (radio selection
>>> boxes). One problem I do have with no-login requirement for submission is
>>> that I get a least one spam submission a day (argh). The user (or bot) is
>>> using obfuscated urls so I can't detect them automatically by keywords. I
>>> will need to start tracking IP and put in place protection from MySQL
>>> injection.
>>>
>>> For this purpose, a simple CSV file for both submission collection and
>>> for the catalog work well for me. My whole user interface is
>>> auto-constructed from a PHP script that reads the CSV file.
>>>
>>> I can understand your concerns about editing. I took a similar tack
>>> here. I put together a generic editing form, which is then populated by a
>>> PHP script from the same CSV file. The text and dropdown boxes allow me to
>>> modify values from the form and then resave back to the CSV file. I press
>>> the MAKE INDEX button and the catalog is fully reconstructed. Below is a
>>> screenshot of the editing form.
>>>
>>> Andrew Ferlitsch
>>> Co-Founder, opengeocode.org
>>>
>>>
>>> On Fri, Jun 6, 2014 at 2:50 AM, Rufus Pollock <rufus.pollock at okfn.org>
>>> wrote:
>>>
>>>> On 2 June 2014 21:52, James McKinney <james at opennorth.ca> wrote:
>>>>
>>>>> The features you describe are more-or-less all on the current
>>>>> datacatalogs.org. It’s just that datacatalogs.org has accumulated
>>>>> technical debt.
>>>>>
>>>>> It seems that Rufus wants to solve the technical debt by rewriting it
>>>>> as a thin git-based system that throws away all the features you mention.
>>>>>
>>>>
>>>> I feel we would get a lot of mileage with the google spreadsheets
>>>> option and deliver most of the other features
>>>>
>>>>
>>>>> I would agree that it’s better to keep the features, and to just pay
>>>>> the price of the debt…
>>>>>
>>>>
>>>> Are the folks out there willing to help manage that debt (along with
>>>> me)?
>>>>
>>>> Rufus
>>>>
>>>>
>>>>>
>>>>> James
>>>>>
>>>>>
>>>>> On Jun 2, 2014, at 4:43 PM, Philip Ashlock <phil at civicagency.org>
>>>>> wrote:
>>>>>
>>>>> For most users I don't think git or Google spreadsheets would be
>>>>> simpler or very useful, but maybe the "simpler" was only referring to
>>>>> maintaining the site. I agree with James' criteria although I might rank
>>>>> good search/API interfaces higher than versioning. Github certainly doesn't
>>>>> provide interfaces for adding or editing that are very user friendly for
>>>>> managing CSV data and Google Spreadsheets (or any spreadsheet interface)
>>>>> isn't very useable either. I guess a Google spreadsheet form would provide
>>>>> a minimal level of useability, but that would only work for submissions,
>>>>> not edits.
>>>>>
>>>>> I think we'd be better off with a traditional CRUD app with well
>>>>> designed UI for submissions and edits than either of those options, but if
>>>>> you wanted git functionality you could provide bi-directional sync to
>>>>> github and treat the github copy as canonical. I'd still want a basic API
>>>>> though.
>>>>>
>>>>> For some recent precedents for doing bidirectional github sync with a
>>>>> CMS see:
>>>>> https://konklone.com/post/writing-in-public-syncing-with-github
>>>>> https://github.com/benbalter/wordpress-github-sync
>>>>>
>>>>> For me the ideal would be:
>>>>>
>>>>>
>>>>>    - Submissions could be made without a user account but they get
>>>>>    moderated. First via Akismet for spam filtering and then by human
>>>>>    verification. Unmoderated submissions could still be public but with
>>>>>    mechanisms to reduce abuse (eg on a separate URL blocked by search indexes
>>>>>    with robots.txt and without any URLs being linked)
>>>>>    - Edits could be made through a similar process or directly with
>>>>>    approved user accounts
>>>>>    - Everything would be accessible via full text search as well as
>>>>>    an API with basic filtering options
>>>>>    - Github syncing could be an optional alternative way to make
>>>>>    submissions/edits
>>>>>
>>>>> For what it's worth, it looks like
>>>>> http://www.opengeocode.org/opendata/ provides a pretty simple
>>>>> interface and currently appears more comprehensive than
>>>>> datacatalogs.org. There's also a list of other precedents at
>>>>> http://wiki.civiccommons.org/Initiatives#Comprehensive_Lists_of_Open_Government_Data_Catalogs
>>>>> though many have been abandoned
>>>>>
>>>>>
>>>>>
>>>>> On Fri, May 9, 2014 at 1:19 PM, Rufus Pollock <rufus.pollock at okfn.org>
>>>>> wrote:
>>>>>
>>>>>> On 9 May 2014 18:02, Ross Jones <ross at servercode.co.uk> wrote:
>>>>>>
>>>>>>>
>>>>>>> On 9 May 2014, at 15:05, Rufus Pollock <rufus.pollock at okfn.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>> *Running Code*
>>>>>>>
>>>>>>> I'm able to put my money where my mouth is here :-) I have a running
>>>>>>> demo:
>>>>>>>
>>>>>>> http://new.datacatalogs.org/
>>>>>>>
>>>>>>>
>>>>>>> http://new.datacatalogs.org/catalog/caib_es has an error
>>>>>>> https://github.com/okfn/datacatalogs.org/pull/20 fixes. Needs more
>>>>>>> meta-data on the detail page.
>>>>>>>
>>>>>>
>>>>>> thanks for the fix and now deployed.
>>>>>>
>>>>>>  Should also root out dead portals, there are one or two, should be
>>>>>>> marked as dead rather than removed I guess.
>>>>>>>
>>>>>>
>>>>>> Nice to get version control in before we do that properly ... (plus i
>>>>>> need to pull latest set from live datacatalogs.org)
>>>>>>
>>>>>> Rufus
>>>>>>
>>>>>>>
>>>>>>> Ross
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> * Rufus Pollock Founder and President | skype: rufuspollock |
>>>>>> @rufuspollock <https://twitter.com/rufuspollock> Open Knowledge
>>>>>> <http://okfn.org/> - see how data can change the world **http://okfn.org/
>>>>>> <http://okfn.org/> | @okfn <http://twitter.com/OKFN> | Open Knowledge on
>>>>>> Facebook <https://www.facebook.com/OKFNetwork> |  Blog
>>>>>> <http://blog.okfn.org/>*
>>>>>>
>>>>>> The Open Knowledge Foundation is a not-for-profit organisation.  It
>>>>>> is incorporated in England & Wales as a company limited by guarantee, with
>>>>>> company number 05133759.  VAT Registration № GB 984404989. Registered
>>>>>> office address: Open Knowledge Foundation, St John’s Innovation Centre,
>>>>>> Cowley Road, Cambridge, CB4 0WS, UK.
>>>>>>
>>>>>> _______________________________________________
>>>>>> data-catalogs mailing list
>>>>>> data-catalogs at lists.okfn.org
>>>>>> https://lists.okfn.org/mailman/listinfo/data-catalogs
>>>>>> Unsubscribe: https://lists.okfn.org/mailman/options/data-catalogs
>>>>>>
>>>>>>
>>>>> _______________________________________________
>>>>> data-catalogs mailing list
>>>>> data-catalogs at lists.okfn.org
>>>>> https://lists.okfn.org/mailman/listinfo/data-catalogs
>>>>> Unsubscribe: https://lists.okfn.org/mailman/options/data-catalogs
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> * Rufus Pollock Founder and President | skype: rufuspollock |
>>>> @rufuspollock <https://twitter.com/rufuspollock> Open Knowledge
>>>> <http://okfn.org/> - see how data can change the world **http://okfn.org/
>>>> <http://okfn.org/> | @okfn <http://twitter.com/OKFN> | Open Knowledge on
>>>> Facebook <https://www.facebook.com/OKFNetwork> |  Blog
>>>> <http://blog.okfn.org/>*
>>>>
>>>> The Open Knowledge Foundation is a not-for-profit organisation.  It is
>>>> incorporated in England & Wales as a company limited by guarantee, with
>>>> company number 05133759.  VAT Registration № GB 984404989. Registered
>>>> office address: Open Knowledge Foundation, St John’s Innovation Centre,
>>>> Cowley Road, Cambridge, CB4 0WS, UK.
>>>>
>>>> _______________________________________________
>>>> data-catalogs mailing list
>>>> data-catalogs at lists.okfn.org
>>>> https://lists.okfn.org/mailman/listinfo/data-catalogs
>>>> Unsubscribe: https://lists.okfn.org/mailman/options/data-catalogs
>>>>
>>>>
>>> <edit.jpg>
>>>
>>>
>>>
>>
>>
>> --
>> * Rufus Pollock Founder and President | skype: rufuspollock |
>> @rufuspollock <https://twitter.com/rufuspollock> Open Knowledge
>> <http://okfn.org/> - see how data can change the world **http://okfn.org/
>> <http://okfn.org/> | @okfn <http://twitter.com/OKFN> | Open Knowledge on
>> Facebook <https://www.facebook.com/OKFNetwork> |  Blog
>> <http://blog.okfn.org/>*
>>
>> The Open Knowledge Foundation is a not-for-profit organisation.  It is
>> incorporated in England & Wales as a company limited by guarantee, with
>> company number 05133759.  VAT Registration № GB 984404989. Registered
>> office address: Open Knowledge Foundation, St John’s Innovation Centre,
>> Cowley Road, Cambridge, CB4 0WS, UK.
>>
>>
>>
>> _______________________________________________
>> data-catalogs mailing list
>> data-catalogs at lists.okfn.org
>> https://lists.okfn.org/mailman/listinfo/data-catalogs
>> Unsubscribe: https://lists.okfn.org/mailman/options/data-catalogs
>>
>>
>

-- 

*Rufus PollockFounder and President | skype: rufuspollock | @rufuspollock
<https://twitter.com/rufuspollock>Open Knowledge <http://okfn.org/> - see
how data can change the world**http://okfn.org/ <http://okfn.org/> | @okfn
<http://twitter.com/OKFN> | Open Knowledge on Facebook
<https://www.facebook.com/OKFNetwork> |  Blog <http://blog.okfn.org/>*

The Open Knowledge Foundation is a not-for-profit organisation.  It is
incorporated in England & Wales as a company limited by guarantee, with
company number 05133759.  VAT Registration № GB 984404989. Registered
office address: Open Knowledge Foundation, St John’s Innovation Centre,
Cowley Road, Cambridge, CB4 0WS, UK.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/data-catalogs/attachments/20140728/2ea73394/attachment-0003.html>