[datacatalogs] Proposal: migrating datacatalogs.org to a new simpler setup

Tue Jul 22 05:10:08 UTC 2014

So it seems that we (myself included) probably don't have enough time or
motivation for building or maintaining something with extra attention to
usability and maybe that's not actually a huge problem, we'll see. Where
are we in terms of just using git or google docs? Looks like the CSV in
github hasn't been updated for about a year. As far as google docs, has
anyone tried any of the more complex collaborative arrangements with google
fusion tables? Seems like one option could be segmenting editing rights per
region or something (eg canada), if that adds any value -
https://support.google.com/fusiontables/answer/2584135?hl=en

On Sun, Jun 22, 2014 at 9:20 PM, James McKinney <james at opennorth.ca> wrote:

> Yeah, it sounds like a fair bit of work, especially for anyone unfamiliar
> with the current deployment.
>
> For my own selfish needs, I just need to easily edit and export the
> Canadian catalogs, which the new solution promises to do.
>
> James
>
> On Jun 9, 2014, at 7:30 AM, Rufus Pollock <rufus.pollock at okfn.org> wrote:
>
> On 6 June 2014 21:05, James McKinney <james at opennorth.ca> wrote:
>
>> Can we make a list of what’s wrong with the current datacatalogs.org,
>> and what would need to change for it to be satisfactory? I don’t think
>> we’ve yet described explicitly what’s bad about the current website.
>>
>
> Here's a couple of examples
>
> A. Easily adding new fields to catalog entries e.g.
>
> - Lon/Lat
> - Size
> - Start Date
> - Official status
>
> Then using these in faceting.
>
> B. Modifying the theme or specific pages (e.g. adding a map on the front
> page)
>
> C. Moving to free or close to free hosting (the new app as a very
> light-weight nodejs app can run on heroku free tier).
>
> To give a bit of context: current site is built on CKAN with a custom
> extension / theme in 2011 and has seen marginal updates. CKAN is awesome
> but is much more sophisticated than we probably need here. Moreover,
> because the CKAN extension is so old, quite a bit of work would be be need
> to get it upgraded (ready for doing further modifications)
>
> Rufus
>
> Once we have a list, it will be easier to commit to saying “I will help
>> close issue X”.
>>
>> James
>>
>> On Jun 6, 2014, at 3:56 PM, Andrew Ferlitsch <aferlitsch at gmail.com>
>> wrote:
>>
>> This is my first timer responding to a thread for this mailing list.
>> First, I like to say thanks for the plug on the pretty simple user
>> interface I have on my catalog of open data portals. I upgraded it a little
>> to make it easier viewing (more tabular) and sortable (radio selection
>> boxes). One problem I do have with no-login requirement for submission is
>> that I get a least one spam submission a day (argh). The user (or bot) is
>> using obfuscated urls so I can't detect them automatically by keywords. I
>> will need to start tracking IP and put in place protection from MySQL
>> injection.
>>
>> For this purpose, a simple CSV file for both submission collection and
>> for the catalog work well for me. My whole user interface is
>> auto-constructed from a PHP script that reads the CSV file.
>>
>> I can understand your concerns about editing. I took a similar tack here.
>> I put together a generic editing form, which is then populated by a PHP
>> script from the same CSV file. The text and dropdown boxes allow me to
>> modify values from the form and then resave back to the CSV file. I press
>> the MAKE INDEX button and the catalog is fully reconstructed. Below is a
>> screenshot of the editing form.
>>
>> Andrew Ferlitsch
>> Co-Founder, opengeocode.org
>>
>>
>> On Fri, Jun 6, 2014 at 2:50 AM, Rufus Pollock <rufus.pollock at okfn.org>
>> wrote:
>>
>>> On 2 June 2014 21:52, James McKinney <james at opennorth.ca> wrote:
>>>
>>>> The features you describe are more-or-less all on the current
>>>> datacatalogs.org. It’s just that datacatalogs.org has accumulated
>>>> technical debt.
>>>>
>>>> It seems that Rufus wants to solve the technical debt by rewriting it
>>>> as a thin git-based system that throws away all the features you mention.
>>>>
>>>
>>> I feel we would get a lot of mileage with the google spreadsheets option
>>> and deliver most of the other features
>>>
>>>
>>>> I would agree that it’s better to keep the features, and to just pay
>>>> the price of the debt…
>>>>
>>>
>>> Are the folks out there willing to help manage that debt (along with me)?
>>>
>>> Rufus
>>>
>>>
>>>>
>>>> James
>>>>
>>>>
>>>> On Jun 2, 2014, at 4:43 PM, Philip Ashlock <phil at civicagency.org>
>>>> wrote:
>>>>
>>>> For most users I don't think git or Google spreadsheets would be
>>>> simpler or very useful, but maybe the "simpler" was only referring to
>>>> maintaining the site. I agree with James' criteria although I might rank
>>>> good search/API interfaces higher than versioning. Github certainly doesn't
>>>> provide interfaces for adding or editing that are very user friendly for
>>>> managing CSV data and Google Spreadsheets (or any spreadsheet interface)
>>>> isn't very useable either. I guess a Google spreadsheet form would provide
>>>> a minimal level of useability, but that would only work for submissions,
>>>> not edits.
>>>>
>>>> I think we'd be better off with a traditional CRUD app with well
>>>> designed UI for submissions and edits than either of those options, but if
>>>> you wanted git functionality you could provide bi-directional sync to
>>>> github and treat the github copy as canonical. I'd still want a basic API
>>>> though.
>>>>
>>>> For some recent precedents for doing bidirectional github sync with a
>>>> CMS see:
>>>> https://konklone.com/post/writing-in-public-syncing-with-github
>>>> https://github.com/benbalter/wordpress-github-sync
>>>>
>>>> For me the ideal would be:
>>>>
>>>>
>>>>    - Submissions could be made without a user account but they get
>>>>    moderated. First via Akismet for spam filtering and then by human
>>>>    verification. Unmoderated submissions could still be public but with
>>>>    mechanisms to reduce abuse (eg on a separate URL blocked by search indexes
>>>>    with robots.txt and without any URLs being linked)
>>>>    - Edits could be made through a similar process or directly with
>>>>    approved user accounts
>>>>    - Everything would be accessible via full text search as well as an
>>>>    API with basic filtering options
>>>>    - Github syncing could be an optional alternative way to make
>>>>    submissions/edits
>>>>
>>>> For what it's worth, it looks like http://www.opengeocode.org/opendata/
>>>> provides a pretty simple interface and currently appears more comprehensive
>>>> than datacatalogs.org. There's also a list of other precedents at
>>>> http://wiki.civiccommons.org/Initiatives#Comprehensive_Lists_of_Open_Government_Data_Catalogs
>>>> though many have been abandoned
>>>>
>>>>
>>>>
>>>> On Fri, May 9, 2014 at 1:19 PM, Rufus Pollock <rufus.pollock at okfn.org>
>>>> wrote:
>>>>
>>>>> On 9 May 2014 18:02, Ross Jones <ross at servercode.co.uk> wrote:
>>>>>
>>>>>>
>>>>>> On 9 May 2014, at 15:05, Rufus Pollock <rufus.pollock at okfn.org>
>>>>>> wrote:
>>>>>>
>>>>>> *Running Code*
>>>>>>
>>>>>> I'm able to put my money where my mouth is here :-) I have a running
>>>>>> demo:
>>>>>>
>>>>>> http://new.datacatalogs.org/
>>>>>>
>>>>>>
>>>>>> http://new.datacatalogs.org/catalog/caib_es has an error
>>>>>> https://github.com/okfn/datacatalogs.org/pull/20 fixes. Needs more
>>>>>> meta-data on the detail page.
>>>>>>
>>>>>
>>>>> thanks for the fix and now deployed.
>>>>>
>>>>>  Should also root out dead portals, there are one or two, should be
>>>>>> marked as dead rather than removed I guess.
>>>>>>
>>>>>
>>>>> Nice to get version control in before we do that properly ... (plus i
>>>>> need to pull latest set from live datacatalogs.org)
>>>>>
>>>>> Rufus
>>>>>
>>>>>>
>>>>>> Ross
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> * Rufus Pollock Founder and President | skype: rufuspollock |
>>>>> @rufuspollock <https://twitter.com/rufuspollock> Open Knowledge
>>>>> <http://okfn.org/> - see how data can change the world **http://okfn.org/
>>>>> <http://okfn.org/> | @okfn <http://twitter.com/OKFN> | Open Knowledge on
>>>>> Facebook <https://www.facebook.com/OKFNetwork> |  Blog
>>>>> <http://blog.okfn.org/>*
>>>>>
>>>>> The Open Knowledge Foundation is a not-for-profit organisation.  It is
>>>>> incorporated in England & Wales as a company limited by guarantee, with
>>>>> company number 05133759.  VAT Registration № GB 984404989. Registered
>>>>> office address: Open Knowledge Foundation, St John’s Innovation Centre,
>>>>> Cowley Road, Cambridge, CB4 0WS, UK.
>>>>>
>>>>> _______________________________________________
>>>>> data-catalogs mailing list
>>>>> data-catalogs at lists.okfn.org
>>>>> https://lists.okfn.org/mailman/listinfo/data-catalogs
>>>>> Unsubscribe: https://lists.okfn.org/mailman/options/data-catalogs
>>>>>
>>>>>
>>>> _______________________________________________
>>>> data-catalogs mailing list
>>>> data-catalogs at lists.okfn.org
>>>> https://lists.okfn.org/mailman/listinfo/data-catalogs
>>>> Unsubscribe: https://lists.okfn.org/mailman/options/data-catalogs
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> * Rufus Pollock Founder and President | skype: rufuspollock |
>>> @rufuspollock <https://twitter.com/rufuspollock> Open Knowledge
>>> <http://okfn.org/> - see how data can change the world **http://okfn.org/
>>> <http://okfn.org/> | @okfn <http://twitter.com/OKFN> | Open Knowledge on
>>> Facebook <https://www.facebook.com/OKFNetwork> |  Blog
>>> <http://blog.okfn.org/>*
>>>
>>> The Open Knowledge Foundation is a not-for-profit organisation.  It is
>>> incorporated in England & Wales as a company limited by guarantee, with
>>> company number 05133759.  VAT Registration № GB 984404989. Registered
>>> office address: Open Knowledge Foundation, St John’s Innovation Centre,
>>> Cowley Road, Cambridge, CB4 0WS, UK.
>>>
>>> _______________________________________________
>>> data-catalogs mailing list
>>> data-catalogs at lists.okfn.org
>>> https://lists.okfn.org/mailman/listinfo/data-catalogs
>>> Unsubscribe: https://lists.okfn.org/mailman/options/data-catalogs
>>>
>>>
>> <edit.jpg>
>>
>>
>>
>
>
> --
> * Rufus Pollock Founder and President | skype: rufuspollock |
> @rufuspollock <https://twitter.com/rufuspollock> Open Knowledge
> <http://okfn.org/> - see how data can change the world **http://okfn.org/
> <http://okfn.org/> | @okfn <http://twitter.com/OKFN> | Open Knowledge on
> Facebook <https://www.facebook.com/OKFNetwork> |  Blog
> <http://blog.okfn.org/>*
>
> The Open Knowledge Foundation is a not-for-profit organisation.  It is
> incorporated in England & Wales as a company limited by guarantee, with
> company number 05133759.  VAT Registration № GB 984404989. Registered
> office address: Open Knowledge Foundation, St John’s Innovation Centre,
> Cowley Road, Cambridge, CB4 0WS, UK.
>
>
>
> _______________________________________________
> data-catalogs mailing list
> data-catalogs at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/data-catalogs
> Unsubscribe: https://lists.okfn.org/mailman/options/data-catalogs
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/data-catalogs/attachments/20140722/4f68231a/attachment-0002.html>