[datacatalogs] Proposal: migrating datacatalogs.org to a new simpler setup

Rufus Pollock rufus.pollock at okfn.org
Tue Jun 10 17:39:03 UTC 2014


On 9 June 2014 13:18, Rebecca Williams <rwilliams at sunlightfoundation.com>
wrote:

> Does 'Official status' mean who runs it, e.g. Government v. Community v.
> University v. other? If so that's a great thing worth adding.
>

Yes :-)


> Might be worth adding what technology each catalog is using too. Cities
> new to open data always ask this and it would be helpful for those
> interested in building tools <https://github.com/tlevine/openprism> from
> various portals to know.
>

Good point - that's another good field to add.

Rufus

On Mon, Jun 9, 2014 at 7:30 AM, Rufus Pollock <rufus.pollock at okfn.org>
> wrote:
>
>> On 6 June 2014 21:05, James McKinney <james at opennorth.ca> wrote:
>>
>>> Can we make a list of what’s wrong with the current datacatalogs.org,
>>> and what would need to change for it to be satisfactory? I don’t think
>>> we’ve yet described explicitly what’s bad about the current website.
>>>
>>
>> Here's a couple of examples
>>
>> A. Easily adding new fields to catalog entries e.g.
>>
>> - Lon/Lat
>> - Size
>> - Start Date
>> - Official status
>>
>> Then using these in faceting.
>>
>> B. Modifying the theme or specific pages (e.g. adding a map on the front
>> page)
>>
>> C. Moving to free or close to free hosting (the new app as a very
>> light-weight nodejs app can run on heroku free tier).
>>
>> To give a bit of context: current site is built on CKAN with a custom
>> extension / theme in 2011 and has seen marginal updates. CKAN is awesome
>> but is much more sophisticated than we probably need here. Moreover,
>> because the CKAN extension is so old, quite a bit of work would be be need
>> to get it upgraded (ready for doing further modifications)
>>
>> Rufus
>>
>> Once we have a list, it will be easier to commit to saying “I will help
>>> close issue X”.
>>>
>>> James
>>>
>>> On Jun 6, 2014, at 3:56 PM, Andrew Ferlitsch <aferlitsch at gmail.com>
>>> wrote:
>>>
>>> This is my first timer responding to a thread for this mailing list.
>>> First, I like to say thanks for the plug on the pretty simple user
>>> interface I have on my catalog of open data portals. I upgraded it a little
>>> to make it easier viewing (more tabular) and sortable (radio selection
>>> boxes). One problem I do have with no-login requirement for submission is
>>> that I get a least one spam submission a day (argh). The user (or bot) is
>>> using obfuscated urls so I can't detect them automatically by keywords. I
>>> will need to start tracking IP and put in place protection from MySQL
>>> injection.
>>>
>>> For this purpose, a simple CSV file for both submission collection and
>>> for the catalog work well for me. My whole user interface is
>>> auto-constructed from a PHP script that reads the CSV file.
>>>
>>> I can understand your concerns about editing. I took a similar tack
>>> here. I put together a generic editing form, which is then populated by a
>>> PHP script from the same CSV file. The text and dropdown boxes allow me to
>>> modify values from the form and then resave back to the CSV file. I press
>>> the MAKE INDEX button and the catalog is fully reconstructed. Below is a
>>> screenshot of the editing form.
>>>
>>> Andrew Ferlitsch
>>> Co-Founder, opengeocode.org
>>>
>>>
>>> On Fri, Jun 6, 2014 at 2:50 AM, Rufus Pollock <rufus.pollock at okfn.org>
>>> wrote:
>>>
>>>> On 2 June 2014 21:52, James McKinney <james at opennorth.ca> wrote:
>>>>
>>>>> The features you describe are more-or-less all on the current
>>>>> datacatalogs.org. It’s just that datacatalogs.org has accumulated
>>>>> technical debt.
>>>>>
>>>>> It seems that Rufus wants to solve the technical debt by rewriting it
>>>>> as a thin git-based system that throws away all the features you mention.
>>>>>
>>>>
>>>> I feel we would get a lot of mileage with the google spreadsheets
>>>> option and deliver most of the other features
>>>>
>>>>
>>>>> I would agree that it’s better to keep the features, and to just pay
>>>>> the price of the debt…
>>>>>
>>>>
>>>> Are the folks out there willing to help manage that debt (along with
>>>> me)?
>>>>
>>>> Rufus
>>>>
>>>>
>>>>>
>>>>> James
>>>>>
>>>>>
>>>>> On Jun 2, 2014, at 4:43 PM, Philip Ashlock <phil at civicagency.org>
>>>>> wrote:
>>>>>
>>>>> For most users I don't think git or Google spreadsheets would be
>>>>> simpler or very useful, but maybe the "simpler" was only referring to
>>>>> maintaining the site. I agree with James' criteria although I might rank
>>>>> good search/API interfaces higher than versioning. Github certainly doesn't
>>>>> provide interfaces for adding or editing that are very user friendly for
>>>>> managing CSV data and Google Spreadsheets (or any spreadsheet interface)
>>>>> isn't very useable either. I guess a Google spreadsheet form would provide
>>>>> a minimal level of useability, but that would only work for submissions,
>>>>> not edits.
>>>>>
>>>>> I think we'd be better off with a traditional CRUD app with well
>>>>> designed UI for submissions and edits than either of those options, but if
>>>>> you wanted git functionality you could provide bi-directional sync to
>>>>> github and treat the github copy as canonical. I'd still want a basic API
>>>>> though.
>>>>>
>>>>> For some recent precedents for doing bidirectional github sync with a
>>>>> CMS see:
>>>>> https://konklone.com/post/writing-in-public-syncing-with-github
>>>>> https://github.com/benbalter/wordpress-github-sync
>>>>>
>>>>> For me the ideal would be:
>>>>>
>>>>>
>>>>>    - Submissions could be made without a user account but they get
>>>>>    moderated. First via Akismet for spam filtering and then by human
>>>>>    verification. Unmoderated submissions could still be public but with
>>>>>    mechanisms to reduce abuse (eg on a separate URL blocked by search indexes
>>>>>    with robots.txt and without any URLs being linked)
>>>>>    - Edits could be made through a similar process or directly with
>>>>>    approved user accounts
>>>>>    - Everything would be accessible via full text search as well as
>>>>>    an API with basic filtering options
>>>>>    - Github syncing could be an optional alternative way to make
>>>>>    submissions/edits
>>>>>
>>>>> For what it's worth, it looks like
>>>>> http://www.opengeocode.org/opendata/ provides a pretty simple
>>>>> interface and currently appears more comprehensive than
>>>>> datacatalogs.org. There's also a list of other precedents at
>>>>> http://wiki.civiccommons.org/Initiatives#Comprehensive_Lists_of_Open_Government_Data_Catalogs
>>>>> though many have been abandoned
>>>>>
>>>>>
>>>>>
>>>>> On Fri, May 9, 2014 at 1:19 PM, Rufus Pollock <rufus.pollock at okfn.org>
>>>>> wrote:
>>>>>
>>>>>> On 9 May 2014 18:02, Ross Jones <ross at servercode.co.uk> wrote:
>>>>>>
>>>>>>>
>>>>>>> On 9 May 2014, at 15:05, Rufus Pollock <rufus.pollock at okfn.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>> *Running Code*
>>>>>>>
>>>>>>> I'm able to put my money where my mouth is here :-) I have a running
>>>>>>> demo:
>>>>>>>
>>>>>>> http://new.datacatalogs.org/
>>>>>>>
>>>>>>>
>>>>>>> http://new.datacatalogs.org/catalog/caib_es has an error
>>>>>>> https://github.com/okfn/datacatalogs.org/pull/20 fixes. Needs more
>>>>>>> meta-data on the detail page.
>>>>>>>
>>>>>>
>>>>>> thanks for the fix and now deployed.
>>>>>>
>>>>>>  Should also root out dead portals, there are one or two, should be
>>>>>>> marked as dead rather than removed I guess.
>>>>>>>
>>>>>>
>>>>>> Nice to get version control in before we do that properly ... (plus i
>>>>>> need to pull latest set from live datacatalogs.org)
>>>>>>
>>>>>> Rufus
>>>>>>
>>>>>>>
>>>>>>> Ross
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> * Rufus Pollock Founder and President | skype: rufuspollock |
>>>>>> @rufuspollock <https://twitter.com/rufuspollock> Open Knowledge
>>>>>> <http://okfn.org/> - see how data can change the world **http://okfn.org/
>>>>>> <http://okfn.org/> | @okfn <http://twitter.com/OKFN> | Open Knowledge on
>>>>>> Facebook <https://www.facebook.com/OKFNetwork> |  Blog
>>>>>> <http://blog.okfn.org/>*
>>>>>>
>>>>>> The Open Knowledge Foundation is a not-for-profit organisation.  It
>>>>>> is incorporated in England & Wales as a company limited by guarantee, with
>>>>>> company number 05133759.  VAT Registration № GB 984404989. Registered
>>>>>> office address: Open Knowledge Foundation, St John’s Innovation Centre,
>>>>>> Cowley Road, Cambridge, CB4 0WS, UK.
>>>>>>
>>>>>> _______________________________________________
>>>>>> data-catalogs mailing list
>>>>>> data-catalogs at lists.okfn.org
>>>>>> https://lists.okfn.org/mailman/listinfo/data-catalogs
>>>>>> Unsubscribe: https://lists.okfn.org/mailman/options/data-catalogs
>>>>>>
>>>>>>
>>>>> _______________________________________________
>>>>> data-catalogs mailing list
>>>>> data-catalogs at lists.okfn.org
>>>>> https://lists.okfn.org/mailman/listinfo/data-catalogs
>>>>> Unsubscribe: https://lists.okfn.org/mailman/options/data-catalogs
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> * Rufus Pollock Founder and President | skype: rufuspollock |
>>>> @rufuspollock <https://twitter.com/rufuspollock> Open Knowledge
>>>> <http://okfn.org/> - see how data can change the world **http://okfn.org/
>>>> <http://okfn.org/> | @okfn <http://twitter.com/OKFN> | Open Knowledge on
>>>> Facebook <https://www.facebook.com/OKFNetwork> |  Blog
>>>> <http://blog.okfn.org/>*
>>>>
>>>> The Open Knowledge Foundation is a not-for-profit organisation.  It is
>>>> incorporated in England & Wales as a company limited by guarantee, with
>>>> company number 05133759.  VAT Registration № GB 984404989. Registered
>>>> office address: Open Knowledge Foundation, St John’s Innovation Centre,
>>>> Cowley Road, Cambridge, CB4 0WS, UK.
>>>>
>>>> _______________________________________________
>>>> data-catalogs mailing list
>>>> data-catalogs at lists.okfn.org
>>>> https://lists.okfn.org/mailman/listinfo/data-catalogs
>>>> Unsubscribe: https://lists.okfn.org/mailman/options/data-catalogs
>>>>
>>>>
>>> <edit.jpg>
>>>
>>>
>>>
>>
>>
>> --
>>
>> * Rufus Pollock Founder and President | skype: rufuspollock |
>> @rufuspollock <https://twitter.com/rufuspollock> Open Knowledge
>> <http://okfn.org/> - see how data can change the world **http://okfn.org/
>> <http://okfn.org/> | @okfn <http://twitter.com/OKFN> | Open Knowledge on
>> Facebook <https://www.facebook.com/OKFNetwork> |  Blog
>> <http://blog.okfn.org/>*
>>
>> The Open Knowledge Foundation is a not-for-profit organisation.  It is
>> incorporated in England & Wales as a company limited by guarantee, with
>> company number 05133759.  VAT Registration № GB 984404989. Registered
>> office address: Open Knowledge Foundation, St John’s Innovation Centre,
>> Cowley Road, Cambridge, CB4 0WS, UK.
>>
>> _______________________________________________
>> data-catalogs mailing list
>> data-catalogs at lists.okfn.org
>> https://lists.okfn.org/mailman/listinfo/data-catalogs
>> Unsubscribe: https://lists.okfn.org/mailman/options/data-catalogs
>>
>>
>
>
> --
> Policy Analyst | Sunlight Foundation <http://www.sunlightfoundation.com/>
> (c) 413-387-8268 | @internetrebecca
> <http://www.twitter.com/internetrebecca>
>



-- 

*Rufus PollockFounder and President | skype: rufuspollock | @rufuspollock
<https://twitter.com/rufuspollock>Open Knowledge <http://okfn.org/> - see
how data can change the world**http://okfn.org/ <http://okfn.org/> | @okfn
<http://twitter.com/OKFN> | Open Knowledge on Facebook
<https://www.facebook.com/OKFNetwork> |  Blog <http://blog.okfn.org/>*

*Come to the Open Knowledge Festival <http://okfestival.org/> in Berlin
15-17 July 2014!*

The Open Knowledge Foundation is a not-for-profit organisation.  It is
incorporated in England & Wales as a company limited by guarantee, with
company number 05133759.  VAT Registration № GB 984404989. Registered
office address: Open Knowledge Foundation, St John’s Innovation Centre,
Cowley Road, Cambridge, CB4 0WS, UK.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/data-catalogs/attachments/20140610/c71c0541/attachment-0003.html>


More information about the data-catalogs mailing list