[datacatalogs] Proposal: migrating datacatalogs.org to a new simpler setup

Fri Jun 6 19:56:18 UTC 2014

This is my first timer responding to a thread for this mailing list. First,
I like to say thanks for the plug on the pretty simple user  interface I
have on my catalog of open data portals. I upgraded it a little to make it
easier viewing (more tabular) and sortable (radio selection boxes). One
problem I do have with no-login requirement for submission is that I get a
least one spam submission a day (argh). The user (or bot) is using
obfuscated urls so I can't detect them automatically by keywords. I will
need to start tracking IP and put in place protection from MySQL injection.

For this purpose, a simple CSV file for both submission collection and for
the catalog work well for me. My whole user interface is auto-constructed
from a PHP script that reads the CSV file.

I can understand your concerns about editing. I took a similar tack here. I
put together a generic editing form, which is then populated by a PHP
script from the same CSV file. The text and dropdown boxes allow me to
modify values from the form and then resave back to the CSV file. I press
the MAKE INDEX button and the catalog is fully reconstructed. Below is a
screenshot of the editing form.

Andrew Ferlitsch
Co-Founder, opengeocode.org

On Fri, Jun 6, 2014 at 2:50 AM, Rufus Pollock <rufus.pollock at okfn.org>
wrote:

> On 2 June 2014 21:52, James McKinney <james at opennorth.ca> wrote:
>
>> The features you describe are more-or-less all on the current
>> datacatalogs.org. It’s just that datacatalogs.org has accumulated
>> technical debt.
>>
>> It seems that Rufus wants to solve the technical debt by rewriting it as
>> a thin git-based system that throws away all the features you mention.
>>
>
> I feel we would get a lot of mileage with the google spreadsheets option
> and deliver most of the other features
>
>
>> I would agree that it’s better to keep the features, and to just pay the
>> price of the debt…
>>
>
> Are the folks out there willing to help manage that debt (along with me)?
>
> Rufus
>
>
>>
>> James
>>
>>
>> On Jun 2, 2014, at 4:43 PM, Philip Ashlock <phil at civicagency.org> wrote:
>>
>> For most users I don't think git or Google spreadsheets would be simpler
>> or very useful, but maybe the "simpler" was only referring to maintaining
>> the site. I agree with James' criteria although I might rank good
>> search/API interfaces higher than versioning. Github certainly doesn't
>> provide interfaces for adding or editing that are very user friendly for
>> managing CSV data and Google Spreadsheets (or any spreadsheet interface)
>> isn't very useable either. I guess a Google spreadsheet form would provide
>> a minimal level of useability, but that would only work for submissions,
>> not edits.
>>
>> I think we'd be better off with a traditional CRUD app with well designed
>> UI for submissions and edits than either of those options, but if you
>> wanted git functionality you could provide bi-directional sync to github
>> and treat the github copy as canonical. I'd still want a basic API though.
>>
>> For some recent precedents for doing bidirectional github sync with a CMS
>> see:
>> https://konklone.com/post/writing-in-public-syncing-with-github
>> https://github.com/benbalter/wordpress-github-sync
>>
>> For me the ideal would be:
>>
>>
>>    - Submissions could be made without a user account but they get
>>    moderated. First via Akismet for spam filtering and then by human
>>    verification. Unmoderated submissions could still be public but with
>>    mechanisms to reduce abuse (eg on a separate URL blocked by search indexes
>>    with robots.txt and without any URLs being linked)
>>    - Edits could be made through a similar process or directly with
>>    approved user accounts
>>    - Everything would be accessible via full text search as well as an
>>    API with basic filtering options
>>    - Github syncing could be an optional alternative way to make
>>    submissions/edits
>>
>> For what it's worth, it looks like http://www.opengeocode.org/opendata/
>> provides a pretty simple interface and currently appears more comprehensive
>> than datacatalogs.org. There's also a list of other precedents at
>> http://wiki.civiccommons.org/Initiatives#Comprehensive_Lists_of_Open_Government_Data_Catalogs
>> though many have been abandoned
>>
>>
>>
>> On Fri, May 9, 2014 at 1:19 PM, Rufus Pollock <rufus.pollock at okfn.org>
>> wrote:
>>
>>> On 9 May 2014 18:02, Ross Jones <ross at servercode.co.uk> wrote:
>>>
>>>>
>>>> On 9 May 2014, at 15:05, Rufus Pollock <rufus.pollock at okfn.org> wrote:
>>>>
>>>> *Running Code*
>>>>
>>>> I'm able to put my money where my mouth is here :-) I have a running
>>>> demo:
>>>>
>>>> http://new.datacatalogs.org/
>>>>
>>>>
>>>> http://new.datacatalogs.org/catalog/caib_es has an error
>>>> https://github.com/okfn/datacatalogs.org/pull/20 fixes. Needs more
>>>> meta-data on the detail page.
>>>>
>>>
>>> thanks for the fix and now deployed.
>>>
>>> Should also root out dead portals, there are one or two, should be
>>>> marked as dead rather than removed I guess.
>>>>
>>>
>>> Nice to get version control in before we do that properly ... (plus i
>>> need to pull latest set from live datacatalogs.org)
>>>
>>> Rufus
>>>
>>>>
>>>> Ross
>>>>
>>>>
>>>
>>>
>>> --
>>> * Rufus Pollock Founder and President | skype: rufuspollock |
>>> @rufuspollock <https://twitter.com/rufuspollock> Open Knowledge
>>> <http://okfn.org/> - see how data can change the world **http://okfn.org/
>>> <http://okfn.org/> | @okfn <http://twitter.com/OKFN> | Open Knowledge on
>>> Facebook <https://www.facebook.com/OKFNetwork> |  Blog
>>> <http://blog.okfn.org/>*
>>>
>>> The Open Knowledge Foundation is a not-for-profit organisation.  It is
>>> incorporated in England & Wales as a company limited by guarantee, with
>>> company number 05133759.  VAT Registration № GB 984404989. Registered
>>> office address: Open Knowledge Foundation, St John’s Innovation Centre,
>>> Cowley Road, Cambridge, CB4 0WS, UK.
>>>
>>> _______________________________________________
>>> data-catalogs mailing list
>>> data-catalogs at lists.okfn.org
>>> https://lists.okfn.org/mailman/listinfo/data-catalogs
>>> Unsubscribe: https://lists.okfn.org/mailman/options/data-catalogs
>>>
>>>
>> _______________________________________________
>> data-catalogs mailing list
>> data-catalogs at lists.okfn.org
>> https://lists.okfn.org/mailman/listinfo/data-catalogs
>> Unsubscribe: https://lists.okfn.org/mailman/options/data-catalogs
>>
>>
>>
>
>
> --
>
> * Rufus Pollock Founder and President | skype: rufuspollock |
> @rufuspollock <https://twitter.com/rufuspollock> Open Knowledge
> <http://okfn.org/> - see how data can change the world **http://okfn.org/
> <http://okfn.org/> | @okfn <http://twitter.com/OKFN> | Open Knowledge on
> Facebook <https://www.facebook.com/OKFNetwork> |  Blog
> <http://blog.okfn.org/>*
>
> The Open Knowledge Foundation is a not-for-profit organisation.  It is
> incorporated in England & Wales as a company limited by guarantee, with
> company number 05133759.  VAT Registration № GB 984404989. Registered
> office address: Open Knowledge Foundation, St John’s Innovation Centre,
> Cowley Road, Cambridge, CB4 0WS, UK.
>
> _______________________________________________
> data-catalogs mailing list
> data-catalogs at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/data-catalogs
> Unsubscribe: https://lists.okfn.org/mailman/options/data-catalogs
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/data-catalogs/attachments/20140606/87216004/attachment-0002.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: edit.jpg
Type: image/jpeg
Size: 720066 bytes
Desc: not available
URL: <http://lists.okfn.org/pipermail/data-catalogs/attachments/20140606/87216004/attachment-0002.jpg>