[datacatalogs] Proposal: migrating datacatalogs.org to a new simpler setup
aferlitsch at gmail.com
Fri Jun 6 19:56:18 UTC 2014
This is my first timer responding to a thread for this mailing list. First,
I like to say thanks for the plug on the pretty simple user interface I
have on my catalog of open data portals. I upgraded it a little to make it
easier viewing (more tabular) and sortable (radio selection boxes). One
problem I do have with no-login requirement for submission is that I get a
least one spam submission a day (argh). The user (or bot) is using
obfuscated urls so I can't detect them automatically by keywords. I will
need to start tracking IP and put in place protection from MySQL injection.
For this purpose, a simple CSV file for both submission collection and for
the catalog work well for me. My whole user interface is auto-constructed
from a PHP script that reads the CSV file.
I can understand your concerns about editing. I took a similar tack here. I
put together a generic editing form, which is then populated by a PHP
script from the same CSV file. The text and dropdown boxes allow me to
modify values from the form and then resave back to the CSV file. I press
the MAKE INDEX button and the catalog is fully reconstructed. Below is a
screenshot of the editing form.
On Fri, Jun 6, 2014 at 2:50 AM, Rufus Pollock <rufus.pollock at okfn.org>
> On 2 June 2014 21:52, James McKinney <james at opennorth.ca> wrote:
>> The features you describe are more-or-less all on the current
>> datacatalogs.org. It’s just that datacatalogs.org has accumulated
>> technical debt.
>> It seems that Rufus wants to solve the technical debt by rewriting it as
>> a thin git-based system that throws away all the features you mention.
> I feel we would get a lot of mileage with the google spreadsheets option
> and deliver most of the other features
>> I would agree that it’s better to keep the features, and to just pay the
>> price of the debt…
> Are the folks out there willing to help manage that debt (along with me)?
>> On Jun 2, 2014, at 4:43 PM, Philip Ashlock <phil at civicagency.org> wrote:
>> For most users I don't think git or Google spreadsheets would be simpler
>> or very useful, but maybe the "simpler" was only referring to maintaining
>> the site. I agree with James' criteria although I might rank good
>> search/API interfaces higher than versioning. Github certainly doesn't
>> provide interfaces for adding or editing that are very user friendly for
>> managing CSV data and Google Spreadsheets (or any spreadsheet interface)
>> isn't very useable either. I guess a Google spreadsheet form would provide
>> a minimal level of useability, but that would only work for submissions,
>> not edits.
>> I think we'd be better off with a traditional CRUD app with well designed
>> UI for submissions and edits than either of those options, but if you
>> wanted git functionality you could provide bi-directional sync to github
>> and treat the github copy as canonical. I'd still want a basic API though.
>> For some recent precedents for doing bidirectional github sync with a CMS
>> For me the ideal would be:
>> - Submissions could be made without a user account but they get
>> moderated. First via Akismet for spam filtering and then by human
>> verification. Unmoderated submissions could still be public but with
>> mechanisms to reduce abuse (eg on a separate URL blocked by search indexes
>> with robots.txt and without any URLs being linked)
>> - Edits could be made through a similar process or directly with
>> approved user accounts
>> - Everything would be accessible via full text search as well as an
>> API with basic filtering options
>> - Github syncing could be an optional alternative way to make
>> For what it's worth, it looks like http://www.opengeocode.org/opendata/
>> provides a pretty simple interface and currently appears more comprehensive
>> than datacatalogs.org. There's also a list of other precedents at
>> though many have been abandoned
>> On Fri, May 9, 2014 at 1:19 PM, Rufus Pollock <rufus.pollock at okfn.org>
>>> On 9 May 2014 18:02, Ross Jones <ross at servercode.co.uk> wrote:
>>>> On 9 May 2014, at 15:05, Rufus Pollock <rufus.pollock at okfn.org> wrote:
>>>> *Running Code*
>>>> I'm able to put my money where my mouth is here :-) I have a running
>>>> http://new.datacatalogs.org/catalog/caib_es has an error
>>>> https://github.com/okfn/datacatalogs.org/pull/20 fixes. Needs more
>>>> meta-data on the detail page.
>>> thanks for the fix and now deployed.
>>> Should also root out dead portals, there are one or two, should be
>>>> marked as dead rather than removed I guess.
>>> Nice to get version control in before we do that properly ... (plus i
>>> need to pull latest set from live datacatalogs.org)
>>> * Rufus Pollock Founder and President | skype: rufuspollock |
>>> @rufuspollock <https://twitter.com/rufuspollock> Open Knowledge
>>> <http://okfn.org/> - see how data can change the world **http://okfn.org/
>>> <http://okfn.org/> | @okfn <http://twitter.com/OKFN> | Open Knowledge on
>>> Facebook <https://www.facebook.com/OKFNetwork> | Blog
>>> The Open Knowledge Foundation is a not-for-profit organisation. It is
>>> incorporated in England & Wales as a company limited by guarantee, with
>>> company number 05133759. VAT Registration № GB 984404989. Registered
>>> office address: Open Knowledge Foundation, St John’s Innovation Centre,
>>> Cowley Road, Cambridge, CB4 0WS, UK.
>>> data-catalogs mailing list
>>> data-catalogs at lists.okfn.org
>>> Unsubscribe: https://lists.okfn.org/mailman/options/data-catalogs
>> data-catalogs mailing list
>> data-catalogs at lists.okfn.org
>> Unsubscribe: https://lists.okfn.org/mailman/options/data-catalogs
> * Rufus Pollock Founder and President | skype: rufuspollock |
> @rufuspollock <https://twitter.com/rufuspollock> Open Knowledge
> <http://okfn.org/> - see how data can change the world **http://okfn.org/
> <http://okfn.org/> | @okfn <http://twitter.com/OKFN> | Open Knowledge on
> Facebook <https://www.facebook.com/OKFNetwork> | Blog
> The Open Knowledge Foundation is a not-for-profit organisation. It is
> incorporated in England & Wales as a company limited by guarantee, with
> company number 05133759. VAT Registration № GB 984404989. Registered
> office address: Open Knowledge Foundation, St John’s Innovation Centre,
> Cowley Road, Cambridge, CB4 0WS, UK.
> data-catalogs mailing list
> data-catalogs at lists.okfn.org
> Unsubscribe: https://lists.okfn.org/mailman/options/data-catalogs
-------------- next part --------------
An HTML attachment was scrubbed...
-------------- next part --------------
A non-text attachment was scrubbed...
Size: 720066 bytes
Desc: not available
More information about the data-catalogs