[ckan-discuss] [datacatalog] Re: Postmortem?

Wed Jun 8 22:38:49 BST 2011

(Cc'ing ckan-discuss list!)

On Wed, Jun 8, 2011 at 11:18 PM, phil at openplans.org <phil at openplans.org> wrote:
> Thanks for the thorough and thoughtful follow-up Tom. I hope to see it
> have some life in the future, even if just to use the nice user
> interface design and the API thinking that went into it.
>
> As for my CKAN + S3 question, it looks like that's actually pretty
> well answered by this post:
> http://ckan.org/2011/05/16/storage-extension-for-ckan/
>
> It looks like the extension uses OFS which supports S3 and other
> things, but I should learn more about the future of that approach,
> because I've seen some architectural diagrams which seem to imply
> possibilities for integrating with version control systems as a
> storage engine, eg
> http://blog.okfn.org/2010/02/23/introducing-datapkg/
>
> This would be much like the approach of another exciting new project,
> the open source dropbox alternative - http://sparkleshare.org/
>
> I really look forward to the future of storage integration with these
> catalog systems. It seems like the storage extension plus something
> like the Drupal integration module that data.gov.uk uses, could
> provide for a pretty robust catalog:
> http://data.gov.uk/blog/datagovuk-releases-open-source-code
>
> On May 25, 5:38 pm, Tom Lee <t... at sunlightfoundation.com> wrote:
>> Hi all -- I'm happy to offer a couple of thoughts on the NDC's demise (I'll
>> cross-post this to the main labs list in a moment).  The main reason is
>> simple: David James, the lead coder on the project and the brains behind it,
>> was lured away from Sunlight not too long ago.  Without his expertise,
>> maintainability became an issue, to the degree that it began to seriously
>> affect other systems that we maintain.  So we made the decision to shutter
>> it.
>>
>> More broadly, I think the NDC was a useful experiment for Sunlight, but it
>> is true that it never achieved the traffic or usage levels that we'd hoped
>> for.  I think it suffered from a tension between the project's vision of
>> social curation of data and our hesitation at moving beyond the metadata
>> level.  Even beyond the technical considerations, the curation task was a
>> tough nut to crack: we threw a lot of technical firepower at what may
>> fundamentally be more of an organizing task.  I'm convinced that finding and
>> attracting expert communities who can engage deeply with particular subsets
>> of the catalog is the path that such a project needs to take.
>>  Unfortunately, it's not one we're organizationally built to tackle.
>>
>> Finally, I ought to confess to a personal lack of vision for the NDC.  Clay
>> conceived of the project, and David executed that vision.  But for my part,
>> I've never been able to achieve a clear conception of how the NDC was
>> compatible with the truth that the discoverability/Google model has clearly
>> beaten the curated hierarchy/early-Yahoo model for organizing information
>> resources.
>>
>> I want to be clear: I think others have a good handle on how and why these
>> catalogs should and will work -- you'd be a fool to look at InfoChimps' last
>> funding round and think otherwise.  I also want to stress that I think
>> data.gov, data.co.uk and other government data catalogs have substantial
>> utility even -- perhaps especially -- beyond the experience of data
>> consumption.
>>
>> But I think that for now Sunlight's efforts will be better spent elsewhere;
>> I'd hate to make the problem worse by balkanizing the catalog landscape
>> unnecessarily.
>>
>> As Friedrich mentioned, the code is all open source, and of course we'd be
>> glad to offer whatever support we can to those who have uses in mind for the
>> project's data or assets.
>>
>>  Tom
>>
>> On Wed, May 25, 2011 at 5:08 PM, Friedrich Lindenberg <
>>
>> friedrich.lindenb... at okfn.org> wrote:
>> > Hi Philip,
>>
>> > I don't know much about the NDC but it truly is a shame to see it go.
>> > Since it is open-source I might try to get the scrapers running to
>> > feed intohttp://opendatasearch.org/for a bit longer as they
>> > "deteriorate". Of course, getting a similar project re-started with
>> > local "catalogue scraping heroes" in the US would be fantastic.
>>
>> > On Wed, May 25, 2011 at 10:53 PM, Philip Ashlock <p... at openplans.org>
>> > wrote:
>> > > I just saw the notice athttp://nationaldatacatalog.com
>>
>> > > Does anyone from Sunlight have a brief sentence or two postmortem on
>> > their
>> > > work with the National Data Catalog?
>>
>> > > I'd love to see a project like this continue to evolve and have been
>> > trying
>> > > to keep track of related efforts at
>> > >http://wiki.civiccommons.org/Data_Platform
>>
>> > > This might be a stupid question, but would it be unreasonable to pair
>> > > something like an amazon s3 uploader with an app like the NDC or CKAN so
>> > > that they could provide not just a data catalog, but effectively a data
>> > > repository as well?
>>
>> > This has been a ticket in CKAN for a while
>> > (http://trac.ckan.org/ticket/852) and if I am not mistaken there is at
>> > least a prototype of it that has been created as an extended
>> > linkchecker (if a recent project of mine is in any way representative,
>> > link-rot on data.gov.uk seems to be quite astonishing, by the way,
>> > highlighting the importance of this). We now have a standard extension
>> > to support storage attached to CKAN (currently in Google, but its
>> > trivial to replace with S3 or local disk) and should really start
>> > mirroring the catalogue contents regularly.
>>
>> > - Friedrich
>>
>> > > Phil
>>
>> > > --
>> > > Philip Ashlock
>> > > Open Government Program Manager | OpenPlans.org | @philipashlock
>>
>> > > --
>> > > You received this message because you are subscribed to the Google Groups
>> > > "datacatalog" group.
>> > > To post to this group, send email to datacatalog at googlegroups.com.
>> > > To unsubscribe from this group, send email to
>> > > datacatalog+unsubscribe at googlegroups.com.
>> > > For more options, visit this group at
>> > >http://groups.google.com/group/datacatalog?hl=en.
>>
>> > --
>> > You received this message because you are subscribed to the Google Groups
>> > "datacatalog" group.
>> > To post to this group, send email to datacatalog at googlegroups.com.
>> > To unsubscribe from this group, send email to
>> > datacatalog+unsubscribe at googlegroups.com.
>> > For more options, visit this group at
>> >http://groups.google.com/group/datacatalog?hl=en.
>>
>>
>
> --
> You received this message because you are subscribed to the Google Groups "datacatalog" group.
> To post to this group, send email to datacatalog at googlegroups.com.
> To unsubscribe from this group, send email to datacatalog+unsubscribe at googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/datacatalog?hl=en.
>
>


-- 
Jonathan Gray

Community Coordinator
The Open Knowledge Foundation
http://blog.okfn.org

http://twitter.com/jwyg
http://identi.ca/jwyg