[ckan-dev] linking data in private S3 buckets

[Stefan Oderbolz]

Hi Anton,

do you need users to upload files through the CKAN frontend to be placed in
S3? Or is this a completely separate process?
In the latter case, you can still use S3 and simply store the links to the
files on S3 in CKAN. The implementation of the Swiss portal does it exactly
like that.

1. Data owners put their data on S3 (manually or automatically)
2. The custom harvester checks, which files are available on S3
3. For each file a resource in CKAN is created using the URL provided by S3

- Stefan


On Wed, Feb 19, 2014 at 1:37 PM, Anton Lundin <anton at dohi.se> wrote:

> Hello.
>
> This choice is a showstopper for upgrading to 2.2 for us.
>
> In our usage of Ckan, we make extensive use of S3 to store files and
> have S3 do all the heavy lifting for us and that makes the database the
> only state we need to keep on the machine.
>
>
> I haven't had the time to dig in deeper into the new implementation, but
> we would require to extend that to store the files in S3 before we can
> upgrade.
>
> I've seen the ckanext-s3archive extention, but that only moves the files
> to S3, after they bin uploaded to local disk.
>
>
> //Anton
>
>
>
> On 19 February, 2014 - Nigel Babu wrote:
>
> > Hello Ivan,
> >
> > On ckan 2.2 and above, we removed the support for external filestores.
> Only
> > local filestores are supported. The old implementation was causing more
> > trouble than it's worth. We will, in the future, build an interface for
> > extensions to support multiple external filestores.
> >
> > Nigel Babu
> >
> > Developer  |  @nigelbabu <https://twitter.com/nigelbabu>
> >
> > The Open Knowledge Foundation <http://okfn.org/>
> >
> > Empowering through Open Knowledge
> >
> > http://okfn.org/  |  @okfn <http://twitter.com/OKFN>  |  OKF on
> > Facebook<https://www.facebook.com/OKFNetwork> |
> > Blog <http://blog.okfn.org/>  |  Newsletter<
> http://okfn.org/about/newsletter>
> >
> >  CKAN | http://ckan.org/ | @CKANproject
> > <http://twitter.com/CKANproject> |the world's leading open-source data
> > portal platform
> >
> >
> > On 12 February 2014 21:06, Ivan <vanzaj at gmail.com> wrote:
> >
> > > Hello,
> > >
> > > Sorry if I'm missing something obvious. I can't find any info in the
> docs,
> > > wikis, github issues, or elsewhere.
> > > Is there a way to create a private dataset linked to a file stored in a
> > > private S3 bucket?
> > >
> > > I have ofs.aws_access_key_id, and ofs.aws_secret_access_key in my
> > > <deploy>.ini, but it doesn't seem to be enough (i know it's not an auth
> > > issue as s3cmd with the same keys from the same host works fine). This
> is
> > > on ckan 2.3a.
> > >
> > > thanks,
> > > Ivan
> > >
> > >
> > > _______________________________________________
> > > ckan-dev mailing list
> > > ckan-dev at lists.okfn.org
> > > https://lists.okfn.org/mailman/listinfo/ckan-dev
> > > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
> > >
>
> > _______________________________________________
> > ckan-dev mailing list
> > ckan-dev at lists.okfn.org
> > https://lists.okfn.org/mailman/listinfo/ckan-dev
> > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>
>
> --
> Anton Lundin
>
> anton at dohi.se
> +46702-161604
>
> http://www.dohi.se/
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>



-- 
Liip AG  //  Feldstrasse 133 //  CH-8004 Zurich
Tel +41 43 500 39 80 // GnuPG 0x7B588C67 // www.liip.ch
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20140219/149588c9/attachment-0003.html>