[ckan-dev] linking data in private S3 buckets

Wed Feb 19 13:22:41 UTC 2014

Hi.

We use the Ckan api to get the authorization to upload the file direct
to S3.

User or script talks to the Ckan api. Ckan checks the authorization for
that client and uses its instance-role to pre-sign a request that allows
the client to upload a file to S3. The client uses those credentials to
upload the file to S3 and registers it with Ckan.

That way the only state is in the database and that we might move to
RDS, so then we have zero precious data on our ec2 instances.

//Anton

On 19 February, 2014 - Stefan Oderbolz wrote:

> Hi Anton,
> 
> do you need users to upload files through the CKAN frontend to be placed in
> S3? Or is this a completely separate process?
> In the latter case, you can still use S3 and simply store the links to the
> files on S3 in CKAN. The implementation of the Swiss portal does it exactly
> like that.
> 
> 1. Data owners put their data on S3 (manually or automatically)
> 2. The custom harvester checks, which files are available on S3
> 3. For each file a resource in CKAN is created using the URL provided by S3
> 
> - Stefan
> 
> 
> On Wed, Feb 19, 2014 at 1:37 PM, Anton Lundin <anton at dohi.se> wrote:
> 
> > Hello.
> >
> > This choice is a showstopper for upgrading to 2.2 for us.
> >
> > In our usage of Ckan, we make extensive use of S3 to store files and
> > have S3 do all the heavy lifting for us and that makes the database the
> > only state we need to keep on the machine.
> >
> >
> > I haven't had the time to dig in deeper into the new implementation, but
> > we would require to extend that to store the files in S3 before we can
> > upgrade.
> >
> > I've seen the ckanext-s3archive extention, but that only moves the files
> > to S3, after they bin uploaded to local disk.
> >
> >
> > //Anton
> >
> >
> >
> > On 19 February, 2014 - Nigel Babu wrote:
> >
> > > Hello Ivan,
> > >
> > > On ckan 2.2 and above, we removed the support for external filestores.
> > Only
> > > local filestores are supported. The old implementation was causing more
> > > trouble than it's worth. We will, in the future, build an interface for
> > > extensions to support multiple external filestores.
> > >
> > > Nigel Babu
> > >
> > > Developer  |  @nigelbabu <https://twitter.com/nigelbabu>
> > >
> > > The Open Knowledge Foundation <http://okfn.org/>
> > >
> > > Empowering through Open Knowledge
> > >
> > > http://okfn.org/  |  @okfn <http://twitter.com/OKFN>  |  OKF on
> > > Facebook<https://www.facebook.com/OKFNetwork> |
> > > Blog <http://blog.okfn.org/>  |  Newsletter<
> > http://okfn.org/about/newsletter>
> > >
> > >  CKAN | http://ckan.org/ | @CKANproject
> > > <http://twitter.com/CKANproject> |the world's leading open-source data
> > > portal platform
> > >
> > >
> > > On 12 February 2014 21:06, Ivan <vanzaj at gmail.com> wrote:
> > >
> > > > Hello,
> > > >
> > > > Sorry if I'm missing something obvious. I can't find any info in the
> > docs,
> > > > wikis, github issues, or elsewhere.
> > > > Is there a way to create a private dataset linked to a file stored in a
> > > > private S3 bucket?
> > > >
> > > > I have ofs.aws_access_key_id, and ofs.aws_secret_access_key in my
> > > > <deploy>.ini, but it doesn't seem to be enough (i know it's not an auth
> > > > issue as s3cmd with the same keys from the same host works fine). This
> > is
> > > > on ckan 2.3a.
> > > >
> > > > thanks,
> > > > Ivan
> > > >
> > > >
> > > > _______________________________________________
> > > > ckan-dev mailing list
> > > > ckan-dev at lists.okfn.org
> > > > https://lists.okfn.org/mailman/listinfo/ckan-dev
> > > > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
> > > >
> >
> > > _______________________________________________
> > > ckan-dev mailing list
> > > ckan-dev at lists.okfn.org
> > > https://lists.okfn.org/mailman/listinfo/ckan-dev
> > > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
> >
> >
> > --
> > Anton Lundin
> >
> > anton at dohi.se
> > +46702-161604
> >
> > http://www.dohi.se/
> > _______________________________________________
> > ckan-dev mailing list
> > ckan-dev at lists.okfn.org
> > https://lists.okfn.org/mailman/listinfo/ckan-dev
> > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
> >
> 
> 
> 
> -- 
> Liip AG  //  Feldstrasse 133 //  CH-8004 Zurich
> Tel +41 43 500 39 80 // GnuPG 0x7B588C67 // www.liip.ch

> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev

-- 
Anton Lundin

anton at dohi.se
+46702-161604

http://www.dohi.se/