[ckan-dev] linking data in private S3 buckets

Wed Dec 17 10:50:43 UTC 2014

On 24 February, 2014 - Nigel Babu wrote:

> We don't have a timetable yet. It's still in the planning stage. We will
> definitely ask for comments on the list when we have a solid plan of how
> it's going to be implemented. You'll also be able to follow the bug when we
> start work.
> 

Started to take a look at this again. Is there any plans on how to
re-introduce other file storages than local filesystem again?

As i read the code, there is no clear path on where to start on
re-implementing such a story. Most of the bits for the old storage model
is still there, but it looks to have bin partially replaced with
ckan/lib/uploader.py.

As said before, one of the main reasons we choosed to use s3 for our
ckan filestorage was to keep the webserver stateless. In our setup there
is no reason to backup the webserver, because it got no state. All the
state is in the database and s3.

Another use case that this move to local-only-storage breaks is if you
would like to have a scaling farm of webservers. Then you would need to
involve a network filesystem to keep the filestorage consistent across
all webservers.
It also moves the heavy lifting, actually sending the files, to the
webserver, away from the storage solution optimized for this exact task.

//Anton

-- 
Anton Lundin

anton at dohi.se
+46702-161604

http://www.dohi.se/