[ckan-dev] Filestore Migration ( 2.1 to 2.2 )

Wed Feb 12 18:31:01 UTC 2014

Hi David,

Thank you for the reply.

Before disabling the old S3 bucket, I thought of 1) Renaming a file in S3
filestore 2) Changing the permission of flle in S3 filestore. After doing
both on checking the web interface, it was seen that both, download and
preview ( through dataproxy ) fail. But on reverting the changes, they work
fine. It seems that the url field is being used for download and preview (
by dataproxy ).

Regards,
Sajan

On Wed, Feb 12, 2014 at 11:24 AM, David Raznick <david.raznick at okfn.org>wrote:

> Hello Sajan
>
> Yes you are correct the migration does not change the url in the
> database.  The links generated by CKAN now depend solely on
> resource_id and the filename part of whatever is in the url field  (so
> even if the url points to a full S3 url it makes no difference it will
> just use the last part of the url i.e myfilename.csv).
> This is intentional as we did not want the migration script to parse
> out the original filename due to complexities of doing it at this
> time. This should not effect the use of the filestore and the correct
> link will be generated through the api.  The parsing of the filename
> is done when we generate the url link.
>
> So the url will look wrong in the database but this should not effect
> any use of the filestore.
>
> David
>
> On 12 February 2014 15:28, Sajan Ravindran <sajanravindran at gmail.com>
> wrote:
> > Thanks Nigel!
> >
> > After running the migrate_filestore command, all files from remote S3
> have
> > been created in the new local filestore ( I just verified the counts of
> > files on both locations as well as contents of some ). The function
> > migrate_filestore in cli.py is also doing the same, however, it is just
> not
> > updating the field url in resource table with the new url value. I was
> > wondering if that is the problem here.  ( not completely sure if I am
> right
> > here ).
> >
> > Regards,
> > Sajan
> >
> >
> > On Wed, Feb 12, 2014 at 1:09 AM, Nigel Babu <nigel.babu at okfn.org> wrote:
> >>
> >> Hi Sajan,
> >>
> >> This file has not been migrated because this file was never in the
> >> filestore. The link to S3 was directly pasted into the url field
> instead of
> >> uploading via the resource upload mechanism in CKAN.
> >>
> >> Nigel Babu
> >>
> >> Developer  |  @nigelbabu
> >>
> >> The Open Knowledge Foundation
> >>
> >> Empowering through Open Knowledge
> >>
> >> http://okfn.org/  |  @okfn  |  OKF on Facebook  |  Blog  |  Newsletter
> >>
> >>
> >> CKAN | http://ckan.org/ | @CKANproject | the world's leading
> open-source
> >> data portal platform
> >>
> >>
> >> On 11 February 2014 20:23, Sajan Ravindran <sajanravindran at gmail.com>
> >> wrote:
> >>>
> >>> Hi Nigel,
> >>>
> >>> Here is the pastebin of a resource_show for a resource:
> >>> http://pastebin.com/DxZbWJwM
> >>>
> >>> I have marked the URL ( which is still pointing to the old S3 url ) as
> >>> XXX.
> >>>
> >>> Regards,
> >>> Sajan
> >>>
> >>>
> >>> On Tue, Feb 11, 2014 at 6:03 AM, Nigel Babu <nigel.babu at okfn.org>
> wrote:
> >>>>
> >>>> Can you send me a pastebin of the output of resource_show for a
> resource
> >>>> that's in the filestore?
> >>>>
> >>>> Nigel Babu
> >>>>
> >>>> Developer  |  @nigelbabu
> >>>>
> >>>> The Open Knowledge Foundation
> >>>>
> >>>> Empowering through Open Knowledge
> >>>>
> >>>> http://okfn.org/  |  @okfn  |  OKF on Facebook  |  Blog  |
>  Newsletter
> >>>>
> >>>>
> >>>> CKAN | http://ckan.org/ | @CKANproject | the world's leading
> open-source
> >>>> data portal platform
> >>>>
> >>>>
> >>>> On 10 February 2014 22:05, Sajan Ravindran <sajanravindran at gmail.com>
> >>>> wrote:
> >>>>>
> >>>>> Hi,
> >>>>>
> >>>>> After a fresh install of 2.2, I was doing a migration of CKAN 2.1
> >>>>> database into the new database using the command : paster db
> >>>>> migrate-filestore. Though new directories and files are getting
> created in
> >>>>> the local storage, the old S3 urls are showing up when viewed
> through the
> >>>>> web interface.
> >>>>>
> >>>>> I did check up the resource table in ckan_default and found that the
> >>>>> field url is still having old values. Should it not be updated with
> new
> >>>>> values during the migration?
> >>>>>
> >>>>> Regards,
> >>>>> Sajan
> >>>>>
> >>>>> _______________________________________________
> >>>>> ckan-dev mailing list
> >>>>> ckan-dev at lists.okfn.org
> >>>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
> >>>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
> >>>>>
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> ckan-dev mailing list
> >>>> ckan-dev at lists.okfn.org
> >>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
> >>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
> >>>>
> >>>
> >>>
> >>> _______________________________________________
> >>> ckan-dev mailing list
> >>> ckan-dev at lists.okfn.org
> >>> https://lists.okfn.org/mailman/listinfo/ckan-dev
> >>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
> >>>
> >>
> >>
> >> _______________________________________________
> >> ckan-dev mailing list
> >> ckan-dev at lists.okfn.org
> >> https://lists.okfn.org/mailman/listinfo/ckan-dev
> >> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
> >>
> >
> >
> > _______________________________________________
> > ckan-dev mailing list
> > ckan-dev at lists.okfn.org
> > https://lists.okfn.org/mailman/listinfo/ckan-dev
> > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
> >
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20140212/456055f4/attachment-0003.html>