[ckan-dev] Filestore Migration ( 2.1 to 2.2 )

Thu Feb 13 15:16:19 UTC 2014

Great, Thanks David. I think I need to go through the databases a bit more.
After doing that, let me get back to you.

On Wed, Feb 12, 2014 at 8:12 PM, David Raznick <david.raznick at okfn.org>wrote:

> Hello
>
> If on version 2.2 and the url_type field is set to upload in the
> resource table the files should be able to be downloaded.
>
> The dataproxy will have problems for private resources as it an
> external service and will not be able to fetch and therefore preview
> any private datasets.
>
> David
>
> On 12 February 2014 18:31, Sajan Ravindran <sajanravindran at gmail.com>
> wrote:
> > Hi David,
> >
> > Thank you for the reply.
> >
> > Before disabling the old S3 bucket, I thought of 1) Renaming a file in S3
> > filestore 2) Changing the permission of flle in S3 filestore. After doing
> > both on checking the web interface, it was seen that both, download and
> > preview ( through dataproxy ) fail. But on reverting the changes, they
> work
> > fine. It seems that the url field is being used for download and preview
> (
> > by dataproxy ).
> >
> > Regards,
> > Sajan
> >
> >
> > On Wed, Feb 12, 2014 at 11:24 AM, David Raznick <david.raznick at okfn.org>
> > wrote:
> >>
> >> Hello Sajan
> >>
> >> Yes you are correct the migration does not change the url in the
> >> database.  The links generated by CKAN now depend solely on
> >> resource_id and the filename part of whatever is in the url field  (so
> >> even if the url points to a full S3 url it makes no difference it will
> >> just use the last part of the url i.e myfilename.csv).
> >> This is intentional as we did not want the migration script to parse
> >> out the original filename due to complexities of doing it at this
> >> time. This should not effect the use of the filestore and the correct
> >> link will be generated through the api.  The parsing of the filename
> >> is done when we generate the url link.
> >>
> >> So the url will look wrong in the database but this should not effect
> >> any use of the filestore.
> >>
> >> David
> >>
> >> On 12 February 2014 15:28, Sajan Ravindran <sajanravindran at gmail.com>
> >> wrote:
> >> > Thanks Nigel!
> >> >
> >> > After running the migrate_filestore command, all files from remote S3
> >> > have
> >> > been created in the new local filestore ( I just verified the counts
> of
> >> > files on both locations as well as contents of some ). The function
> >> > migrate_filestore in cli.py is also doing the same, however, it is
> just
> >> > not
> >> > updating the field url in resource table with the new url value. I was
> >> > wondering if that is the problem here.  ( not completely sure if I am
> >> > right
> >> > here ).
> >> >
> >> > Regards,
> >> > Sajan
> >> >
> >> >
> >> > On Wed, Feb 12, 2014 at 1:09 AM, Nigel Babu <nigel.babu at okfn.org>
> wrote:
> >> >>
> >> >> Hi Sajan,
> >> >>
> >> >> This file has not been migrated because this file was never in the
> >> >> filestore. The link to S3 was directly pasted into the url field
> >> >> instead of
> >> >> uploading via the resource upload mechanism in CKAN.
> >> >>
> >> >> Nigel Babu
> >> >>
> >> >> Developer  |  @nigelbabu
> >> >>
> >> >> The Open Knowledge Foundation
> >> >>
> >> >> Empowering through Open Knowledge
> >> >>
> >> >> http://okfn.org/  |  @okfn  |  OKF on Facebook  |  Blog  |
>  Newsletter
> >> >>
> >> >>
> >> >> CKAN | http://ckan.org/ | @CKANproject | the world's leading
> >> >> open-source
> >> >> data portal platform
> >> >>
> >> >>
> >> >> On 11 February 2014 20:23, Sajan Ravindran <sajanravindran at gmail.com
> >
> >> >> wrote:
> >> >>>
> >> >>> Hi Nigel,
> >> >>>
> >> >>> Here is the pastebin of a resource_show for a resource:
> >> >>> http://pastebin.com/DxZbWJwM
> >> >>>
> >> >>> I have marked the URL ( which is still pointing to the old S3 url )
> as
> >> >>> XXX.
> >> >>>
> >> >>> Regards,
> >> >>> Sajan
> >> >>>
> >> >>>
> >> >>> On Tue, Feb 11, 2014 at 6:03 AM, Nigel Babu <nigel.babu at okfn.org>
> >> >>> wrote:
> >> >>>>
> >> >>>> Can you send me a pastebin of the output of resource_show for a
> >> >>>> resource
> >> >>>> that's in the filestore?
> >> >>>>
> >> >>>> Nigel Babu
> >> >>>>
> >> >>>> Developer  |  @nigelbabu
> >> >>>>
> >> >>>> The Open Knowledge Foundation
> >> >>>>
> >> >>>> Empowering through Open Knowledge
> >> >>>>
> >> >>>> http://okfn.org/  |  @okfn  |  OKF on Facebook  |  Blog  |
> >> >>>> Newsletter
> >> >>>>
> >> >>>>
> >> >>>> CKAN | http://ckan.org/ | @CKANproject | the world's leading
> >> >>>> open-source
> >> >>>> data portal platform
> >> >>>>
> >> >>>>
> >> >>>> On 10 February 2014 22:05, Sajan Ravindran <
> sajanravindran at gmail.com>
> >> >>>> wrote:
> >> >>>>>
> >> >>>>> Hi,
> >> >>>>>
> >> >>>>> After a fresh install of 2.2, I was doing a migration of CKAN 2.1
> >> >>>>> database into the new database using the command : paster db
> >> >>>>> migrate-filestore. Though new directories and files are getting
> >> >>>>> created in
> >> >>>>> the local storage, the old S3 urls are showing up when viewed
> >> >>>>> through the
> >> >>>>> web interface.
> >> >>>>>
> >> >>>>> I did check up the resource table in ckan_default and found that
> the
> >> >>>>> field url is still having old values. Should it not be updated
> with
> >> >>>>> new
> >> >>>>> values during the migration?
> >> >>>>>
> >> >>>>> Regards,
> >> >>>>> Sajan
> >> >>>>>
> >> >>>>> _______________________________________________
> >> >>>>> ckan-dev mailing list
> >> >>>>> ckan-dev at lists.okfn.org
> >> >>>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
> >> >>>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
> >> >>>>>
> >> >>>>
> >> >>>>
> >> >>>> _______________________________________________
> >> >>>> ckan-dev mailing list
> >> >>>> ckan-dev at lists.okfn.org
> >> >>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
> >> >>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
> >> >>>>
> >> >>>
> >> >>>
> >> >>> _______________________________________________
> >> >>> ckan-dev mailing list
> >> >>> ckan-dev at lists.okfn.org
> >> >>> https://lists.okfn.org/mailman/listinfo/ckan-dev
> >> >>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
> >> >>>
> >> >>
> >> >>
> >> >> _______________________________________________
> >> >> ckan-dev mailing list
> >> >> ckan-dev at lists.okfn.org
> >> >> https://lists.okfn.org/mailman/listinfo/ckan-dev
> >> >> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
> >> >>
> >> >
> >> >
> >> > _______________________________________________
> >> > ckan-dev mailing list
> >> > ckan-dev at lists.okfn.org
> >> > https://lists.okfn.org/mailman/listinfo/ckan-dev
> >> > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
> >> >
> >> _______________________________________________
> >> ckan-dev mailing list
> >> ckan-dev at lists.okfn.org
> >> https://lists.okfn.org/mailman/listinfo/ckan-dev
> >> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
> >
> >
> >
> > _______________________________________________
> > ckan-dev mailing list
> > ckan-dev at lists.okfn.org
> > https://lists.okfn.org/mailman/listinfo/ckan-dev
> > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
> >
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20140213/15a56cc6/attachment-0003.html>