[ckan-dev] Filestore Migration ( 2.1 to 2.2 )

Thu Feb 13 01:12:42 UTC 2014

Hello

If on version 2.2 and the url_type field is set to upload in the
resource table the files should be able to be downloaded.

The dataproxy will have problems for private resources as it an
external service and will not be able to fetch and therefore preview
any private datasets.

David

On 12 February 2014 18:31, Sajan Ravindran <sajanravindran at gmail.com> wrote:
> Hi David,
>
> Thank you for the reply.
>
> Before disabling the old S3 bucket, I thought of 1) Renaming a file in S3
> filestore 2) Changing the permission of flle in S3 filestore. After doing
> both on checking the web interface, it was seen that both, download and
> preview ( through dataproxy ) fail. But on reverting the changes, they work
> fine. It seems that the url field is being used for download and preview (
> by dataproxy ).
>
> Regards,
> Sajan
>
>
> On Wed, Feb 12, 2014 at 11:24 AM, David Raznick <david.raznick at okfn.org>
> wrote:
>>
>> Hello Sajan
>>
>> Yes you are correct the migration does not change the url in the
>> database.  The links generated by CKAN now depend solely on
>> resource_id and the filename part of whatever is in the url field  (so
>> even if the url points to a full S3 url it makes no difference it will
>> just use the last part of the url i.e myfilename.csv).
>> This is intentional as we did not want the migration script to parse
>> out the original filename due to complexities of doing it at this
>> time. This should not effect the use of the filestore and the correct
>> link will be generated through the api.  The parsing of the filename
>> is done when we generate the url link.
>>
>> So the url will look wrong in the database but this should not effect
>> any use of the filestore.
>>
>> David
>>
>> On 12 February 2014 15:28, Sajan Ravindran <sajanravindran at gmail.com>
>> wrote:
>> > Thanks Nigel!
>> >
>> > After running the migrate_filestore command, all files from remote S3
>> > have
>> > been created in the new local filestore ( I just verified the counts of
>> > files on both locations as well as contents of some ). The function
>> > migrate_filestore in cli.py is also doing the same, however, it is just
>> > not
>> > updating the field url in resource table with the new url value. I was
>> > wondering if that is the problem here.  ( not completely sure if I am
>> > right
>> > here ).
>> >
>> > Regards,
>> > Sajan
>> >
>> >
>> > On Wed, Feb 12, 2014 at 1:09 AM, Nigel Babu <nigel.babu at okfn.org> wrote:
>> >>
>> >> Hi Sajan,
>> >>
>> >> This file has not been migrated because this file was never in the
>> >> filestore. The link to S3 was directly pasted into the url field
>> >> instead of
>> >> uploading via the resource upload mechanism in CKAN.
>> >>
>> >> Nigel Babu
>> >>
>> >> Developer  |  @nigelbabu
>> >>
>> >> The Open Knowledge Foundation
>> >>
>> >> Empowering through Open Knowledge
>> >>
>> >> http://okfn.org/  |  @okfn  |  OKF on Facebook  |  Blog  |  Newsletter
>> >>
>> >>
>> >> CKAN | http://ckan.org/ | @CKANproject | the world's leading
>> >> open-source
>> >> data portal platform
>> >>
>> >>
>> >> On 11 February 2014 20:23, Sajan Ravindran <sajanravindran at gmail.com>
>> >> wrote:
>> >>>
>> >>> Hi Nigel,
>> >>>
>> >>> Here is the pastebin of a resource_show for a resource:
>> >>> http://pastebin.com/DxZbWJwM
>> >>>
>> >>> I have marked the URL ( which is still pointing to the old S3 url ) as
>> >>> XXX.
>> >>>
>> >>> Regards,
>> >>> Sajan
>> >>>
>> >>>
>> >>> On Tue, Feb 11, 2014 at 6:03 AM, Nigel Babu <nigel.babu at okfn.org>
>> >>> wrote:
>> >>>>
>> >>>> Can you send me a pastebin of the output of resource_show for a
>> >>>> resource
>> >>>> that's in the filestore?
>> >>>>
>> >>>> Nigel Babu
>> >>>>
>> >>>> Developer  |  @nigelbabu
>> >>>>
>> >>>> The Open Knowledge Foundation
>> >>>>
>> >>>> Empowering through Open Knowledge
>> >>>>
>> >>>> http://okfn.org/  |  @okfn  |  OKF on Facebook  |  Blog  |
>> >>>> Newsletter
>> >>>>
>> >>>>
>> >>>> CKAN | http://ckan.org/ | @CKANproject | the world's leading
>> >>>> open-source
>> >>>> data portal platform
>> >>>>
>> >>>>
>> >>>> On 10 February 2014 22:05, Sajan Ravindran <sajanravindran at gmail.com>
>> >>>> wrote:
>> >>>>>
>> >>>>> Hi,
>> >>>>>
>> >>>>> After a fresh install of 2.2, I was doing a migration of CKAN 2.1
>> >>>>> database into the new database using the command : paster db
>> >>>>> migrate-filestore. Though new directories and files are getting
>> >>>>> created in
>> >>>>> the local storage, the old S3 urls are showing up when viewed
>> >>>>> through the
>> >>>>> web interface.
>> >>>>>
>> >>>>> I did check up the resource table in ckan_default and found that the
>> >>>>> field url is still having old values. Should it not be updated with
>> >>>>> new
>> >>>>> values during the migration?
>> >>>>>
>> >>>>> Regards,
>> >>>>> Sajan
>> >>>>>
>> >>>>> _______________________________________________
>> >>>>> ckan-dev mailing list
>> >>>>> ckan-dev at lists.okfn.org
>> >>>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> >>>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>> >>>>>
>> >>>>
>> >>>>
>> >>>> _______________________________________________
>> >>>> ckan-dev mailing list
>> >>>> ckan-dev at lists.okfn.org
>> >>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> >>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>> >>>>
>> >>>
>> >>>
>> >>> _______________________________________________
>> >>> ckan-dev mailing list
>> >>> ckan-dev at lists.okfn.org
>> >>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> >>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>> >>>
>> >>
>> >>
>> >> _______________________________________________
>> >> ckan-dev mailing list
>> >> ckan-dev at lists.okfn.org
>> >> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> >> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>> >>
>> >
>> >
>> > _______________________________________________
>> > ckan-dev mailing list
>> > ckan-dev at lists.okfn.org
>> > https://lists.okfn.org/mailman/listinfo/ckan-dev
>> > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>> >
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>
>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>