[ckan-dev] Filestore Migration ( 2.1 to 2.2 )

Wed Feb 12 16:24:53 UTC 2014

Hello Sajan

Yes you are correct the migration does not change the url in the
database.  The links generated by CKAN now depend solely on
resource_id and the filename part of whatever is in the url field  (so
even if the url points to a full S3 url it makes no difference it will
just use the last part of the url i.e myfilename.csv).
This is intentional as we did not want the migration script to parse
out the original filename due to complexities of doing it at this
time. This should not effect the use of the filestore and the correct
link will be generated through the api.  The parsing of the filename
is done when we generate the url link.

So the url will look wrong in the database but this should not effect
any use of the filestore.

David

On 12 February 2014 15:28, Sajan Ravindran <sajanravindran at gmail.com> wrote:
> Thanks Nigel!
>
> After running the migrate_filestore command, all files from remote S3 have
> been created in the new local filestore ( I just verified the counts of
> files on both locations as well as contents of some ). The function
> migrate_filestore in cli.py is also doing the same, however, it is just not
> updating the field url in resource table with the new url value. I was
> wondering if that is the problem here.  ( not completely sure if I am right
> here ).
>
> Regards,
> Sajan
>
>
> On Wed, Feb 12, 2014 at 1:09 AM, Nigel Babu <nigel.babu at okfn.org> wrote:
>>
>> Hi Sajan,
>>
>> This file has not been migrated because this file was never in the
>> filestore. The link to S3 was directly pasted into the url field instead of
>> uploading via the resource upload mechanism in CKAN.
>>
>> Nigel Babu
>>
>> Developer  |  @nigelbabu
>>
>> The Open Knowledge Foundation
>>
>> Empowering through Open Knowledge
>>
>> http://okfn.org/  |  @okfn  |  OKF on Facebook  |  Blog  |  Newsletter
>>
>>
>> CKAN | http://ckan.org/ | @CKANproject | the world's leading open-source
>> data portal platform
>>
>>
>> On 11 February 2014 20:23, Sajan Ravindran <sajanravindran at gmail.com>
>> wrote:
>>>
>>> Hi Nigel,
>>>
>>> Here is the pastebin of a resource_show for a resource:
>>> http://pastebin.com/DxZbWJwM
>>>
>>> I have marked the URL ( which is still pointing to the old S3 url ) as
>>> XXX.
>>>
>>> Regards,
>>> Sajan
>>>
>>>
>>> On Tue, Feb 11, 2014 at 6:03 AM, Nigel Babu <nigel.babu at okfn.org> wrote:
>>>>
>>>> Can you send me a pastebin of the output of resource_show for a resource
>>>> that's in the filestore?
>>>>
>>>> Nigel Babu
>>>>
>>>> Developer  |  @nigelbabu
>>>>
>>>> The Open Knowledge Foundation
>>>>
>>>> Empowering through Open Knowledge
>>>>
>>>> http://okfn.org/  |  @okfn  |  OKF on Facebook  |  Blog  |  Newsletter
>>>>
>>>>
>>>> CKAN | http://ckan.org/ | @CKANproject | the world's leading open-source
>>>> data portal platform
>>>>
>>>>
>>>> On 10 February 2014 22:05, Sajan Ravindran <sajanravindran at gmail.com>
>>>> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> After a fresh install of 2.2, I was doing a migration of CKAN 2.1
>>>>> database into the new database using the command : paster db
>>>>> migrate-filestore. Though new directories and files are getting created in
>>>>> the local storage, the old S3 urls are showing up when viewed through the
>>>>> web interface.
>>>>>
>>>>> I did check up the resource table in ckan_default and found that the
>>>>> field url is still having old values. Should it not be updated with new
>>>>> values during the migration?
>>>>>
>>>>> Regards,
>>>>> Sajan
>>>>>
>>>>> _______________________________________________
>>>>> ckan-dev mailing list
>>>>> ckan-dev at lists.okfn.org
>>>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> ckan-dev mailing list
>>>> ckan-dev at lists.okfn.org
>>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>>>
>>>
>>>
>>> _______________________________________________
>>> ckan-dev mailing list
>>> ckan-dev at lists.okfn.org
>>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>>
>>
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>
>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>