[ckan-dev] Wrong URL after file upload
Richard Gomes
rgomes.info at gmail.com
Wed Sep 4 23:55:41 UTC 2013
Hi Ian,
You could copy some 10 lines from what I did and you can basically
'tunnel' calls to FileStore. It's easy and dirty.
http://paste.ubuntu.com/6062158/
Honestly, I'm a newbie in CKAN and I'm not willing to reinvent the wheel
and/or send nonsense push requests. So, I prefer to expose my code via
pastebin in case someone more skilled than me finds something useful.
Richard Gomes
http://rgomes.info
http://www.linkedin.com/in/rgomes
mobile: +44(77)9955-6813
inum <http://www.inum.net/>: +883(5100)0800-9804
On 04/09/13 17:07, Ian Ward wrote:
> Richard:
>
> That's right, ckanapi doesn't have FileStore support yet. We're not
> using FileStore at data.gc.ca at the moment so I didn't have a chance
> to add that code.
>
> I would very much like to add support for uploading to FileStore to
> ckanapi, though. I'll probably get to it eventually, but pull requests
> are welcome!
>
> Ian
>
> On Wed, Sep 4, 2013 at 6:54 AM, Richard Gomes <rgomes.info at gmail.com> wrote:
>> Hello Hendrik,
>>
>> Thanks a lot for your answer. Very clarifying!
>> I will try the version from github :)
>>
>> Regarding ckanapi.... it only supports the Action API, correct? I mean:
>> FileStore is not directly supported and, in particular, add_package_resource
>> is not supported, correct? I don't see the Storage API defined properly and
>> explicitly anywhere... am I missing something?
>>
>> For your information: at the moment, I'm using something similar to ckanapi,
>> with a hack which provides access to some methods from CkanClient. The code
>> can be seen here: http://paste.ubuntu.com/6062158/ This code supports early
>> binding and code completion.
>>
>> Thanks
>>
>>
>> Richard Gomes
>> http://rgomes.info
>> http://www.linkedin.com/in/rgomes
>> mobile: +44(77)9955-6813
>> inum: +883(5100)0800-9804
>>
>> On 04/09/13 09:17, Hendrik Bunke wrote:
>>
>> Hi Richard,
>>
>> you are most probably using an old version of ckanclient, perhaps the one
>> from pypi. That version has several issues that have been fixed in the code
>> at github. Please use this one https://github.com/okfn/ckanclient
>>
>> Furthermore, ckanclient is pretty much abondoned. It still uses the
>> deprecated CKAN API 1 (REST). AFAIK the recommended python client would now
>> be https://github.com/open-data/ckanapi.
>>
>> regards
>> hendrik
>>
>>
>> On 3 September 2013 14:02, Richard Gomes <rgomes.info at gmail.com> wrote:
>>> hello,
>>>
>>> I'm uploading files like this:
>>>
>>> cli = ckanclient.CkanClient( ... )
>>> pkg = 'my dataset'
>>> path = '/home/rgomes/tmp/my.csv'
>>> mimetype = 'text/csv'
>>> cli.add_package_resource(pkg, path, resource_type='data',
>>> mimetype=mimetype)
>>>
>>> Apparently it works fine.
>>>
>>> But in the web interface (CKAN website), the URL for the resource misses
>>> the CKAN site_url in it, for example:
>>> URL:
>>> http://storage/f/2013-09-03T115539/Kolanovic-et-al.---Rise-of-Cross-Asset-Correlations.pdf
>>> ... whilst it should be:
>>> URL:
>>> http://ckan.localdomain:5000/storage/f/2013-09-03T115539/Kolanovic-et-al.---Rise-of-Cross-Asset-Correlations.pdf
>>>
>>> What would be a recommended way to fix this?
>>>
>>> By recommended I mean: best practices. Should I use DataStorer instead?
>>> Can it be considered mature enough?
>>>
>>> Thanks a lot :)
>>>
>>>
>>> Richard Gomes
>>> http://rgomes.info
>>> http://www.linkedin.com/in/rgomes
>>> mobile: +44(77)9955-6813
>>> inum: +883(5100)0800-9804
>>>
>>> On 30/08/13 16:45, Richard Gomes wrote:
>>>
>>> Hi Dave,
>>>
>>> Yes, I think it makes sense.
>>> At the moment I'm reinstalling everything into a VM, but next time I will
>>> try this.
>>>
>>> Cheers :)
>>>
>>> Richard Gomes
>>> http://rgomes.info
>>> http://www.linkedin.com/in/rgomes
>>> mobile: +44(77)9955-6813
>>> inum: +883(5100)0800-9804
>>>
>>> On 30/08/13 12:40, Dave Caraway wrote:
>>>
>>> hi richard,
>>>
>>> not sure if this helps, but i've seen ghost entries before due to solr
>>> being out of sync. i used paster command to reindex solr to fix.
>>>
>>> -Dave
>>>
>>>
>>>
>>> On Aug 29, 2013, at 3:38 PM, Richard Gomes <rgomes.info at gmail.com> wrote:
>>>
>>> hello,
>>>
>>> I loaded test data provided by CKAN into the database.
>>>
>>> I don't know if this is relavant... anyway, for your information:
>>> After that I did some tests using the FileStore API, when I associated
>>> documents to one of the datasets, like this:
>>> 1 base_location='http://ckan.localdomain:5000/api'
>>> 2 api_key= ...
>>> 3 http_user='rgomes'
>>> 4 http_pass=...
>>> 5
>>> 6
>>> 7 def filelist(folder='.'):
>>> 8 import os
>>> 9 for root, dir, files in os.walk(folder):
>>> 10 for file in files:
>>> 11 path = '%s/%s' % (root, file)
>>> 12 yield path
>>> 13 return
>>> 14
>>> 15
>>> 16 import ckanclient
>>> 17 cli = ckanclient.CkanClient(base_location=base_location,
>>> api_key=api_key, http_user=http_user, http_pass=http_pass, is_verbose=True)
>>> 18
>>> 19 pkg = 'warandpeace'
>>> 20 for path in filelist('/home/rgomes/tmp/upload'):
>>> 21 print path
>>> 22 print cli.add_package_resource(pkg, path)
>>>
>>>
>>> Then I've tried to recreate the database from scratch, like this:
>>>
>>> #!/bin/bash
>>>
>>> set -x
>>> sudo -u postgres dropdb ${CKAN_DATABASE}
>>> sudo -u postgres dropuser ${CKAN_USERNAME}
>>>
>>> sudo -u postgres createuser -S -D -R -P ${CKAN_USERNAME}
>>> sudo -u postgres createdb -O ${CKAN_USERNAME} ${CKAN_DATABASE} -E utf-8
>>>
>>> paster db init -c $CKAN_ETC/default/development.ini
>>> paster sysadmin add rgomes -c $CKAN_ETC/default/development.ini
>>> set +x
>>>
>>>
>>> Notice that I didn't loaded the test data after that!
>>>
>>>
>>> For my surprise, CKAN says that there are 2 datasets available.
>>> The datasets are not actually listed, but it says that there are 2
>>> datasets available.
>>> The faceted search tags associates 2 datasets to david and 1 to roger,
>>> etc... like if the datasets where still there.
>>>
>>> Any idea about how this sort of information survived a complete wipe out
>>> of database tables ?
>>>
>>> Thanks
>>>
>>> --
>>> Richard Gomes
>>> http://rgomes.info
>>> http://www.linkedin.com/in/rgomes
>>> mobile: +44(77)9955-6813
>>> inum: +883(5100)0800-9804
>>>
>>> _______________________________________________
>>> ckan-dev mailing list
>>> ckan-dev at lists.okfn.org
>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>>
>>>
>>>
>>> _______________________________________________
>>> ckan-dev mailing list
>>> ckan-dev at lists.okfn.org
>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> ckan-dev mailing list
>>> ckan-dev at lists.okfn.org
>>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>>
>>
>>
>> --
>> Dr. Hendrik Bunke
>> http://gplus.to/hbunke
>> http://twitter.com/hbunke
>> http://www.hbxt.org
>>
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>
>>
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20130905/ce1a8ee8/attachment-0001.html>
More information about the ckan-dev
mailing list