[ckan-dev] Wrong URL after file upload

Wed Sep 4 16:07:18 UTC 2013

Richard:

That's right, ckanapi doesn't have FileStore support yet. We're not
using FileStore at data.gc.ca at the moment so I didn't have a chance
to add that code.

I would very much like to add support for uploading to FileStore to
ckanapi, though. I'll probably get to it eventually, but pull requests
are welcome!

Ian

On Wed, Sep 4, 2013 at 6:54 AM, Richard Gomes <rgomes.info at gmail.com> wrote:
> Hello Hendrik,
>
> Thanks a lot for your answer. Very clarifying!
> I will try the version from github :)
>
> Regarding ckanapi.... it only supports the Action API, correct? I mean:
> FileStore is not directly supported and, in particular, add_package_resource
> is not supported, correct? I don't see the Storage API defined properly and
> explicitly anywhere... am I missing something?
>
> For your information: at the moment, I'm using something similar to ckanapi,
> with a hack which provides access to some methods from CkanClient. The code
> can be seen here: http://paste.ubuntu.com/6062158/ This code supports early
> binding and code completion.
>
> Thanks
>
>
> Richard Gomes
> http://rgomes.info
> http://www.linkedin.com/in/rgomes
> mobile: +44(77)9955-6813
> inum: +883(5100)0800-9804
>
> On 04/09/13 09:17, Hendrik Bunke wrote:
>
> Hi Richard,
>
> you are most probably using an old version of ckanclient, perhaps the one
> from pypi. That version has several issues that have been fixed in the code
> at github. Please use this one https://github.com/okfn/ckanclient
>
> Furthermore, ckanclient is pretty much abondoned. It still uses the
> deprecated CKAN API 1 (REST).  AFAIK the recommended python client would now
> be https://github.com/open-data/ckanapi.
>
> regards
> hendrik
>
>
> On 3 September 2013 14:02, Richard Gomes <rgomes.info at gmail.com> wrote:
>>
>> hello,
>>
>> I'm uploading files like this:
>>
>> cli = ckanclient.CkanClient( ... )
>> pkg = 'my dataset'
>> path = '/home/rgomes/tmp/my.csv'
>> mimetype = 'text/csv'
>> cli.add_package_resource(pkg, path, resource_type='data',
>> mimetype=mimetype)
>>
>> Apparently it works fine.
>>
>> But in the web interface (CKAN website), the URL for the resource misses
>> the CKAN site_url in it, for example:
>>     URL:
>> http://storage/f/2013-09-03T115539/Kolanovic-et-al.---Rise-of-Cross-Asset-Correlations.pdf
>> ... whilst it should be:
>>     URL:
>> http://ckan.localdomain:5000/storage/f/2013-09-03T115539/Kolanovic-et-al.---Rise-of-Cross-Asset-Correlations.pdf
>>
>> What would be a recommended way to fix this?
>>
>> By recommended I mean: best practices. Should I use DataStorer instead?
>> Can it be considered mature enough?
>>
>> Thanks a lot :)
>>
>>
>> Richard Gomes
>> http://rgomes.info
>> http://www.linkedin.com/in/rgomes
>> mobile: +44(77)9955-6813
>> inum: +883(5100)0800-9804
>>
>> On 30/08/13 16:45, Richard Gomes wrote:
>>
>> Hi Dave,
>>
>> Yes, I think it makes sense.
>> At the moment I'm reinstalling everything into a VM, but next time I will
>> try this.
>>
>> Cheers :)
>>
>> Richard Gomes
>> http://rgomes.info
>> http://www.linkedin.com/in/rgomes
>> mobile: +44(77)9955-6813
>> inum: +883(5100)0800-9804
>>
>> On 30/08/13 12:40, Dave Caraway wrote:
>>
>> hi richard,
>>
>> not sure if this helps, but i've seen ghost entries before due to solr
>> being out of sync. i used paster command to reindex solr to fix.
>>
>> -Dave
>>
>>
>>
>> On Aug 29, 2013, at 3:38 PM, Richard Gomes <rgomes.info at gmail.com> wrote:
>>
>> hello,
>>
>> I loaded test data provided by CKAN into the database.
>>
>> I don't know if this is relavant... anyway, for your information:
>> After that I did some tests using the FileStore API, when I associated
>> documents to one of the datasets, like this:
>>       1 base_location='http://ckan.localdomain:5000/api'
>>       2 api_key= ...
>>       3 http_user='rgomes'
>>       4 http_pass=...
>>       5
>>       6
>>       7 def filelist(folder='.'):
>>       8     import os
>>       9     for root, dir, files in os.walk(folder):
>>      10         for file in files:
>>      11             path = '%s/%s' % (root, file)
>>      12             yield path
>>      13     return
>>      14
>>      15
>>      16 import ckanclient
>>      17 cli = ckanclient.CkanClient(base_location=base_location,
>> api_key=api_key, http_user=http_user, http_pass=http_pass, is_verbose=True)
>>      18
>>      19 pkg = 'warandpeace'
>>      20 for path in filelist('/home/rgomes/tmp/upload'):
>>      21     print path
>>      22     print cli.add_package_resource(pkg, path)
>>
>>
>> Then I've tried to recreate the database from scratch, like this:
>>
>> #!/bin/bash
>>
>> set -x
>> sudo -u postgres dropdb ${CKAN_DATABASE}
>> sudo -u postgres dropuser ${CKAN_USERNAME}
>>
>> sudo -u postgres createuser -S -D -R -P ${CKAN_USERNAME}
>> sudo -u postgres createdb -O ${CKAN_USERNAME} ${CKAN_DATABASE} -E utf-8
>>
>> paster db init -c $CKAN_ETC/default/development.ini
>> paster sysadmin add rgomes -c $CKAN_ETC/default/development.ini
>> set +x
>>
>>
>> Notice that I didn't loaded the test data after that!
>>
>>
>> For my surprise, CKAN says that there are 2 datasets available.
>> The datasets are not actually listed, but it says that there are 2
>> datasets available.
>> The faceted search tags associates 2 datasets to david and 1 to roger,
>> etc... like if the datasets where still there.
>>
>> Any idea about how this sort of information survived a complete wipe out
>> of database tables ?
>>
>> Thanks
>>
>> --
>> Richard Gomes
>> http://rgomes.info
>> http://www.linkedin.com/in/rgomes
>> mobile: +44(77)9955-6813
>> inum: +883(5100)0800-9804
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>
>>
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>
>>
>>
>>
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> http://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>>
>
>
>
> --
> Dr. Hendrik Bunke
> http://gplus.to/hbunke
> http://twitter.com/hbunke
> http://www.hbxt.org
>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>
>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>