[ckan-dev] Datasource and resource id - how do I create the initial resource in the datastore?

Wed May 15 09:04:37 UTC 2013

Hey,

I have it working now. There was several errors in my approach to
installing the datastore and postgres db / users.

When i finally hit an existing resource id, I got a SLL connection error
from sqlalchemy. When googling that error, I found some interesting
information. Normally, it is not recommended to have a identical names
between the database and user, i.e. ckan_default as both db and db user
(your guide indicates otherwise).

This led me to remove postgresql and reinstall the database for both CKAN
and Datastore. When I did so, I noticed a language/locale conflict and
warning from c/perl/postgres. When investigating, it turned out that there
could be an error using perl/psql from SSH. I don't know the details, but I
switched to a RDP (Cord).

Here I installed the postgresql, did the ckan db creation and datastore db
creation. I took care not creating databases and users with conflicting
names.

Then I made a data resource by uploading an .csv file via the GUI and used
that id in the curl. == success!

I do not know if it is indeed an issue with sqlalchemy, when using similar
db and user names in postgresql, but it works now. Moreover, I do not know
how the SSH perl/locale error conflicted with the postgres permissions.

Without knowing when the exact error got introduced, I would still
recommend that the guide indicate the potential naming conflict with the db
and user. Now I have datastore_db as my datastore database, and
datastore_read as my readonly user.

Thank you for your replies and help :) - it both confirmed that I
understood the APIs and indicated a more fundamental error in my
installation :P

-Henrik

On 15 May 2013 10:28, Dominik Moritz <dominik.moritz at okfn.org> wrote:

> Hey,
>
> First I have to admit that the whole process of creating a datastore table
> is not very intuitive. We plan to make it much easier but it will require
> some major changes which we couldn't finish for 2.0. Apologies for the
> confusion.
>
> At the moment, if you want a datastore-only resource (without an actual
> file, just the data), you have to create a resource through the CKAN
> interface or the resource_create[1] action (not package_create, which
> creates a dataset, which contains resources). Since you have to set a URL
> for every resource, you might have to use a bogus URL.
>
> Once you have created a resource in CKAN, copy the resource id. You can
> either find it in the url or in the table on the resource page[2]. This
> resource id can then be used to create a datastore table using the
> datastore_create action. We do this so that there is always a resource in
> CKAN if there is one in the datastore. If you want to check, what's in the
> datastore, you can head over to
> `/api/action/datastore_search?resource_id=_table_metadata` and see which
> tables exist.
>
> Again, apologies for the confusion.
>
> @ Henrik I think you used a package/dataset id, not a resource id.
>
> Best wishes,
> Dominik
>
> [1]
> http://docs.ckan.org/en/latest/ckan.logic.action.create.html#ckan.logic.action.create.resource_create
> [2] https://dl.dropboxusercontent.com/u/12770094/resource_get_id.png
>
> On 14 May 2013, at 22:24, "Lime, Steve D (MNIT)" <Steve.Lime at state.mn.us>
> wrote:
>
> > I just went though this using git master so I'm guessing it's the same
> in 2.0. Key was creating a resource via the GUI first and then referencing
> its id via the API. It's definitely not intuitive but it's the way it
> works. For example, I created a data resource by uploading a CSV file then
> created a datastore associated with that CSV file. In my case I have the
> CSV file for download and then the datastore for visualization and API
> access (it's the same data). You could use a bogus URL or maybe reference a
> PDF file containing a data dictionary for the data resource as well.
> >
> > You can get the data resource ID from either the PostgreSQL tables or
> from the GUI (you'll see the resource key in various URLs).
> >
> > Steve
> >
> > ________________________________
> > From: ckan-dev-bounces at lists.okfn.org [ckan-dev-bounces at lists.okfn.org]
> on behalf of Henrik Korsgaard [henrikkorsgaard at gmail.com]
> > Sent: Tuesday, May 14, 2013 2:47 PM
> > To: ckan-dev at lists.okfn.org
> > Subject: [ckan-dev] Datasource and resource id - how do I create the
> initial resource in the datastore?
> >
> > Hi all,
> >
> > This is a very basic question and properly belong elsewhere, but here
> goes...
> >
> > I have installed CKAN 2.0 from package install on Ubuntu 12.04, postgres
> 9.1.
> >
> > In the final setup of the datastore, the guide provides a GET and a POST
> example via curl. I can do the GET, but I have a very basic question
> regarding the POST example. It says I have to provide a resource id of an
> existing CKAN datasource (in datastore?).
> >
> > The examples in the final part of the datastore setup guide are a bit
> ambiguous, e.g. create a new resource  in datastore, up against an existing
> resource id, on a fresh (empty) install.
> >
> > Now, I have created a resource through the CKAN site and with the action
> API (package_create), but the POST example does not recognise (Not Found
> Error) any IDs.
> >
> > I have checked the apache log, the nginx log and the postgres log, no
> errors. Postgres write permissions are checked and double checked.
> >
> > [Question] So, can anyone provide a short description of how to create a
> resource, obtain resource id and add create a resource in the datastore?
> >
> > ...disclaimer: I might have overlooked something trivial in the various
> documentation, if so I am sorry :)
> >
> > --
> > Med venlig hilsen - Best regards,
> >
> > Henrik Korsgaard
> > Phone: +45 22377114
> >
> > _______________________________________________
> > ckan-dev mailing list
> > ckan-dev at lists.okfn.org
> > http://lists.okfn.org/mailman/listinfo/ckan-dev
> > Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>
> Dominik Moritz
> CKAN developer  |  skype: d.moritz  |  @doobly_doo
> The Open Knowledge Foundation
> Empowering through Open Knowledge
> http://okfn.org/  |  @okfn  |  http://ckan.org  |  @CKANproject
>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>

-- 
Med venlig hilsen - Best regards,

*Henrik Korsgaard*
Phone: +45 22377114
Office: CAVI 114b, Aarhus University

NB. I am slowly migrating my university related correspondence to my
official AU mail: korsgaard at cavi.au.dk - feel free to contact me at that
address if relevant.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20130515/c5fd9e7f/attachment-0001.html>