[ckan-dev] Harvester - Problem

Michael Reichart michael.reichart at gmail.com
Sun Jan 20 21:57:49 UTC 2013


Hi,

maybe I'm understanding something totally wrong, but my problem seems to be
much more fundamental.
I've setup a clean CKAN 1.8.1b with ckanext-harvest (master branch from
19/01/2013) and organizations plugin enabled and ckan.auth.profile =
publisher. What I'm trying to do is to insert data with a harvester and add
all the datasets to a certain organization. But this doesn't work, even if
I use the standard ckan_harvester. If I add some logging, the package_dict
includes "groups" and "organizations" before creating the package, but they
seem to get lost there. Creating the package returns true.
I thought when using the organizations - extension the groups are used as
organizations. But when I try to add the datasets to the group, they are
not correctly added altough there are no errors in the harvesters view. How
do I have to set the correct organization? Are there any samples of this
configuration?

When I try to set ckan.harvest.auth.profile to "publisher" I have the
problem, that my admin user cannot add a publisher to the harvester (at
least the harvester-form tells me this) altough the admin user is member of
an organization. So I figured this may be because he's sysadmin, but with
any other user, no matter if editor or admin of the group I cannot open the
harvester administration and therefore also cannot add a publisher to the
harvester. Can anybody tell me how this is meant to use? Are there any
other roles I have to set? If yes, which and where do I find this
information? Does this solve my problem?
It's okay for me to have a different harvester or even different
harvester-users for each organization if this works ...

Thanks for your help, I really appreciate it!
Michael


2013/1/18 Michael Reichart <michael.reichart at gmail.com>

> Hi,
>
> I've tried to change my harvesting and made a new extension only for
> harvesting and didn't enable our other extension (with our custom schema)
> in the paster ini. But with this setup groups are also not correctly
> applied. Altough our custom scheme is not applied (our own
> form_to_db_scheme is not called).
>
> Now I've tried to use the publisher setting in ckan.harvest.auth.profile,
> but this gives me the error: "Please choose an organization".
> But in the harvester web-interface I get the message "Cannot add any
> publishers" - what do I make wrong? Does anybody have an example for a
> harvester with organizations and publisher-profile?
>
> Thanks for the help!
> Michael
>
>
> 2013/1/17 Michael Reichart <michael.reichart at gmail.com>
>
>> Hi,
>>
>> seems that my first email didn't come through the mailing-list:
>>
>>  Hi,
>> datasets created and updated by the webinterface do not loose their
>> groups. I haven't tested the direct api-calls yet.
>> I can't share our full extension, but i've attached the full code of the
>> IDatasetForm
>> We're using a custom CKAN-Harvester (which only does some mapping to our
>> custom fields) based on ckanext-harvest from beginning of July 2012 could
>> be Version 1.6.12??
>>
>> From my understanding, it shouldn't be necessary to call our custom
>> schema to import from another CKAN as the data-objects are already in the
>> right form.
>>
>> Thanks!
>> Michael
>>
>>
>> class DgvatForm(SingletonPlugin):
>>     implements(IRoutes)
>>     implements(IConfigurer)
>>     implements(IGenshiStreamFilter)
>>     implements(IDatasetForm)
>>
>>     log.fatal("Enter: %s" % __name__)
>>     def package_form(self):
>>         return 'package/datagvat_organization_form.html'
>>
>>
>>     def before_map(self, map):
>>         map.connect('/error/{action}',
>> controller='ckanext.dgvat_por.controllers.data_gv_at:DgvatErrorController')
>>         map.connect('/error/{action}/{id}',
>> controller='ckanext.dgvat_por.controllers.data_gv_at:DgvatErrorController')
>>
>>
>>         map.connect('/dataset/new',
>> controller='ckanext.dgvat_por.controllers.data_gv_at:DgvatPackageController',
>> action='new')
>>         map.connect('/dataset/edit/{id}',
>> controller='ckanext.dgvat_por.controllers.data_gv_at:DgvatPackageController',
>> action='edit')
>>         map.connect('/dataset/{id}.{format}',
>> controller='ckanext.dgvat_por.controllers.data_gv_at:DgvatPackageController',
>> action='read')
>>         map.connect('/dataset/{id}',
>> controller='ckanext.dgvat_por.controllers.data_gv_at:DgvatPackageController',
>> action='read')
>>         map.connect('/dataset/{id}/resource/{resource_id}',
>> controller='ckanext.dgvat_por.controllers.data_gv_at:DgvatPackageController',
>> action='resource_read')
>>         map.connect('/dataset/{id}/resource/{resource_id}/embed',
>> controller='ckanext.dgvat_por.controllers.data_gv_at:DgvatPackageController',
>> action='resource_embedded_dataviewer')
>>         map.connect('/dataset/editresources/{id}',
>> controller='ckanext.dgvat_por.controllers.data_gv_at:DgvatPackageController',
>> action='editresources')
>>         map.connect('/user/edit/{id:.*}',
>> controller='ckanext.dgvat_por.controllers.dgvat_user:DgvatUserController',
>> action='edit')
>>         #m.connect('/user/reset/{id:.*}', action='perform_reset')
>>         map.connect('/user/register',
>> controller='ckanext.dgvat_por.controllers.dgvat_user:DgvatUserController',
>> action='register')
>>         map.connect('/user/login',
>> controller='ckanext.dgvat_por.controllers.dgvat_user:DgvatUserController',
>> action='login')
>>         map.connect('/user/_logout', '/user/logout',
>> controller='ckanext.dgvat_por.controllers.dgvat_user:DgvatUserController',
>> action='logout')
>>         map.connect('/user/logged_in',
>> controller='ckanext.dgvat_por.controllers.dgvat_user:DgvatUserController',
>> action='logged_in')
>>         map.connect('/user/logged_out',
>> controller='ckanext.dgvat_por.controllers.dgvat_user:DgvatUserController',
>> action='logged_out')
>>         map.connect('/user/logged_out_redirect',
>> controller='ckanext.dgvat_por.controllers.dgvat_user:DgvatUserController',
>> action='logged_out_page')
>>         #map.connect('/user/reset', action='request_reset')
>>         #m.connect('/user/me', action='me')
>>         map.connect('/user/set_lang/{lang}',
>> controller='ckanext.dgvat_por.controllers.dgvat_user:DgvatUserController',
>> action='set_lang')
>>         #m.connect('/user/{id:.*}', action='read')
>>         map.connect('/user',
>> controller='ckanext.dgvat_por.controllers.dgvat_user:DgvatUserController',
>> action='index')
>>
>>         map.connect('home', '/',
>> controller='ckanext.dgvat_por.controllers.dgvat_cockpit:DgvatCockpitController',
>> action='search')
>>         map.connect('about', '/about',
>> controller='ckanext.dgvat_por.controllers.dgvat_cockpit:DgvatCockpitController',
>> action='about')
>>         map.connect('terms', '/terms',
>> controller='ckanext.dgvat_por.controllers.dgvat_cockpit:DgvatCockpitController',
>> action='terms')
>>         map.connect('help', '/help',
>> controller='ckanext.dgvat_por.controllers.dgvat_cockpit:DgvatCockpitController',
>> action='help')
>>         map.connect('register_api', '/register_api',
>> controller='ckanext.dgvat_por.controllers.dgvat_cockpit:DgvatCockpitController',
>> action='register')
>>
>>         #log.fatal("==================================> %s" % map)
>>         return map
>>
>>
>>     def after_map(self, map):
>>         #log.fatal("==================================> %s" % map)
>>         return map
>>
>>
>>     def update_config(self, config):
>>         log.debug("update_config")
>>         config['package_form'] = 'data_gv_at'
>>         configure_template_directory(config, 'templates')
>>         configure_public_directory(config, 'public')
>>
>>
>>     def filter(self, stream):
>>
>>         from pylons import request, tmpl_context as c
>>         routes = request.environ.get('pylons.routes_dict')
>>         log.debug(routes)
>>         if routes and \
>>                routes.get('controller') == 'package' and \
>>                routes.get('action') == 'read' and c.pkg.id:
>>
>>             # Add dataset id to the UI
>>             stream = stream_filters.package_id_filter(stream, c.pkg)
>>         return stream
>>
>>
>>     def setup_template_variables(self, context, data_dict):
>>         log.debug("setup_template_variables")
>>
>>     def package_types(self):
>>         return ['dataset']
>>
>>     def is_fallback(self):
>>         return True
>>
>>     def form_to_db_schema(self):
>>         log.fatal("Enter form to db!!!")
>>     def add_to_extras(self, key, data, errors, context):
>>         # get current number of extras
>>         extras = data.get(('extras',), [])
>>         if not extras:
>>             data[('extras',)] = extras
>>         #extras.append({'key': key[-1], 'value': data[key]})
>>         #log.fatal("add to extras: %s, %s" % (key, data[key]))
>>         #log.fatal("extra_number: %s" % extra_number)
>>         #data[('extras', extra_number, 'key')] = key[0]
>>         cats =""
>>         allCat=[]
>>         for cur in data[key]:
>>             if cur != '[' and cur!=']':
>>                 cats += cur
>>         for cat_desc, id in c.categorization:
>>             if id in cats:
>>                 allCat.append(id)
>>         log.fatal("added categorizations:id %s" % allCat)
>>         log.fatal("cats, key: %s,%s" % (cats, data[key]))
>>         log.fatal("is list: %s" % isinstance(data[key], list))
>>         #data[('extras', extra_number, 'value')] = allCat
>>         extras.append({'key': key[-1], 'value': allCat})
>>      def read_template(self):
>>         return 'package/read.html'
>>
>>
>>     def check_data_dict(self, schema=None):
>>         return
>>
>>     def new_template(self):
>>         return 'package/new.html'
>>
>>     def comments_template(self):
>>         return 'package/comments.html'
>>
>>     def search_template(self):
>>         return 'package/search.html'
>>
>>     def read_template(self):
>>         return 'package/read.html'
>>
>>     def history_template(self):
>>         return 'package/history.html'
>>
>>     def package_form(self):
>>         return 'package/datagvat_organization_form.html'
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20130120/5f11fe8f/attachment-0001.html>


More information about the ckan-dev mailing list