[ckan-dev] Harvester - Problem
Michael Reichart
michael.reichart at gmail.com
Tue Jan 15 07:07:43 UTC 2013
Hi,
I also thought it might be the custom schema, but I couldn't figure out
what's wrong with it.
That's our code:
def form_to_db_schema(self):
log.debug("Enter form to db!!!")
schema = {
'title': [if_empty_same_as("name"), unicode],
'name': [not_empty, unicode, val.name_validator,
val.package_name_validator],
'license_id': [ignore_missing, unicode],
'maintainer': [ignore_missing, unicode],
'schema_name': [ignore_missing, unicode,
convert_to_extras],
'maintainer_link': [ignore_missing, unicode,
convert_to_extras],
'schema_language': [ignore_missing,
unicode,convert_to_extras],
'schema_characterset': [ignore_missing,
unicode,convert_to_extras],
'date_released': [ignore_missing, unicode,
convert_to_extras],
'begin_datetime': [ignore_missing, unicode,
convert_to_extras],
'end_datetime': [ignore_missing, unicode,
convert_to_extras],
'metadata_linkage': [ignore_missing, unicode,
convert_to_extras],
'attribute_description': [ignore_missing, unicode,
convert_to_extras],
'publisher': [ignore_missing, unicode, convert_to_extras],
'geographic_toponym': [ignore_missing, unicode,
convert_to_extras],
'geographic_bbox': [ignore_missing, unicode,
convert_to_extras],
'lineage_quality': [ignore_missing, unicode,
convert_to_extras],
'en_title_and_desc': [ignore_missing, unicode,
convert_to_extras],
'license_citation':[ignore_missing, unicode,
convert_to_extras],
'metadata_identifier':[ignore_missing, unicode,
convert_to_extras],
'metadata_modified':[ignore_missing, unicode,
convert_to_extras],
'date_updated':[ignore_missing, unicode,
convert_to_extras],
'url': [ignore_missing, unicode],
'resources': default_schema.default_resource_schema(),
'state': [val.ignore_not_admin, ignore_missing],
'log_message': [unicode, val.no_http],
'__extras': [ignore],
'revision_id': [ignore],
'update_frequency': [ignore_missing, unicode,
convert_to_extras],
'categorization': [ignore_missing, unicode,
self.add_to_extras],
'notes': [ignore_missing, unicode],
'tag_string': [ignore_missing, val.tag_string_convert],
'groups': {
'id': [ignore_missing, unicode],
'capacity': [ignore_missing, unicode],
'__extras': [ignore],
},
}
schema['resources'].update({
'language':[ignore_missing],
'characterset':[ignore_missing]
})
return schema
def db_to_form_schema(self):
log.debug("Enter db to form")
#c.categorization = gv.categorization
#c.update_frequency = gv.update_frequency
schema = logic.schema.package_form_schema()
schema.update({
'schema_name': [convert_from_extras, ignore_missing, unicode],
'schema_language': [convert_from_extras, ignore_missing,
unicode],
'maintainer_link': [convert_from_extras, ignore_missing,
unicode],
'date_released': [convert_from_extras, ignore_missing, unicode],
'begin_datetime': [convert_from_extras, ignore_missing,
unicode],
'end_datetime': [convert_from_extras, ignore_missing, unicode],
'metadata_linkage': [convert_from_extras, ignore_missing,
unicode],
'attribute_description': [convert_from_extras, ignore_missing,
unicode],
'publisher': [convert_from_extras, ignore_missing, unicode],
'geographic_toponym': [convert_from_extras, ignore_missing,
unicode],
'geographic_bbox': [convert_from_extras, ignore_missing,
unicode],
'lineage_quality': [convert_from_extras, ignore_missing,
unicode],
'en_title_and_desc': [convert_from_extras, ignore_missing,
unicode],
'license_citation':[convert_from_extras, ignore_missing,
unicode],
'metadata_identifier':[convert_from_extras, ignore_missing,
unicode],
'metadata_modified':[convert_from_extras, ignore_missing,
unicode],
'date_updated':[convert_from_extras, ignore_missing, unicode],
'resources': default_schema.default_resource_schema(),
'categorization': [convert_from_extras, ignore_missing],
'update_frequency': [convert_from_extras, ignore_missing],
'extras': {
'key': [],
'value': [],
'__extras': [keep_extras]
},
'tags': {
'__extras': [keep_extras]
},
'mandate': [convert_from_extras, ignore_missing],
'national_statistic': [convert_from_extras, ignore_missing],
'__extras': [keep_extras],
})
schema['groups'].update({
'name': [not_empty, unicode],
'title': [ignore_missing],
'capacity': [ignore_missing, unicode]
})
schema['resources'].update({
'created': [ignore_missing],
'position': [not_empty],
'last_modified': [ignore_missing],
'cache_last_updated': [ignore_missing],
'webstore_last_updated': [ignore_missing],
'language':[ignore_missing],
'characterset':[ignore_missing]
})
return schema
Thanks for your help!
Michael
2013/1/14 Sean Hammond <sean.hammond at okfn.org>
> Hi Michael,
>
> This can probably be fixed by changing your custom schema.
>
> This sounds like it's a custom schema problem and probably has nothing
> to do with the harvesters. If you create a dataset belonging to a group
> using the package_create APIv3 call, does it get added to the group?
> What if you update a dataset using package_update? I suspect you might
> be able to reproduce your problem just by calling the API with your
> custom schema active, without using harvester at all. That would confirm
> it's a schema problem.
>
> You're using quite an old version of CKAN, I know there were some bugs
> in IDatasetForm's default schemas in older versions of CKAN and I think
> the fixes may have gone into CKAN 1.8, so you may still have the bugs.
> The bugs can be worked around by adding stuff to your custom schema. For
> example, look at ckanext-ecportal:
>
>
> https://github.com/okfn/ckanext-ecportal/blob/master/ckanext/ecportal/forms.py#L181
>
> Finally, can you post your custom schema, or rather, your
> form_to_db_schema, db_to_form_schema, etc. code? That might give us some
> clues.
>
> On Mon, Jan 14, 2013 at 09:25:30AM +0100, Michael Reichart wrote:
> > Hi,
> >
> > I've got a problem with our harvester-extensions.
> >
> > We are harvesting CKAN-Instances which worked actually really good, until
> > we changed our CKAN - inputform to insert our extra fields into CKAN. For
> > this we have also changed our schema, which now seems to be the problem.
> > The objects from the harvesters are already in the right form to put into
> > CKAN, but in the harvesting process, they are also sent through our
> > "form_to_db_schema" methods.
> > The problem is, that this results in not being able to correctly add
> > datasets to groups. Even existing datasets in correct groups, that are
> > being updated loose their relation to the group.
> >
> > I've tried to hardcode the groups or set them in the
> > "default_groups"-setting of the harvester, or put them in the same way as
> > the input-form does, but nothing worked.
> > Any ideas?? Can I bypass the "form_to_db_schema"?? Or change the schema?
> >
> > We are currently using CKAN 1.7.1
> >
> > Thanks for help
> > michi
>
> > _______________________________________________
> > ckan-dev mailing list
> > ckan-dev at lists.okfn.org
> > http://lists.okfn.org/mailman/listinfo/ckan-dev
> > Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20130115/77e003ba/attachment-0001.html>
More information about the ckan-dev
mailing list