[ckan-dev] ckan-dev Digest, Vol 29, Issue 39

Fri Mar 15 19:25:14 UTC 2013

How much RAM is required to run CKAN (on KVM)?

2013/3/15 <ckan-dev-request at lists.okfn.org>

> Send ckan-dev mailing list submissions to
>         ckan-dev at lists.okfn.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         http://lists.okfn.org/mailman/listinfo/ckan-dev
> or, via email, send a message with subject or body 'help' to
>         ckan-dev-request at lists.okfn.org
>
> You can reach the person managing the list at
>         ckan-dev-owner at lists.okfn.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of ckan-dev digest..."
>
>
> Today's Topics:
>
>    1. Re: ckanext-harvest crontab (Joe Tsoi)
>    2. Info on Open ID (Prahadeesh S)
>    3. Re: Info on Open ID (Toby Dacre)
>    4. IDatasetForm: form_to_db_schema_api_create() etc (Sean Hammond)
>    5. Re: IDatasetForm: form_to_db_schema_api_create() etc (Toby Dacre)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Fri, 15 Mar 2013 14:26:43 +0000
> From: Joe Tsoi <joe.tsoi at okfn.org>
> Subject: Re: [ckan-dev] ckanext-harvest crontab
> To: CKAN Development Discussions <ckan-dev at lists.okfn.org>
> Message-ID:
>         <CAA3agXf816r=_6gE4djE84P_n4e-v=tewA-CzhqCPNd2W=
> 8KRg at mail.gmail.com>
> Content-Type: text/plain; charset="iso-8859-1"
>
> oh i missed that part, are you running in the virtual env when you are
> running from the command line, try editing the cronjob to be
>
>
> */5 * * * * /products/ckan/pyenv/bin/python /products/ckan/pyenv/bin/paster
> --plugin=ckanext-harvest harvester run --config=/home/ckan/my.paster.ini
>
>
>
> On 15 March 2013 13:48, Michael Reichart <michael.reichart at gmail.com>
> wrote:
>
> > This is on SLES
> >
> > The question is why it works perfectly, when I run it directly and only
> > fails when using a cronjob...
> > I will try to install the postgresql-libs...
> >
> > thanks!
> > michael
> >
> > 2013/3/15 Joe Tsoi <joe.tsoi at okfn.org>
> >
> >> Looks like a missing shared library issue, what distro are you using?
> Run
> >> a  yum install postgresql-libs or apt-get install libpq5
> >>
> >>
> >>  On 15 March 2013 12:22, Michael Reichart <michael.reichart at gmail.com
> >wrote:
> >>
> >>>  Hi,
> >>>
> >>> I'm trying to set up harvester for automatic import via crontab.
> >>> My problem is: The paster command run by crontab fails, but the same
> >>> script executed manually works with no problem at all.
> >>>
> >>> my cronjob:
> >>> */5 * * * * /products/ckan/pyenv/bin/paster --plugin=ckanext-harvest
> >>> harvester run --config=/home/ckan/my.paster.ini
> >>>
> >>>
> >>> when I run /products/ckan/pyenv/bin/paster --plugin=ckanext-harvest
> >>> harvester run --config=/home/ckan/my.paster.ini from console
> everything is
> >>> ok.
> >>>
> >>> Crontab produces the following error:
> >>> No handlers could be found for logger "vdm"
> >>> Traceback (most recent call last):
> >>>   File "/products/ckan/pyenv/bin/paster", line 8, in <module>
> >>>     load_entry_point('PasteScript==1.7.3', 'console_scripts',
> 'paster')()
> >>>   File
> >>>
> "/products/ckan/pyenv/lib/python2.6/site-packages/paste/script/command.py",
> >>> line 84, in run
> >>>     invoke(command, command_name, options, args[1:])
> >>>   File
> >>>
> "/products/ckan/pyenv/lib/python2.6/site-packages/paste/script/command.py",
> >>> line 123, in invoke
> >>>     exit_code = runner.run(args)
> >>>   File
> >>>
> "/products/ckan/pyenv/lib/python2.6/site-packages/paste/script/command.py",
> >>> line 218, in run
> >>>     result = self.command()
> >>>   File
> >>>
> "/products/ckan/pyenv/lib/python2.6/site-packages/ckanext/harvest/commands/harvester.py",
> >>> line 65, in command
> >>>     self._load_config()
> >>>   File
> >>>
> "/products/ckan/pyenv/lib/python2.6/site-packages/ckanext/harvest/commands/harvester.py",
> >>> line 117, in _load_config
> >>>     super(Harvester, self)._load_config()
> >>>   File
> >>> "/products/ckan/pyenv/lib/python2.6/site-packages/ckan/lib/cli.py",
> line
> >>> 53, in _load_config
> >>>     load_environment(conf.global_conf, conf.local_conf)
> >>>   File
> >>>
> "/products/ckan/pyenv/lib/python2.6/site-packages/ckan/config/environment.py",
> >>> line 192, in load_environment
> >>>     engine = sqlalchemy.engine_from_config(config, 'sqlalchemy.')
> >>>   File
> >>>
> "/products/ckan/pyenv/lib/python2.6/site-packages/sqlalchemy/engine/__init__.py",
> >>> line 298, in engine_from_config
> >>>     return create_engine(url, **opts)
> >>>   File
> >>>
> "/products/ckan/pyenv/lib/python2.6/site-packages/sqlalchemy/engine/__init__.py",
> >>> line 280, in create_engine
> >>>     return strategy.create(*args, **kwargs)
> >>>   File
> >>>
> "/products/ckan/pyenv/lib/python2.6/site-packages/sqlalchemy/engine/strategies.py",
> >>> line 64, in create
> >>>     dbapi = dialect_cls.dbapi(**dbapi_args)
> >>>   File
> >>>
> "/products/ckan/pyenv/lib/python2.6/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py",
> >>> line 283, in dbapi
> >>>     psycopg = __import__('psycopg2')
> >>>   File
> >>>
> "/products/ckan/pyenv/lib/python2.6/site-packages/psycopg2/__init__.py",
> >>> line 60, in <module>
> >>>     from _psycopg import BINARY, NUMBER, STRING, DATETIME, ROWID
> >>> ImportError: libpq.so.5: cannot open shared object file: No such file
> or
> >>> directory
> >>>
> >>>
> >>> Any hints?
> >>>
> >>> Thanks!
> >>> Michael
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> _______________________________________________
> >>> ckan-dev mailing list
> >>> ckan-dev at lists.okfn.org
> >>> http://lists.okfn.org/mailman/listinfo/ckan-dev
> >>> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
> >>>
> >>>
> >>
> >> _______________________________________________
> >> ckan-dev mailing list
> >> ckan-dev at lists.okfn.org
> >> http://lists.okfn.org/mailman/listinfo/ckan-dev
> >> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
> >>
> >>
> >
> > _______________________________________________
> > ckan-dev mailing list
> > ckan-dev at lists.okfn.org
> > http://lists.okfn.org/mailman/listinfo/ckan-dev
> > Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
> >
> >
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://lists.okfn.org/pipermail/ckan-dev/attachments/20130315/fa837aa0/attachment-0001.htm
> >
>
> ------------------------------
>
> Message: 2
> Date: Fri, 15 Mar 2013 10:33:31 -0400
> From: Prahadeesh S <prahad at gmail.com>
> Subject: [ckan-dev] Info on Open ID
> To: ckan-dev at lists.okfn.org
> Message-ID:
>         <CABcTA9gMkULXP4KEqRN6CKEUvRmmi3sOnGZFymPe=
> afnuDM5oA at mail.gmail.com>
> Content-Type: text/plain; charset="iso-8859-1"
>
> Hello Everyone!
>
> I am looking into using Open ID with my CKAN login.
>
> Couldn't find much documentation about this and also found some discussion
> going on about removing the support of OpenID in the future releases.
>
> Currently I am playing with CKAN 2.0  release. Whether "Open ID" support is
>  available with version 2.0? If so can someone throw some info or doc about
> using with Open ID.
>
> Thanks
> Prahadeesh
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://lists.okfn.org/pipermail/ckan-dev/attachments/20130315/d064e529/attachment-0001.htm
> >
>
> ------------------------------
>
> Message: 3
> Date: Fri, 15 Mar 2013 15:01:12 +0000
> From: Toby Dacre <toby.okfn at gmail.com>
> Subject: Re: [ckan-dev] Info on Open ID
> To: CKAN Development Discussions <ckan-dev at lists.okfn.org>
> Message-ID:
>         <CAKZ3V-MfXmBUdJ43sGKaNqWSQ9a2niXqu3z=
> nOs-GNopJC_XOw at mail.gmail.com>
> Content-Type: text/plain; charset=ISO-8859-1
>
> On 15 March 2013 14:33, Prahadeesh S <prahad at gmail.com> wrote:
> >
> > Hello Everyone!
> >
> > I am looking into using Open ID with my CKAN login.
> >
> > Couldn't find much documentation about this and also found some
> discussion
> > going on about removing the support of OpenID in the future releases.
> >
> > Currently I am playing with CKAN 2.0  release. Whether "Open ID" support
> is
> > available with version 2.0? If so can someone throw some info or doc
> about
> > using with Open ID.
>
> Open ID support in ckan is a bit broken imho.  We either need to get
> it fixed properly or remove it.  Currently as I understand things we
> do not have any resources to fix it.  Removing it would also require
> some work so it is in a limbo state at the moment.  Getting it to work
> with the new 2.0 templates would need some work too.
>
> I'm not sure if it has any users at the moment
>
> Toby
> >
> > Thanks
> > Prahadeesh
> >
> >
> >
> > _______________________________________________
> > ckan-dev mailing list
> > ckan-dev at lists.okfn.org
> > http://lists.okfn.org/mailman/listinfo/ckan-dev
> > Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
> >
>
>
>
> ------------------------------
>
> Message: 4
> Date: Fri, 15 Mar 2013 16:36:52 +0100
> From: Sean Hammond <sean.hammond at okfn.org>
> Subject: [ckan-dev] IDatasetForm: form_to_db_schema_api_create() etc
> To: CKAN Development Discussions <ckan-dev at lists.okfn.org>
> Message-ID: <20130315153652.GD9038 at kaeru>
> Content-Type: text/plain; charset=us-ascii
>
> Hey,
>
> The IDatasetForm plugin interface has a method:
>
>     def form_to_db_schema(self):
>         '''Return the schema for mapping dataset dicts from form to db.
>
>         CKAN will use the returned schema to validate and convert data
> coming
>         from users (via the dataset form) before entering that data into
> the
>         database.
>
>         ...
>
> But what really happens is:
>
> - If creating or updating a dataset using the web interface,
>   form_to_db_schema() is called
> - If creating a dataset using the API form_to_db_schema_api_create()
>   is called instead
> - If updating a dataset using the API form_to_db_schema_api_update()
>   is called instead
>
> This only happens if your IDatasetForm plugin inherits from
> DefaultDatasetForm, if it doesn't then form_to_db_schema() is called in
> all three cases.
>
> This logic (selecting when to call form_to_db_schema(),
> form_to_db_schema_api_update() or form_to_db_schema_api_create()) seems
> like it probably belongs in CKAN's logic functions not in a
> form_to_db_schema_options() method in DefaultDatasetForm.
>
> We need to either document form_to_db_schema_options(), _api_create()
> and _api_update() in IDatasetForm, or we need to get rid of the whole
> lot and just call form_to_db_schema() always.
>
> (There's also db_to_form_schema_options() which is also missing from
> IDatasetForm.)
>
> Do we really want to support using different schemas for different
> purposes - web UI vs API, create vs update? It seems that in most cases
> people will want and expect the same schema always. I'm not sure what
> the use-case for this is.
>
> If I'm not mistaken this seems to be the commit that added this feature
> (but that doesn't shed much light on things):
>
>
> https://github.com/okfn/ckan/commit/6f743b636001f6f78f40b0dc9a53cffdd972c851
>
>
>
> ------------------------------
>
> Message: 5
> Date: Fri, 15 Mar 2013 15:51:41 +0000
> From: Toby Dacre <toby.okfn at gmail.com>
> Subject: Re: [ckan-dev] IDatasetForm: form_to_db_schema_api_create()
>         etc
> To: CKAN Development Discussions <ckan-dev at lists.okfn.org>
> Message-ID:
>         <CAKZ3V-Pdpn3PoB1U970g-WYQ6KEwbj1=
> NKKT9tDrZhsYXboBNw at mail.gmail.com>
> Content-Type: text/plain; charset=ISO-8859-1
>
> On 15 March 2013 15:36, Sean Hammond <sean.hammond at okfn.org> wrote:
> > Hey,
> >
> > The IDatasetForm plugin interface has a method:
> >
> >     def form_to_db_schema(self):
> >         '''Return the schema for mapping dataset dicts from form to db.
> >
> >         CKAN will use the returned schema to validate and convert data
> coming
> >         from users (via the dataset form) before entering that data into
> the
> >         database.
> >
> >         ...
> >
> > But what really happens is:
> >
> > - If creating or updating a dataset using the web interface,
> >   form_to_db_schema() is called
> > - If creating a dataset using the API form_to_db_schema_api_create()
> >   is called instead
> > - If updating a dataset using the API form_to_db_schema_api_update()
> >   is called instead
> >
> > This only happens if your IDatasetForm plugin inherits from
> > DefaultDatasetForm, if it doesn't then form_to_db_schema() is called in
> > all three cases.
> >
> > This logic (selecting when to call form_to_db_schema(),
> > form_to_db_schema_api_update() or form_to_db_schema_api_create()) seems
> > like it probably belongs in CKAN's logic functions not in a
> > form_to_db_schema_options() method in DefaultDatasetForm.
> >
> > We need to either document form_to_db_schema_options(), _api_create()
> > and _api_update() in IDatasetForm, or we need to get rid of the whole
> > lot and just call form_to_db_schema() always.
> >
> > (There's also db_to_form_schema_options() which is also missing from
> > IDatasetForm.)
> >
> > Do we really want to support using different schemas for different
> > purposes - web UI vs API, create vs update? It seems that in most cases
> > people will want and expect the same schema always. I'm not sure what
> > the use-case for this is.
> >
> > If I'm not mistaken this seems to be the commit that added this feature
> > (but that doesn't shed much light on things):
> >
> >
> https://github.com/okfn/ckan/commit/6f743b636001f6f78f40b0dc9a53cffdd972c851
> >
>
> I am indeed responsible for this sick twistedness.  This stems from
> various things.  Essentially this is due to the change from the all in
> one dataset creation of 1.8 and the 3 stage 2.0.  The thing is that
> changing the schema to allow partial creation for the front-end broke
> lots of tests among other things.
>
> I think that we should do the following.
>
> have a `create` and an `update` schema that is used by both the api
> and the front end it is stupid to have data counted as ok one way but
> not the other.  Hopefully the allow partial updates stuff done will
> help here - before the data would get stripped out of datasets if not
> supplied.
>
> this just feels the way we should do things - maybe we could even get
> to a single schema not even a create/update
>
> kill the old template tests and replace with new template based front
> end testing
>
> clean up the IDatasetForm to deprecate and then kill all these extra
> methods.
>
> Toby
>
> > _______________________________________________
> > ckan-dev mailing list
> > ckan-dev at lists.okfn.org
> > http://lists.okfn.org/mailman/listinfo/ckan-dev
> > Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
>
>
>
> ------------------------------
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/optionss/ckan-dev
>
>
> End of ckan-dev Digest, Vol 29, Issue 39
> ****************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20130315/bdb748cd/attachment.html>