[ckan-dev] ckan-dev Digest, Vol 29, Issue 37
Илья Громислав
igromislav at gmail.com
Fri Mar 15 14:06:20 UTC 2013
Say, what version of CKAN should I use? What are the hardware requirements
for comfortable operation of the project on KVM (Ubuntu)? Can I use Ubuntu
12.4? Do you know any russian-speaking professionals who use CKAN?
or the Sun () to set and ?
2013/3/15 <ckan-dev-request at lists.okfn.org>
> Send ckan-dev mailing list submissions to
> ckan-dev at lists.okfn.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> or, via email, send a message with subject or body 'help' to
> ckan-dev-request at lists.okfn.org
>
> You can reach the person managing the list at
> ckan-dev-owner at lists.okfn.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of ckan-dev digest..."
>
>
> Today's Topics:
>
> 1. ckanext-harvest crontab (Michael Reichart)
> 2. Re: FIxing the DataStore vs FileStore duality confusion
> (Sean Hammond)
> 3. Re: FIxing the DataStore vs FileStore duality confusion
> (Sean Hammond)
> 4. Re: ckanext-harvest crontab (Joe Tsoi)
> 5. Re: ckanext-harvest crontab (Michael Reichart)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Fri, 15 Mar 2013 13:22:45 +0100
> From: Michael Reichart <michael.reichart at gmail.com>
> Subject: [ckan-dev] ckanext-harvest crontab
> To: CKAN Development Discussions <ckan-dev at lists.okfn.org>
> Message-ID:
> <
> CAPCDpiTtOUp7OVc66O1FR21K-ZMntNTjhwXr43m6i18MmeuoBg at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Hi,
>
> I'm trying to set up harvester for automatic import via crontab.
> My problem is: The paster command run by crontab fails, but the same script
> executed manually works with no problem at all.
>
> my cronjob:
> */5 * * * * /products/ckan/pyenv/bin/paster --plugin=ckanext-harvest
> harvester run --config=/home/ckan/my.paster.ini
>
>
> when I run /products/ckan/pyenv/bin/paster --plugin=ckanext-harvest
> harvester run --config=/home/ckan/my.paster.ini from console everything is
> ok.
>
> Crontab produces the following error:
> No handlers could be found for logger "vdm"
> Traceback (most recent call last):
> File "/products/ckan/pyenv/bin/paster", line 8, in <module>
> load_entry_point('PasteScript==1.7.3', 'console_scripts', 'paster')()
> File
> "/products/ckan/pyenv/lib/python2.6/site-packages/paste/script/command.py",
> line 84, in run
> invoke(command, command_name, options, args[1:])
> File
> "/products/ckan/pyenv/lib/python2.6/site-packages/paste/script/command.py",
> line 123, in invoke
> exit_code = runner.run(args)
> File
> "/products/ckan/pyenv/lib/python2.6/site-packages/paste/script/command.py",
> line 218, in run
> result = self.command()
> File
>
> "/products/ckan/pyenv/lib/python2.6/site-packages/ckanext/harvest/commands/harvester.py",
> line 65, in command
> self._load_config()
> File
>
> "/products/ckan/pyenv/lib/python2.6/site-packages/ckanext/harvest/commands/harvester.py",
> line 117, in _load_config
> super(Harvester, self)._load_config()
> File "/products/ckan/pyenv/lib/python2.6/site-packages/ckan/lib/cli.py",
> line 53, in _load_config
> load_environment(conf.global_conf, conf.local_conf)
> File
>
> "/products/ckan/pyenv/lib/python2.6/site-packages/ckan/config/environment.py",
> line 192, in load_environment
> engine = sqlalchemy.engine_from_config(config, 'sqlalchemy.')
> File
>
> "/products/ckan/pyenv/lib/python2.6/site-packages/sqlalchemy/engine/__init__.py",
> line 298, in engine_from_config
> return create_engine(url, **opts)
> File
>
> "/products/ckan/pyenv/lib/python2.6/site-packages/sqlalchemy/engine/__init__.py",
> line 280, in create_engine
> return strategy.create(*args, **kwargs)
> File
>
> "/products/ckan/pyenv/lib/python2.6/site-packages/sqlalchemy/engine/strategies.py",
> line 64, in create
> dbapi = dialect_cls.dbapi(**dbapi_args)
> File
>
> "/products/ckan/pyenv/lib/python2.6/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py",
> line 283, in dbapi
> psycopg = __import__('psycopg2')
> File
> "/products/ckan/pyenv/lib/python2.6/site-packages/psycopg2/__init__.py",
> line 60, in <module>
> from _psycopg import BINARY, NUMBER, STRING, DATETIME, ROWID
> ImportError: libpq.so.5: cannot open shared object file: No such file or
> directory
>
>
> Any hints?
>
> Thanks!
> Michael
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://lists.okfn.org/pipermail/ckan-dev/attachments/20130315/01bfd6b6/attachment-0001.htm
> >
>
> ------------------------------
>
> Message: 2
> Date: Fri, 15 Mar 2013 13:42:17 +0100
> From: Sean Hammond <sean.hammond at okfn.org>
> Subject: Re: [ckan-dev] FIxing the DataStore vs FileStore duality
> confusion
> To: ckan-dev at lists.okfn.org
> Message-ID: <20130315124217.GA9038 at kaeru>
> Content-Type: text/plain; charset=us-ascii
>
> > By the way, I admit my user stories covered functionality beyond the
> > basic datastore/filestore - I included them because I think they all
> > nevertheless bear on what kind of behaviour users will expect from
> > CKAN's storage, and in all cases, this is that the user just wants to
> > know 'Have you got the data or not?' not 'Have you got it in your pink
> > folder? How about your purple folder?' So I agree that the 'suggested
> > behaviour' is far too complicated and confusing (and so is the current
> > behaviour).
>
> I think the functionality implied by these user stories sounds really
> great though, if we ever decide to implement those features, they take
> CKAN closer to being the "github of open data". With the 'alternative
> suggestion' on the wiki page, I think space is still left to implement
> this sort of thing (whereas with the original suggestion, when the
> datastore was read-only except for special datastore resources, I was a bit
> worried that we were closing off future development).
>
> There are some pain points with the alternative suggested behaviour:
>
> - Need for separate 'download' and 'download original file' buttons,
> will users understand the difference/can we present it clearly?
>
> - Need for warning messages when uploading a new copy of a file: this
> will overwrite the previously uploaded file and (if the data has been
> edited in the datastore) the modified data in CKAN - will this warning
> confuse users?
>
> > One comment: I appreciate that datastore versioning, nice though it
> > might be, is a long way off. But how feasible is it to at least keep a
> > list of edit time + user for each (editable) datastore resource?
>
> I haven't looked at how the datastore works internally but I would guess
> this would be quite easy.
>
>
>
> ------------------------------
>
> Message: 3
> Date: Fri, 15 Mar 2013 14:01:32 +0100
> From: Sean Hammond <sean.hammond at okfn.org>
> Subject: Re: [ckan-dev] FIxing the DataStore vs FileStore duality
> confusion
> To: ckan-dev at lists.okfn.org
> Message-ID: <20130315130132.GB9038 at kaeru>
> Content-Type: text/plain; charset=us-ascii
>
> > >Just to update the dev list on this, there's a wiki page here outlining
> > >the problems and suggested solutions:
> > >
> >
> > Hi, Sean.
> >
> > Some of these ideas look good for clearing up confusion....
> >
> > But they don't address any of the major architectural issues I've
> > run into in actually trying to deploy datastore:
>
> Alright, this spec is about dealing with the user confusion caused by
> the datastore and filestore being independent of eachother and
> potentially diverging etc. Let's try and keep this on topic as it's
> already wandered quite a lot. If there are other problems with the
> datastore lets deal with them separately..
>
> If you're working on a datastore loader script, you may want to
> coordinate with Dominik Moritz and Nigel Babu as they're currently
> working on a datastore loading service and script, which are intended to
> replace the existing ckanext-datastorer. It sounds like they're also
> trying to solve some of the same issues as you are.
>
> https://github.com/okfn/datapusher
> https://github.com/okfn/ckanext-sa/pull/3
>
>
>
> ------------------------------
>
> Message: 4
> Date: Fri, 15 Mar 2013 13:22:26 +0000
> From: Joe Tsoi <joe.tsoi at okfn.org>
> Subject: Re: [ckan-dev] ckanext-harvest crontab
> To: CKAN Development Discussions <ckan-dev at lists.okfn.org>
> Message-ID:
> <
> CAA3agXcGNabjgZarXyDpN1LOFBCv8fa15+DsyttJo6BDHatgSg at mail.gmail.com>
> Content-Type: text/plain; charset="iso-8859-1"
>
> Looks like a missing shared library issue, what distro are you using? Run
> a yum install postgresql-libs or apt-get install libpq5
>
>
> On 15 March 2013 12:22, Michael Reichart <michael.reichart at gmail.com>
> wrote:
>
> > Hi,
> >
> > I'm trying to set up harvester for automatic import via crontab.
> > My problem is: The paster command run by crontab fails, but the same
> > script executed manually works with no problem at all.
> >
> > my cronjob:
> > */5 * * * * /products/ckan/pyenv/bin/paster --plugin=ckanext-harvest
> > harvester run --config=/home/ckan/my.paster.ini
> >
> >
> > when I run /products/ckan/pyenv/bin/paster --plugin=ckanext-harvest
> > harvester run --config=/home/ckan/my.paster.ini from console everything
> is
> > ok.
> >
> > Crontab produces the following error:
> > No handlers could be found for logger "vdm"
> > Traceback (most recent call last):
> > File "/products/ckan/pyenv/bin/paster", line 8, in <module>
> > load_entry_point('PasteScript==1.7.3', 'console_scripts', 'paster')()
> > File
> >
> "/products/ckan/pyenv/lib/python2.6/site-packages/paste/script/command.py",
> > line 84, in run
> > invoke(command, command_name, options, args[1:])
> > File
> >
> "/products/ckan/pyenv/lib/python2.6/site-packages/paste/script/command.py",
> > line 123, in invoke
> > exit_code = runner.run(args)
> > File
> >
> "/products/ckan/pyenv/lib/python2.6/site-packages/paste/script/command.py",
> > line 218, in run
> > result = self.command()
> > File
> >
> "/products/ckan/pyenv/lib/python2.6/site-packages/ckanext/harvest/commands/harvester.py",
> > line 65, in command
> > self._load_config()
> > File
> >
> "/products/ckan/pyenv/lib/python2.6/site-packages/ckanext/harvest/commands/harvester.py",
> > line 117, in _load_config
> > super(Harvester, self)._load_config()
> > File
> "/products/ckan/pyenv/lib/python2.6/site-packages/ckan/lib/cli.py",
> > line 53, in _load_config
> > load_environment(conf.global_conf, conf.local_conf)
> > File
> >
> "/products/ckan/pyenv/lib/python2.6/site-packages/ckan/config/environment.py",
> > line 192, in load_environment
> > engine = sqlalchemy.engine_from_config(config, 'sqlalchemy.')
> > File
> >
> "/products/ckan/pyenv/lib/python2.6/site-packages/sqlalchemy/engine/__init__.py",
> > line 298, in engine_from_config
> > return create_engine(url, **opts)
> > File
> >
> "/products/ckan/pyenv/lib/python2.6/site-packages/sqlalchemy/engine/__init__.py",
> > line 280, in create_engine
> > return strategy.create(*args, **kwargs)
> > File
> >
> "/products/ckan/pyenv/lib/python2.6/site-packages/sqlalchemy/engine/strategies.py",
> > line 64, in create
> > dbapi = dialect_cls.dbapi(**dbapi_args)
> > File
> >
> "/products/ckan/pyenv/lib/python2.6/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py",
> > line 283, in dbapi
> > psycopg = __import__('psycopg2')
> > File
> > "/products/ckan/pyenv/lib/python2.6/site-packages/psycopg2/__init__.py",
> > line 60, in <module>
> > from _psycopg import BINARY, NUMBER, STRING, DATETIME, ROWID
> > ImportError: libpq.so.5: cannot open shared object file: No such file or
> > directory
> >
> >
> > Any hints?
> >
> > Thanks!
> > Michael
> >
> >
> >
> >
> >
> > _______________________________________________
> > ckan-dev mailing list
> > ckan-dev at lists.okfn.org
> > http://lists.okfn.org/mailman/listinfo/ckan-dev
> > Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
> >
> >
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://lists.okfn.org/pipermail/ckan-dev/attachments/20130315/90c5c1d0/attachment-0001.htm
> >
>
> ------------------------------
>
> Message: 5
> Date: Fri, 15 Mar 2013 14:48:17 +0100
> From: Michael Reichart <michael.reichart at gmail.com>
> Subject: Re: [ckan-dev] ckanext-harvest crontab
> To: CKAN Development Discussions <ckan-dev at lists.okfn.org>
> Message-ID:
> <
> CAPCDpiQirCUtiMumCypcT62hx3iXQAEOaG772NKJpJVyXR2Vfg at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> This is on SLES
>
> The question is why it works perfectly, when I run it directly and only
> fails when using a cronjob...
> I will try to install the postgresql-libs...
>
> thanks!
> michael
>
> 2013/3/15 Joe Tsoi <joe.tsoi at okfn.org>
>
> > Looks like a missing shared library issue, what distro are you using? Run
> > a yum install postgresql-libs or apt-get install libpq5
> >
> >
> > On 15 March 2013 12:22, Michael Reichart <michael.reichart at gmail.com
> >wrote:
> >
> >> Hi,
> >>
> >> I'm trying to set up harvester for automatic import via crontab.
> >> My problem is: The paster command run by crontab fails, but the same
> >> script executed manually works with no problem at all.
> >>
> >> my cronjob:
> >> */5 * * * * /products/ckan/pyenv/bin/paster --plugin=ckanext-harvest
> >> harvester run --config=/home/ckan/my.paster.ini
> >>
> >>
> >> when I run /products/ckan/pyenv/bin/paster --plugin=ckanext-harvest
> >> harvester run --config=/home/ckan/my.paster.ini from console everything
> is
> >> ok.
> >>
> >> Crontab produces the following error:
> >> No handlers could be found for logger "vdm"
> >> Traceback (most recent call last):
> >> File "/products/ckan/pyenv/bin/paster", line 8, in <module>
> >> load_entry_point('PasteScript==1.7.3', 'console_scripts',
> 'paster')()
> >> File
> >>
> "/products/ckan/pyenv/lib/python2.6/site-packages/paste/script/command.py",
> >> line 84, in run
> >> invoke(command, command_name, options, args[1:])
> >> File
> >>
> "/products/ckan/pyenv/lib/python2.6/site-packages/paste/script/command.py",
> >> line 123, in invoke
> >> exit_code = runner.run(args)
> >> File
> >>
> "/products/ckan/pyenv/lib/python2.6/site-packages/paste/script/command.py",
> >> line 218, in run
> >> result = self.command()
> >> File
> >>
> "/products/ckan/pyenv/lib/python2.6/site-packages/ckanext/harvest/commands/harvester.py",
> >> line 65, in command
> >> self._load_config()
> >> File
> >>
> "/products/ckan/pyenv/lib/python2.6/site-packages/ckanext/harvest/commands/harvester.py",
> >> line 117, in _load_config
> >> super(Harvester, self)._load_config()
> >> File
> >> "/products/ckan/pyenv/lib/python2.6/site-packages/ckan/lib/cli.py", line
> >> 53, in _load_config
> >> load_environment(conf.global_conf, conf.local_conf)
> >> File
> >>
> "/products/ckan/pyenv/lib/python2.6/site-packages/ckan/config/environment.py",
> >> line 192, in load_environment
> >> engine = sqlalchemy.engine_from_config(config, 'sqlalchemy.')
> >> File
> >>
> "/products/ckan/pyenv/lib/python2.6/site-packages/sqlalchemy/engine/__init__.py",
> >> line 298, in engine_from_config
> >> return create_engine(url, **opts)
> >> File
> >>
> "/products/ckan/pyenv/lib/python2.6/site-packages/sqlalchemy/engine/__init__.py",
> >> line 280, in create_engine
> >> return strategy.create(*args, **kwargs)
> >> File
> >>
> "/products/ckan/pyenv/lib/python2.6/site-packages/sqlalchemy/engine/strategies.py",
> >> line 64, in create
> >> dbapi = dialect_cls.dbapi(**dbapi_args)
> >> File
> >>
> "/products/ckan/pyenv/lib/python2.6/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py",
> >> line 283, in dbapi
> >> psycopg = __import__('psycopg2')
> >> File
> >> "/products/ckan/pyenv/lib/python2.6/site-packages/psycopg2/__init__.py",
> >> line 60, in <module>
> >> from _psycopg import BINARY, NUMBER, STRING, DATETIME, ROWID
> >> ImportError: libpq.so.5: cannot open shared object file: No such file or
> >> directory
> >>
> >>
> >> Any hints?
> >>
> >> Thanks!
> >> Michael
> >>
> >>
> >>
> >>
> >>
> >> _______________________________________________
> >> ckan-dev mailing list
> >> ckan-dev at lists.okfn.org
> >> http://lists.okfn.org/mailman/listinfo/ckan-dev
> >> Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
> >>
> >>
> >
> > _______________________________________________
> > ckan-dev mailing list
> > ckan-dev at lists.okfn.org
> > http://lists.okfn.org/mailman/listinfo/ckan-dev
> > Unsubscribe: http://lists.okfn.org/mailman/options/ckan-dev
> >
> >
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://lists.okfn.org/pipermail/ckan-dev/attachments/20130315/90666035/attachment.htm
> >
>
> ------------------------------
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> http://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: http://lists.okfn.org/mailman/optionss/ckan-dev
>
>
> End of ckan-dev Digest, Vol 29, Issue 37
> ****************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20130315/c940afde/attachment.html>
More information about the ckan-dev
mailing list