[ckan-dev] problems with paster db init - can't parse URL
Jason Tan
jason.tan at ersa.edu.au
Thu Jun 19 01:30:12 UTC 2014
Thanks for your reply Nigel,
It's taken me a few days of expended time but I finally worked this problem
out.
The hint was in the stack trace - of course.
*The problem was I had "CKAN_DB=ckan_prd" set as an env var.*
It took a while to work out - I had been to the point of removing all of
the instances of postgres connection strings and "ckan_prd" from the config
file and was still getting errors when I ran the db init
So I wondered if I or someone else might have inadvertently edited some of
the code, e.g. edited a python file without realizing.
So I installed from src and tried again. Same error.
At that point I was sort of stuffed.
It's not in the config, it's not in the code. Where can it be?
Then after staring at the screen for 10 minutes or so I read the
stacktrace again and saw:
File "/usr/lib/ckan/prd/src/ckan/ckan/config/*environment*.py", line 344,
in load_environment
engine = sqlalchemy.engine_from_config(config, 'sqlalchemy.', **extras)
Hmmm.. are you looking at the shell environment Mr CKAN?
Well, yes as it turns out.
Problem solved.
Regards
Jason Tan
On 13 June 2014 18:10, Nigel Babu <nigel.babu at okfn.org> wrote:
> Since we know it's one particular line that it can't parse, try removing
> disabling the datastore and trying each of those URLs for sqlalchemy.url
> field
> and see which one throws and error.
>
> This is at the sqlalchemy error and the only string it gets is 'ckan_prd',
> instead of the full URL.
>
> On Fri, Jun 13, 2014 at 02:44:58PM +0930, Jason Tan wrote:
> > Thanks for taking a look NIgel.
> >
> > The passwords use the exact same characters as in the string. I've just
> > messed the order of the chars around in the email.
> > I was conscious that it might be a special character so I was careful to
> > include all the actual characters - though as you say - no character
> looks
> > very special in those strings.
> > I was also concerned about unprintable characters so I've checked in vi
> > with "set list" set , with no unprintable shown, and I've also cut and
> > paste the passwords. usernames and hostnames in the email into psql on
> the
> > command line, before I moved the order around and it works fine.
> > My very initial thought were "oh I've screwed up pg_hba.conf, what's
> new?".
> > But that appears not be the case since psql from the app server works
> where
> > paster run is OK.
> >
> > I'll keep plugging away and let you all know when I find a solution/work
> > around or cause.
> > Mean while if anyone has any suggestions, I'm all ears.
> >
> > Thanks again Nigel.
> >
> > Regards
> > Jason
> >
> >
> >
> > On 13 June 2014 13:31, Nigel Babu <nigel.babu at okfn.org> wrote:
> >
> > > Hi Jason,
> > >
> > > I can't find anything obviously wrong. But from the error, it's fairly
> > > clear,
> > > it's a typo of some sort. Do you happen to have '@' or ':' characters
> in
> > > your
> > > password?
> > >
> > > On Thu, Jun 12, 2014 at 09:55:45AM +0930, Jason Tan wrote:
> > > > Hi Folks,
> > > >
> > > > Sorry to be asking here, this feels like a problem I should be able
> to
> > > > solve myself, but I've been trying to work it out for most of two
> days
> > > with
> > > > no luck.
> > > >
> > > > I am installing CKAN 2.1.1 and am having a problem with the
> > > > paster db init command.
> > > >
> > > > The complete error is below, but the last line of the error is:
> > > > "Could not parse rfc1738 URL from string '%s'" % name)
> > > > sqlalchemy.exc.ArgumentError: Could not parse rfc1738 URL from string
> > > > * 'ckan_prd'*
> > > >
> > > >
> > > > The problem appears to be with one of the database connection
> strings.
> > > > If I copy and paste the sername, passwd and host from the database
> > > > connection strings to psql on the app server the strings seem to
> work OK
> > > >
> > > > Here are the database strings:
> > > > (prd)[ckan at ckan-app ckan]$ echo $NODE
> > > > prd
> > > > (prd)[ckan at ckan-app ckan]$ grep -i postgresql
> > > /etc/ckan/${NODE}/${NODE}.ini
> > > >
> > > > sqlalchemy.url = postgresql://
> > > > ckan_prd:O1eakfeo at ckan-db-prd.green.ersa.edu.au/ckan_prd
> > > >
> > > > ckan.datastore.write_url = postgresql://
> > > > ckan_prd:O1eakfeo at ckan-db-prd.green.ersa.edu.au/datastore_prd
> > > >
> > > > ckan.datastore.read_url = postgresql://
> > > > datastore_prd:P5ngoueU at ckan-db-prd.green.ersa.edu.au/datastore_prd
> > > >
> > > > I have this running in dev and apart from the fact that I am using
> > > > "default" instead of prd in database and usernames and I have
> different
> > > > passwords, the database strings look the same to me.
> > > >
> > > > I've found this link
> > > > https://code.google.com/p/sqlalchemy-migrate/issues/detail?id=54
> which
> > > > suggests there was a bug under python 2.5 but I am using 2.6 and the
> same
> > > > version in dev, which appears to work OK and I've got the most recent
> > > > pyhton package available for the distro (centos 6.5).
> > > >
> > > > Below is a more complete output of the steps I am taking with
> commentary
> > > > inbold.
> > > >
> > > > Thanks and Regards
> > > > Jason
> > > >
> > > >
> > > >
> > > > *I have an env var NODE which I use in the command lines. In this
> > > instance
> > > > it evaluates to prd*
> > > > (prd)[ckan at ckan-app ckan]$ echo $NODE
> > > > prd
> > > >
> > > > *I think I am in the correct dir*
> > > > (prd)[ckan at ckan-app ckan]$ pwd
> > > > /usr/lib/ckan/prd/src/ckan
> > > >
> > > > *lets look for ckan_ - which is a string the exception stack trace
> refers
> > > > to:*
> > > > (prd)[ckan at ckan-app ckan]$ grep -i ckan_ /etc/ckan/prd/prd.ini
> > > > sqlalchemy.url = postgresql://
> > > > ckan_prd:O1eakfeo at ckan-db-prd.green.ersa.edu.au/ckan_prd
> > > > #sqlalchemy.url = postgresql://
> > > > ckan_prd:O1eakfeo at ckan-db-prd.green.ersa.edu.au/ckan_prd
> > > > #sqlalchemy.url = postgresql://
> > > > ckan_prd:O1eakfeo at ckan-db-prd.green.ersa.edu.au/ckan_prd
> > > > ckan.datastore.write_url = postgresql://
> > > > ckan_prd:O1eakfeo at ckan-db-prd.green.ersa.edu.au/datastore_prd
> > > >
> > > > *check that env var NODE is set, then run the command*
> > > > (prd)[ckan at ckan-app ckan]$ echo $NODE
> > > > prd
> > > > (prd)[ckan at ckan-app ckan]$ paster db init -c
> > > /etc/ckan/${NODE}/${NODE}.ini
> > > > Traceback (most recent call last):
> > > > File "/usr/lib/ckan/prd/bin/paster", line 9, in <module>
> > > > load_entry_point('PasteScript==1.7.5', 'console_scripts',
> 'paster')()
> > > > File
> > > >
> "/usr/lib/ckan/prd/lib/python2.6/site-packages/paste/script/command.py",
> > > > line 104, in run
> > > > invoke(command, command_name, options, args[1:])
> > > > File
> > > >
> "/usr/lib/ckan/prd/lib/python2.6/site-packages/paste/script/command.py",
> > > > line 143, in invoke
> > > > exit_code = runner.run(args)
> > > > File
> > > >
> "/usr/lib/ckan/prd/lib/python2.6/site-packages/paste/script/command.py",
> > > > line 238, in run
> > > > result = self.command()
> > > > File "/usr/lib/ckan/prd/src/ckan/ckan/lib/cli.py", line 129, in
> command
> > > > self._load_config()
> > > > File "/usr/lib/ckan/prd/src/ckan/ckan/lib/cli.py", line 91, in
> > > > _load_config
> > > > load_environment(conf.global_conf, conf.local_conf)
> > > > File "/usr/lib/ckan/prd/src/ckan/ckan/config/environment.py", line
> 344,
> > > > in load_environment
> > > > engine = sqlalchemy.engine_from_config(config, 'sqlalchemy.',
> > > **extras)
> > > > File
> > > >
> > >
> "/usr/lib/ckan/prd/lib/python2.6/site-packages/sqlalchemy/engine/__init__.py",
> > > > line 356, in engine_from_config
> > > > return create_engine(url, **opts)
> > > > File
> > > >
> > >
> "/usr/lib/ckan/prd/lib/python2.6/site-packages/sqlalchemy/engine/__init__.py",
> > > > line 338, in create_engine
> > > > return strategy.create(*args, **kwargs)
> > > > File
> > > >
> > >
> "/usr/lib/ckan/prd/lib/python2.6/site-packages/sqlalchemy/engine/strategies.py",
> > > > line 48, in create
> > > > u = url.make_url(name_or_url)
> > > > File
> > > >
> "/usr/lib/ckan/prd/lib/python2.6/site-packages/sqlalchemy/engine/url.py",
> > > > line 178, in make_url
> > > > return _parse_rfc1738_args(name_or_url)
> > > > File
> > > >
> "/usr/lib/ckan/prd/lib/python2.6/site-packages/sqlalchemy/engine/url.py",
> > > > line 219, in _parse_rfc1738_args
> > > > "Could not parse rfc1738 URL from string '%s'" % name)
> > > > sqlalchemy.exc.ArgumentError: Could not parse rfc1738 URL from string
> > > > 'ckan_prd'
> > > > (prd)[ckan at ckan-app ckan]$ paster db init -c /etc/ckan/prd/prd.ini
> > > > Traceback (most recent call last):
> > > > File "/usr/lib/ckan/prd/bin/paster", line 9, in <module>
> > > > load_entry_point('PasteScript==1.7.5', 'console_scripts',
> 'paster')()
> > > > File
> > > >
> "/usr/lib/ckan/prd/lib/python2.6/site-packages/paste/script/command.py",
> > > > line 104, in run
> > > > invoke(command, command_name, options, args[1:])
> > > > File
> > > >
> "/usr/lib/ckan/prd/lib/python2.6/site-packages/paste/script/command.py",
> > > > line 143, in invoke
> > > > exit_code = runner.run(args)
> > > > File
> > > >
> "/usr/lib/ckan/prd/lib/python2.6/site-packages/paste/script/command.py",
> > > > line 238, in run
> > > > result = self.command()
> > > > File "/usr/lib/ckan/prd/src/ckan/ckan/lib/cli.py", line 129, in
> command
> > > > self._load_config()
> > > > File "/usr/lib/ckan/prd/src/ckan/ckan/lib/cli.py", line 91, in
> > > > _load_config
> > > > load_environment(conf.global_conf, conf.local_conf)
> > > > File "/usr/lib/ckan/prd/src/ckan/ckan/config/environment.py", line
> 344,
> > > > in load_environment
> > > > engine = sqlalchemy.engine_from_config(config, 'sqlalchemy.',
> > > **extras)
> > > > File
> > > >
> > >
> "/usr/lib/ckan/prd/lib/python2.6/site-packages/sqlalchemy/engine/__init__.py",
> > > > line 356, in engine_from_config
> > > > return create_engine(url, **opts)
> > > > File
> > > >
> > >
> "/usr/lib/ckan/prd/lib/python2.6/site-packages/sqlalchemy/engine/__init__.py",
> > > > line 338, in create_engine
> > > > return strategy.create(*args, **kwargs)
> > > > File
> > > >
> > >
> "/usr/lib/ckan/prd/lib/python2.6/site-packages/sqlalchemy/engine/strategies.py",
> > > > line 48, in create
> > > > u = url.make_url(name_or_url)
> > > > File
> > > >
> "/usr/lib/ckan/prd/lib/python2.6/site-packages/sqlalchemy/engine/url.py",
> > > > line 178, in make_url
> > > > return _parse_rfc1738_args(name_or_url)
> > > > File
> > > >
> "/usr/lib/ckan/prd/lib/python2.6/site-packages/sqlalchemy/engine/url.py",
> > > > line 219, in _parse_rfc1738_args
> > > > "Could not parse rfc1738 URL from string '%s'" % name)
> > > > sqlalchemy.exc.ArgumentError: Could not parse rfc1738 URL from
> string*
> > > > 'ckan_prd'*
> > > >
> > > > *OK we get an error lets check for things with "ckan_prd" in the
> config
> > > > file*
> > > > (prd)[ckan at ckan-app ckan]$ grep -i ckan_prd
> > > /etc/ckan/${NODE}/${NODE}.ini
> > > > sqlalchemy.url = postgresql://
> > > > ckan_prd:O1eakfeo at ckan-db-prd.green.ersa.edu.au/ckan_prd
> > > > #sqlalchemy.url = postgresql://
> > > > ckan_prd:O1eakfeo at ckan-db-prd.green.ersa.edu.au/ckan_prd
> > > > #sqlalchemy.url = postgresql://
> > > > ckan_prd:O1eakfeo at ckan-db-prd.green.ersa.edu.au/ckan_prd
> > > > ckan.datastore.write_url = postgresql://
> > > > ckan_prd:O1eakfeo at ckan-db-prd.green.ersa.edu.au/datastore_prd
> > > >
> > > > *OK so cutting and pasting the database name, username and password
> from
> > > > the above grep output lets try to connect with psql*
> > > > (prd)[ckan at ckan-app ckan]$ psql -h ckan-db-prd.green.ersa.edu.au
> > > ckan_prd
> > > > ckan_prd
> > > > Password for user ckan_prd:
> > > > psql (9.0.17)
> > > > Type "help" for help.
> > > >
> > > > ckan_prd=> \q
> > > > (prd)[ckan at ckan-app ckan]$ psql -h ckan-db-prd.green.ersa.edu.au
> > > > datastore_prd ckan_prd
> > > > Password for user ckan_prd:
> > > > psql (9.0.17)
> > > > Type "help" for help.
> > > > datastore_prd=> \q
> > > >
> > > > *The last database string is the datastore.read_url - it does not
> include
> > > > "ckan_prd" but lets test it anyway:*
> > > > (prd)[ckan at ckan-app ckan]$ grep read_url
> /etc/ckan/${NODE}/${NODE}.ini
> > > > ckan.datastore.read_url = postgresql://
> > > > datastore_prd:P5ngoueU at ckan-db-prd.green.ersa.edu.au/datastore_prd
> > > >
> > > > (prd)[ckan at ckan-app ckan]$ psql -h ckan-db-prd.green.ersa.edu.au
> > > > datastore_prd datastore_prd
> > > > Password for user datastore_prd:
> > > > psql (9.0.17)
> > > > Type "help" for help.
> > > > datastore_prd=> \q
> > > > (prd)[ckan at ckan-app ckan]$
> > > > *Datasotre read URL seems to be OK*
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Jason Tan
> > > > Data Infrastructure Specialist, eResearch SA
> > > >
> > > > phone: +61 8 8313 2184
> > > > Mob: +61 407 268 270
> > > > email: jason.tan at ersa.edu.au
> > > > web: http://www.ersa.edu.au
> > >
> > > > _______________________________________________
> > > > ckan-dev mailing list
> > > > ckan-dev at lists.okfn.org
> > > > https://lists.okfn.org/mailman/listinfo/ckan-dev
> > > > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
> > >
> > >
> > > --
> > > Nigel Babu
> > > Developer, Open Knowledge
> > > _______________________________________________
> > > ckan-dev mailing list
> > > ckan-dev at lists.okfn.org
> > > https://lists.okfn.org/mailman/listinfo/ckan-dev
> > > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
> > >
> >
> >
> >
> > --
> > Jason Tan
> > Data Infrastructure Specialist, eResearch SA
> >
> > phone: +61 8 8313 2184
> > Mob: +61 407 268 270
> > email: jason.tan at ersa.edu.au
> > web: http://www.ersa.edu.au
>
> > _______________________________________________
> > ckan-dev mailing list
> > ckan-dev at lists.okfn.org
> > https://lists.okfn.org/mailman/listinfo/ckan-dev
> > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>
>
> --
> Nigel Babu
> Developer, Open Knowledge
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>
--
Jason Tan
Data Infrastructure Specialist, eResearch SA
phone: +61 8 8313 2184
Mob: +61 407 268 270
email: jason.tan at ersa.edu.au
web: http://www.ersa.edu.au
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20140619/4d6510d2/attachment-0003.html>
More information about the ckan-dev
mailing list