[ckan-dev] problems with paster db init - can't parse URL

Nigel Babu nigel.babu at okfn.org
Fri Jun 13 08:40:05 UTC 2014 

Since we know it's one particular line that it can't parse, try removing
disabling the datastore and trying each of those URLs for sqlalchemy.url field
and see which one throws and error.

This is at the sqlalchemy error and the only string it gets is 'ckan_prd',
instead of the full URL.

On Fri, Jun 13, 2014 at 02:44:58PM +0930, Jason Tan wrote:
> Thanks for taking a look NIgel.
>
> The passwords use the exact same characters as in the string. I've just
> messed the order of the chars around in the email.
> I was conscious that it might be a special character so I was careful to
> include all the actual characters - though as you say - no character looks
> very special in those strings.
> I was also concerned about unprintable characters so I've checked in vi
> with "set list" set , with no unprintable shown, and I've also cut and
> paste the passwords. usernames and hostnames in the email into psql on the
> command line, before I moved the order around and it works fine.
> My very initial thought were "oh I've screwed up pg_hba.conf, what's new?".
> But that appears not be the case since psql from the app server works where
> paster run is OK.
>
> I'll keep plugging away and let you all know when I find a solution/work
> around or cause.
> Mean while if anyone has any suggestions, I'm all ears.
>
> Thanks again Nigel.
>
> Regards
> Jason
>
>
>
> On 13 June 2014 13:31, Nigel Babu <nigel.babu at okfn.org> wrote:
>
> > Hi Jason,
> >
> > I can't find anything obviously wrong. But from the error, it's fairly
> > clear,
> > it's a typo of some sort. Do you happen to have '@' or ':' characters in
> > your
> > password?
> >
> > On Thu, Jun 12, 2014 at 09:55:45AM +0930, Jason Tan wrote:
> > > Hi Folks,
> > >
> > > Sorry to be asking here, this feels like a problem I should be able to
> > > solve myself, but I've been trying to work it out for most of two days
> > with
> > > no luck.
> > >
> > > I am installing CKAN 2.1.1 and am having a problem with the
> > > paster db init command.
> > >
> > > The complete error is below, but the last line of the error is:
> > >     "Could not parse rfc1738 URL from string '%s'" % name)
> > > sqlalchemy.exc.ArgumentError: Could not parse rfc1738 URL from string
> > > * 'ckan_prd'*
> > >
> > >
> > > The problem appears to be with one of the database connection strings.
> > > If I copy and paste the sername, passwd and host from the database
> > > connection strings to psql on the app server the strings seem to work OK
> > >
> > > Here are the database strings:
> > > (prd)[ckan at ckan-app ckan]$ echo $NODE
> > > prd
> > > (prd)[ckan at ckan-app ckan]$ grep -i postgresql
> >  /etc/ckan/${NODE}/${NODE}.ini
> > >
> > > sqlalchemy.url = postgresql://
> > > ckan_prd:O1eakfeo at ckan-db-prd.green.ersa.edu.au/ckan_prd
> > >
> > > ckan.datastore.write_url = postgresql://
> > > ckan_prd:O1eakfeo at ckan-db-prd.green.ersa.edu.au/datastore_prd
> > >
> > > ckan.datastore.read_url = postgresql://
> > > datastore_prd:P5ngoueU at ckan-db-prd.green.ersa.edu.au/datastore_prd
> > >
> > > I have this running in dev and apart from the fact that I am using
> > > "default" instead of prd in database and usernames and I have different
> > > passwords, the database strings look the same to me.
> > >
> > > I've found this link
> > > https://code.google.com/p/sqlalchemy-migrate/issues/detail?id=54 which
> > > suggests there was a bug under python 2.5 but I am using 2.6 and the same
> > > version in dev, which appears to work OK and I've got the most recent
> > > pyhton package available for the distro (centos 6.5).
> > >
> > > Below is a more complete output of the steps I am taking with commentary
> > > inbold.
> > >
> > > Thanks and Regards
> > > Jason
> > >
> > >
> > >
> > > *I have an env var NODE which I use in the command lines. In this
> > instance
> > > it evaluates to prd*
> > > (prd)[ckan at ckan-app ckan]$ echo $NODE
> > > prd
> > >
> > > *I think I am in the correct dir*
> > > (prd)[ckan at ckan-app ckan]$ pwd
> > > /usr/lib/ckan/prd/src/ckan
> > >
> > > *lets look for ckan_ - which is a string the exception stack trace refers
> > > to:*
> > > (prd)[ckan at ckan-app ckan]$ grep -i ckan_ /etc/ckan/prd/prd.ini
> > > sqlalchemy.url = postgresql://
> > > ckan_prd:O1eakfeo at ckan-db-prd.green.ersa.edu.au/ckan_prd
> > > #sqlalchemy.url = postgresql://
> > > ckan_prd:O1eakfeo at ckan-db-prd.green.ersa.edu.au/ckan_prd
> > > #sqlalchemy.url = postgresql://
> > > ckan_prd:O1eakfeo at ckan-db-prd.green.ersa.edu.au/ckan_prd
> > > ckan.datastore.write_url = postgresql://
> > > ckan_prd:O1eakfeo at ckan-db-prd.green.ersa.edu.au/datastore_prd
> > >
> > > *check that env var NODE is set, then run the command*
> > > (prd)[ckan at ckan-app ckan]$ echo $NODE
> > > prd
> > > (prd)[ckan at ckan-app ckan]$ paster db init -c
> > /etc/ckan/${NODE}/${NODE}.ini
> > > Traceback (most recent call last):
> > >   File "/usr/lib/ckan/prd/bin/paster", line 9, in <module>
> > >     load_entry_point('PasteScript==1.7.5', 'console_scripts', 'paster')()
> > >   File
> > > "/usr/lib/ckan/prd/lib/python2.6/site-packages/paste/script/command.py",
> > > line 104, in run
> > >     invoke(command, command_name, options, args[1:])
> > >   File
> > > "/usr/lib/ckan/prd/lib/python2.6/site-packages/paste/script/command.py",
> > > line 143, in invoke
> > >     exit_code = runner.run(args)
> > >   File
> > > "/usr/lib/ckan/prd/lib/python2.6/site-packages/paste/script/command.py",
> > > line 238, in run
> > >     result = self.command()
> > >   File "/usr/lib/ckan/prd/src/ckan/ckan/lib/cli.py", line 129, in command
> > >     self._load_config()
> > >   File "/usr/lib/ckan/prd/src/ckan/ckan/lib/cli.py", line 91, in
> > > _load_config
> > >     load_environment(conf.global_conf, conf.local_conf)
> > >   File "/usr/lib/ckan/prd/src/ckan/ckan/config/environment.py", line 344,
> > > in load_environment
> > >     engine = sqlalchemy.engine_from_config(config, 'sqlalchemy.',
> > **extras)
> > >   File
> > >
> > "/usr/lib/ckan/prd/lib/python2.6/site-packages/sqlalchemy/engine/__init__.py",
> > > line 356, in engine_from_config
> > >     return create_engine(url, **opts)
> > >   File
> > >
> > "/usr/lib/ckan/prd/lib/python2.6/site-packages/sqlalchemy/engine/__init__.py",
> > > line 338, in create_engine
> > >     return strategy.create(*args, **kwargs)
> > >   File
> > >
> > "/usr/lib/ckan/prd/lib/python2.6/site-packages/sqlalchemy/engine/strategies.py",
> > > line 48, in create
> > >     u = url.make_url(name_or_url)
> > >   File
> > > "/usr/lib/ckan/prd/lib/python2.6/site-packages/sqlalchemy/engine/url.py",
> > > line 178, in make_url
> > >     return _parse_rfc1738_args(name_or_url)
> > >   File
> > > "/usr/lib/ckan/prd/lib/python2.6/site-packages/sqlalchemy/engine/url.py",
> > > line 219, in _parse_rfc1738_args
> > >     "Could not parse rfc1738 URL from string '%s'" % name)
> > > sqlalchemy.exc.ArgumentError: Could not parse rfc1738 URL from string
> > > 'ckan_prd'
> > > (prd)[ckan at ckan-app ckan]$ paster db init -c /etc/ckan/prd/prd.ini
> > > Traceback (most recent call last):
> > >   File "/usr/lib/ckan/prd/bin/paster", line 9, in <module>
> > >     load_entry_point('PasteScript==1.7.5', 'console_scripts', 'paster')()
> > >   File
> > > "/usr/lib/ckan/prd/lib/python2.6/site-packages/paste/script/command.py",
> > > line 104, in run
> > >     invoke(command, command_name, options, args[1:])
> > >   File
> > > "/usr/lib/ckan/prd/lib/python2.6/site-packages/paste/script/command.py",
> > > line 143, in invoke
> > >     exit_code = runner.run(args)
> > >   File
> > > "/usr/lib/ckan/prd/lib/python2.6/site-packages/paste/script/command.py",
> > > line 238, in run
> > >     result = self.command()
> > >   File "/usr/lib/ckan/prd/src/ckan/ckan/lib/cli.py", line 129, in command
> > >     self._load_config()
> > >   File "/usr/lib/ckan/prd/src/ckan/ckan/lib/cli.py", line 91, in
> > > _load_config
> > >     load_environment(conf.global_conf, conf.local_conf)
> > >   File "/usr/lib/ckan/prd/src/ckan/ckan/config/environment.py", line 344,
> > > in load_environment
> > >     engine = sqlalchemy.engine_from_config(config, 'sqlalchemy.',
> > **extras)
> > >   File
> > >
> > "/usr/lib/ckan/prd/lib/python2.6/site-packages/sqlalchemy/engine/__init__.py",
> > > line 356, in engine_from_config
> > >     return create_engine(url, **opts)
> > >   File
> > >
> > "/usr/lib/ckan/prd/lib/python2.6/site-packages/sqlalchemy/engine/__init__.py",
> > > line 338, in create_engine
> > >     return strategy.create(*args, **kwargs)
> > >   File
> > >
> > "/usr/lib/ckan/prd/lib/python2.6/site-packages/sqlalchemy/engine/strategies.py",
> > > line 48, in create
> > >     u = url.make_url(name_or_url)
> > >   File
> > > "/usr/lib/ckan/prd/lib/python2.6/site-packages/sqlalchemy/engine/url.py",
> > > line 178, in make_url
> > >     return _parse_rfc1738_args(name_or_url)
> > >   File
> > > "/usr/lib/ckan/prd/lib/python2.6/site-packages/sqlalchemy/engine/url.py",
> > > line 219, in _parse_rfc1738_args
> > >     "Could not parse rfc1738 URL from string '%s'" % name)
> > > sqlalchemy.exc.ArgumentError: Could not parse rfc1738 URL from string*
> > > 'ckan_prd'*
> > >
> > > *OK we get an error lets check for things with "ckan_prd" in the config
> > > file*
> > > (prd)[ckan at ckan-app ckan]$ grep -i ckan_prd
> >  /etc/ckan/${NODE}/${NODE}.ini
> > > sqlalchemy.url = postgresql://
> > > ckan_prd:O1eakfeo at ckan-db-prd.green.ersa.edu.au/ckan_prd
> > > #sqlalchemy.url = postgresql://
> > > ckan_prd:O1eakfeo at ckan-db-prd.green.ersa.edu.au/ckan_prd
> > > #sqlalchemy.url = postgresql://
> > > ckan_prd:O1eakfeo at ckan-db-prd.green.ersa.edu.au/ckan_prd
> > > ckan.datastore.write_url = postgresql://
> > > ckan_prd:O1eakfeo at ckan-db-prd.green.ersa.edu.au/datastore_prd
> > >
> > > *OK so cutting and pasting the database name, username and password from
> > > the above grep output lets try to connect with psql*
> > > (prd)[ckan at ckan-app ckan]$ psql -h ckan-db-prd.green.ersa.edu.au
> > ckan_prd
> > > ckan_prd
> > > Password for user ckan_prd:
> > > psql (9.0.17)
> > > Type "help" for help.
> > >
> > > ckan_prd=> \q
> > > (prd)[ckan at ckan-app ckan]$ psql -h ckan-db-prd.green.ersa.edu.au
> > > datastore_prd ckan_prd
> > > Password for user ckan_prd:
> > > psql (9.0.17)
> > > Type "help" for help.
> > > datastore_prd=> \q
> > >
> > > *The last database string is the datastore.read_url - it does not include
> > > "ckan_prd" but lets test it anyway:*
> > > (prd)[ckan at ckan-app ckan]$ grep read_url  /etc/ckan/${NODE}/${NODE}.ini
> > > ckan.datastore.read_url = postgresql://
> > > datastore_prd:P5ngoueU at ckan-db-prd.green.ersa.edu.au/datastore_prd
> > >
> > > (prd)[ckan at ckan-app ckan]$ psql -h ckan-db-prd.green.ersa.edu.au
> > > datastore_prd datastore_prd
> > > Password for user datastore_prd:
> > > psql (9.0.17)
> > > Type "help" for help.
> > > datastore_prd=> \q
> > > (prd)[ckan at ckan-app ckan]$
> > > *Datasotre read URL seems to be OK*
> > >
> > >
> > >
> > >
> > >  --
> > > Jason Tan
> > > Data Infrastructure Specialist, eResearch SA
> > >
> > > phone: +61 8 8313 2184
> > > Mob: +61 407 268 270
> > > email: jason.tan at ersa.edu.au
> > > web: http://www.ersa.edu.au
> >
> > > _______________________________________________
> > > ckan-dev mailing list
> > > ckan-dev at lists.okfn.org
> > > https://lists.okfn.org/mailman/listinfo/ckan-dev
> > > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
> >
> >
> > --
> > Nigel Babu
> > Developer, Open Knowledge
> > _______________________________________________
> > ckan-dev mailing list
> > ckan-dev at lists.okfn.org
> > https://lists.okfn.org/mailman/listinfo/ckan-dev
> > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
> >
>
>
>
> --
> Jason Tan
> Data Infrastructure Specialist, eResearch SA
>
> phone: +61 8 8313 2184
> Mob: +61 407 268 270
> email: jason.tan at ersa.edu.au
> web: http://www.ersa.edu.au

> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev


--
Nigel Babu
Developer, Open Knowledge

More information about the ckan-dev mailing list