[ckan-dev] problems with paster db init - can't parse URL

Nigel Babu nigel.babu at okfn.org
Tue Jun 24 03:37:10 UTC 2014 

Wow, nice debugging!

I can see how that one's nasty to find out :-)

Nigel Babu
Developer, Open Knowledge


On 19 June 2014 07:00, Jason Tan <jason.tan at ersa.edu.au> wrote:

>
> Thanks for your reply Nigel,
>
>
> It's taken me a few days of expended time but I finally worked this
> problem out.
> The hint was in the stack trace - of course.
>
> *The problem was I had "CKAN_DB=ckan_prd" set as an env var.*
>
> It took a while to work out - I had been to the point of removing all of
> the instances of postgres connection strings and "ckan_prd" from the config
> file and was still getting errors when I ran the db init
> So I wondered if I or someone else might have inadvertently edited some of
> the code, e.g. edited a python file without realizing.
> So I installed from src and tried again. Same error.
>
> At that point I was sort of stuffed.
> It's not in the config, it's not in the code. Where can it be?
> Then after staring at the screen for 10 minutes or so  I read the
> stacktrace again and saw:
>
>
>   File "/usr/lib/ckan/prd/src/ckan/ckan/config/*environment*.py", line
> 344, in load_environment
>     engine = sqlalchemy.engine_from_config(config, 'sqlalchemy.', **extras)
>
> Hmmm.. are you looking at the shell environment Mr CKAN?
>
> Well, yes as it turns out.
> Problem solved.
>
> Regards
> Jason Tan
>
>
>
> On 13 June 2014 18:10, Nigel Babu <nigel.babu at okfn.org> wrote:
>
>> Since we know it's one particular line that it can't parse, try removing
>> disabling the datastore and trying each of those URLs for sqlalchemy.url
>> field
>> and see which one throws and error.
>>
>> This is at the sqlalchemy error and the only string it gets is 'ckan_prd',
>> instead of the full URL.
>>
>> On Fri, Jun 13, 2014 at 02:44:58PM +0930, Jason Tan wrote:
>> > Thanks for taking a look NIgel.
>> >
>> > The passwords use the exact same characters as in the string. I've just
>> > messed the order of the chars around in the email.
>> > I was conscious that it might be a special character so I was careful to
>> > include all the actual characters - though as you say - no character
>> looks
>> > very special in those strings.
>> > I was also concerned about unprintable characters so I've checked in vi
>> > with "set list" set , with no unprintable shown, and I've also cut and
>> > paste the passwords. usernames and hostnames in the email into psql on
>> the
>> > command line, before I moved the order around and it works fine.
>> > My very initial thought were "oh I've screwed up pg_hba.conf, what's
>> new?".
>> > But that appears not be the case since psql from the app server works
>> where
>> > paster run is OK.
>> >
>> > I'll keep plugging away and let you all know when I find a solution/work
>> > around or cause.
>> > Mean while if anyone has any suggestions, I'm all ears.
>> >
>> > Thanks again Nigel.
>> >
>> > Regards
>> > Jason
>> >
>> >
>> >
>> > On 13 June 2014 13:31, Nigel Babu <nigel.babu at okfn.org> wrote:
>> >
>> > > Hi Jason,
>> > >
>> > > I can't find anything obviously wrong. But from the error, it's fairly
>> > > clear,
>> > > it's a typo of some sort. Do you happen to have '@' or ':' characters
>> in
>> > > your
>> > > password?
>> > >
>> > > On Thu, Jun 12, 2014 at 09:55:45AM +0930, Jason Tan wrote:
>> > > > Hi Folks,
>> > > >
>> > > > Sorry to be asking here, this feels like a problem I should be able
>> to
>> > > > solve myself, but I've been trying to work it out for most of two
>> days
>> > > with
>> > > > no luck.
>> > > >
>> > > > I am installing CKAN 2.1.1 and am having a problem with the
>> > > > paster db init command.
>> > > >
>> > > > The complete error is below, but the last line of the error is:
>> > > >     "Could not parse rfc1738 URL from string '%s'" % name)
>> > > > sqlalchemy.exc.ArgumentError: Could not parse rfc1738 URL from
>> string
>> > > > * 'ckan_prd'*
>> > > >
>> > > >
>> > > > The problem appears to be with one of the database connection
>> strings.
>> > > > If I copy and paste the sername, passwd and host from the database
>> > > > connection strings to psql on the app server the strings seem to
>> work OK
>> > > >
>> > > > Here are the database strings:
>> > > > (prd)[ckan at ckan-app ckan]$ echo $NODE
>> > > > prd
>> > > > (prd)[ckan at ckan-app ckan]$ grep -i postgresql
>> > >  /etc/ckan/${NODE}/${NODE}.ini
>> > > >
>> > > > sqlalchemy.url = postgresql://
>> > > > ckan_prd:O1eakfeo at ckan-db-prd.green.ersa.edu.au/ckan_prd
>> > > >
>> > > > ckan.datastore.write_url = postgresql://
>> > > > ckan_prd:O1eakfeo at ckan-db-prd.green.ersa.edu.au/datastore_prd
>> > > >
>> > > > ckan.datastore.read_url = postgresql://
>> > > > datastore_prd:P5ngoueU at ckan-db-prd.green.ersa.edu.au/datastore_prd
>> > > >
>> > > > I have this running in dev and apart from the fact that I am using
>> > > > "default" instead of prd in database and usernames and I have
>> different
>> > > > passwords, the database strings look the same to me.
>> > > >
>> > > > I've found this link
>> > > > https://code.google.com/p/sqlalchemy-migrate/issues/detail?id=54
>> which
>> > > > suggests there was a bug under python 2.5 but I am using 2.6 and
>> the same
>> > > > version in dev, which appears to work OK and I've got the most
>> recent
>> > > > pyhton package available for the distro (centos 6.5).
>> > > >
>> > > > Below is a more complete output of the steps I am taking with
>> commentary
>> > > > inbold.
>> > > >
>> > > > Thanks and Regards
>> > > > Jason
>> > > >
>> > > >
>> > > >
>> > > > *I have an env var NODE which I use in the command lines. In this
>> > > instance
>> > > > it evaluates to prd*
>> > > > (prd)[ckan at ckan-app ckan]$ echo $NODE
>> > > > prd
>> > > >
>> > > > *I think I am in the correct dir*
>> > > > (prd)[ckan at ckan-app ckan]$ pwd
>> > > > /usr/lib/ckan/prd/src/ckan
>> > > >
>> > > > *lets look for ckan_ - which is a string the exception stack trace
>> refers
>> > > > to:*
>> > > > (prd)[ckan at ckan-app ckan]$ grep -i ckan_ /etc/ckan/prd/prd.ini
>> > > > sqlalchemy.url = postgresql://
>> > > > ckan_prd:O1eakfeo at ckan-db-prd.green.ersa.edu.au/ckan_prd
>> > > > #sqlalchemy.url = postgresql://
>> > > > ckan_prd:O1eakfeo at ckan-db-prd.green.ersa.edu.au/ckan_prd
>> > > > #sqlalchemy.url = postgresql://
>> > > > ckan_prd:O1eakfeo at ckan-db-prd.green.ersa.edu.au/ckan_prd
>> > > > ckan.datastore.write_url = postgresql://
>> > > > ckan_prd:O1eakfeo at ckan-db-prd.green.ersa.edu.au/datastore_prd
>> > > >
>> > > > *check that env var NODE is set, then run the command*
>> > > > (prd)[ckan at ckan-app ckan]$ echo $NODE
>> > > > prd
>> > > > (prd)[ckan at ckan-app ckan]$ paster db init -c
>> > > /etc/ckan/${NODE}/${NODE}.ini
>> > > > Traceback (most recent call last):
>> > > >   File "/usr/lib/ckan/prd/bin/paster", line 9, in <module>
>> > > >     load_entry_point('PasteScript==1.7.5', 'console_scripts',
>> 'paster')()
>> > > >   File
>> > > >
>> "/usr/lib/ckan/prd/lib/python2.6/site-packages/paste/script/command.py",
>> > > > line 104, in run
>> > > >     invoke(command, command_name, options, args[1:])
>> > > >   File
>> > > >
>> "/usr/lib/ckan/prd/lib/python2.6/site-packages/paste/script/command.py",
>> > > > line 143, in invoke
>> > > >     exit_code = runner.run(args)
>> > > >   File
>> > > >
>> "/usr/lib/ckan/prd/lib/python2.6/site-packages/paste/script/command.py",
>> > > > line 238, in run
>> > > >     result = self.command()
>> > > >   File "/usr/lib/ckan/prd/src/ckan/ckan/lib/cli.py", line 129, in
>> command
>> > > >     self._load_config()
>> > > >   File "/usr/lib/ckan/prd/src/ckan/ckan/lib/cli.py", line 91, in
>> > > > _load_config
>> > > >     load_environment(conf.global_conf, conf.local_conf)
>> > > >   File "/usr/lib/ckan/prd/src/ckan/ckan/config/environment.py",
>> line 344,
>> > > > in load_environment
>> > > >     engine = sqlalchemy.engine_from_config(config, 'sqlalchemy.',
>> > > **extras)
>> > > >   File
>> > > >
>> > >
>> "/usr/lib/ckan/prd/lib/python2.6/site-packages/sqlalchemy/engine/__init__.py",
>> > > > line 356, in engine_from_config
>> > > >     return create_engine(url, **opts)
>> > > >   File
>> > > >
>> > >
>> "/usr/lib/ckan/prd/lib/python2.6/site-packages/sqlalchemy/engine/__init__.py",
>> > > > line 338, in create_engine
>> > > >     return strategy.create(*args, **kwargs)
>> > > >   File
>> > > >
>> > >
>> "/usr/lib/ckan/prd/lib/python2.6/site-packages/sqlalchemy/engine/strategies.py",
>> > > > line 48, in create
>> > > >     u = url.make_url(name_or_url)
>> > > >   File
>> > > >
>> "/usr/lib/ckan/prd/lib/python2.6/site-packages/sqlalchemy/engine/url.py",
>> > > > line 178, in make_url
>> > > >     return _parse_rfc1738_args(name_or_url)
>> > > >   File
>> > > >
>> "/usr/lib/ckan/prd/lib/python2.6/site-packages/sqlalchemy/engine/url.py",
>> > > > line 219, in _parse_rfc1738_args
>> > > >     "Could not parse rfc1738 URL from string '%s'" % name)
>> > > > sqlalchemy.exc.ArgumentError: Could not parse rfc1738 URL from
>> string
>> > > > 'ckan_prd'
>> > > > (prd)[ckan at ckan-app ckan]$ paster db init -c /etc/ckan/prd/prd.ini
>> > > > Traceback (most recent call last):
>> > > >   File "/usr/lib/ckan/prd/bin/paster", line 9, in <module>
>> > > >     load_entry_point('PasteScript==1.7.5', 'console_scripts',
>> 'paster')()
>> > > >   File
>> > > >
>> "/usr/lib/ckan/prd/lib/python2.6/site-packages/paste/script/command.py",
>> > > > line 104, in run
>> > > >     invoke(command, command_name, options, args[1:])
>> > > >   File
>> > > >
>> "/usr/lib/ckan/prd/lib/python2.6/site-packages/paste/script/command.py",
>> > > > line 143, in invoke
>> > > >     exit_code = runner.run(args)
>> > > >   File
>> > > >
>> "/usr/lib/ckan/prd/lib/python2.6/site-packages/paste/script/command.py",
>> > > > line 238, in run
>> > > >     result = self.command()
>> > > >   File "/usr/lib/ckan/prd/src/ckan/ckan/lib/cli.py", line 129, in
>> command
>> > > >     self._load_config()
>> > > >   File "/usr/lib/ckan/prd/src/ckan/ckan/lib/cli.py", line 91, in
>> > > > _load_config
>> > > >     load_environment(conf.global_conf, conf.local_conf)
>> > > >   File "/usr/lib/ckan/prd/src/ckan/ckan/config/environment.py",
>> line 344,
>> > > > in load_environment
>> > > >     engine = sqlalchemy.engine_from_config(config, 'sqlalchemy.',
>> > > **extras)
>> > > >   File
>> > > >
>> > >
>> "/usr/lib/ckan/prd/lib/python2.6/site-packages/sqlalchemy/engine/__init__.py",
>> > > > line 356, in engine_from_config
>> > > >     return create_engine(url, **opts)
>> > > >   File
>> > > >
>> > >
>> "/usr/lib/ckan/prd/lib/python2.6/site-packages/sqlalchemy/engine/__init__.py",
>> > > > line 338, in create_engine
>> > > >     return strategy.create(*args, **kwargs)
>> > > >   File
>> > > >
>> > >
>> "/usr/lib/ckan/prd/lib/python2.6/site-packages/sqlalchemy/engine/strategies.py",
>> > > > line 48, in create
>> > > >     u = url.make_url(name_or_url)
>> > > >   File
>> > > >
>> "/usr/lib/ckan/prd/lib/python2.6/site-packages/sqlalchemy/engine/url.py",
>> > > > line 178, in make_url
>> > > >     return _parse_rfc1738_args(name_or_url)
>> > > >   File
>> > > >
>> "/usr/lib/ckan/prd/lib/python2.6/site-packages/sqlalchemy/engine/url.py",
>> > > > line 219, in _parse_rfc1738_args
>> > > >     "Could not parse rfc1738 URL from string '%s'" % name)
>> > > > sqlalchemy.exc.ArgumentError: Could not parse rfc1738 URL from
>> string*
>> > > > 'ckan_prd'*
>> > > >
>> > > > *OK we get an error lets check for things with "ckan_prd" in the
>> config
>> > > > file*
>> > > > (prd)[ckan at ckan-app ckan]$ grep -i ckan_prd
>> > >  /etc/ckan/${NODE}/${NODE}.ini
>> > > > sqlalchemy.url = postgresql://
>> > > > ckan_prd:O1eakfeo at ckan-db-prd.green.ersa.edu.au/ckan_prd
>> > > > #sqlalchemy.url = postgresql://
>> > > > ckan_prd:O1eakfeo at ckan-db-prd.green.ersa.edu.au/ckan_prd
>> > > > #sqlalchemy.url = postgresql://
>> > > > ckan_prd:O1eakfeo at ckan-db-prd.green.ersa.edu.au/ckan_prd
>> > > > ckan.datastore.write_url = postgresql://
>> > > > ckan_prd:O1eakfeo at ckan-db-prd.green.ersa.edu.au/datastore_prd
>> > > >
>> > > > *OK so cutting and pasting the database name, username and password
>> from
>> > > > the above grep output lets try to connect with psql*
>> > > > (prd)[ckan at ckan-app ckan]$ psql -h ckan-db-prd.green.ersa.edu.au
>> > > ckan_prd
>> > > > ckan_prd
>> > > > Password for user ckan_prd:
>> > > > psql (9.0.17)
>> > > > Type "help" for help.
>> > > >
>> > > > ckan_prd=> \q
>> > > > (prd)[ckan at ckan-app ckan]$ psql -h ckan-db-prd.green.ersa.edu.au
>> > > > datastore_prd ckan_prd
>> > > > Password for user ckan_prd:
>> > > > psql (9.0.17)
>> > > > Type "help" for help.
>> > > > datastore_prd=> \q
>> > > >
>> > > > *The last database string is the datastore.read_url - it does not
>> include
>> > > > "ckan_prd" but lets test it anyway:*
>> > > > (prd)[ckan at ckan-app ckan]$ grep read_url
>>  /etc/ckan/${NODE}/${NODE}.ini
>> > > > ckan.datastore.read_url = postgresql://
>> > > > datastore_prd:P5ngoueU at ckan-db-prd.green.ersa.edu.au/datastore_prd
>> > > >
>> > > > (prd)[ckan at ckan-app ckan]$ psql -h ckan-db-prd.green.ersa.edu.au
>> > > > datastore_prd datastore_prd
>> > > > Password for user datastore_prd:
>> > > > psql (9.0.17)
>> > > > Type "help" for help.
>> > > > datastore_prd=> \q
>> > > > (prd)[ckan at ckan-app ckan]$
>> > > > *Datasotre read URL seems to be OK*
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >  --
>> > > > Jason Tan
>> > > > Data Infrastructure Specialist, eResearch SA
>> > > >
>> > > > phone: +61 8 8313 2184
>> > > > Mob: +61 407 268 270
>> > > > email: jason.tan at ersa.edu.au
>> > > > web: http://www.ersa.edu.au
>> > >
>> > > > _______________________________________________
>> > > > ckan-dev mailing list
>> > > > ckan-dev at lists.okfn.org
>> > > > https://lists.okfn.org/mailman/listinfo/ckan-dev
>> > > > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>> > >
>> > >
>> > > --
>> > > Nigel Babu
>> > > Developer, Open Knowledge
>> > > _______________________________________________
>> > > ckan-dev mailing list
>> > > ckan-dev at lists.okfn.org
>> > > https://lists.okfn.org/mailman/listinfo/ckan-dev
>> > > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>> > >
>> >
>> >
>> >
>> > --
>> > Jason Tan
>> > Data Infrastructure Specialist, eResearch SA
>> >
>> > phone: +61 8 8313 2184
>> > Mob: +61 407 268 270
>> > email: jason.tan at ersa.edu.au
>> > web: http://www.ersa.edu.au
>>
>> > _______________________________________________
>> > ckan-dev mailing list
>> > ckan-dev at lists.okfn.org
>> > https://lists.okfn.org/mailman/listinfo/ckan-dev
>> > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>
>>
>> --
>> Nigel Babu
>> Developer, Open Knowledge
>> _______________________________________________
>> ckan-dev mailing list
>> ckan-dev at lists.okfn.org
>> https://lists.okfn.org/mailman/listinfo/ckan-dev
>> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>>
>
>
>
> --
> Jason Tan
> Data Infrastructure Specialist, eResearch SA
>
> phone: +61 8 8313 2184
> Mob: +61 407 268 270
> email: jason.tan at ersa.edu.au
> web: http://www.ersa.edu.au
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20140624/7e1de3b7/attachment-0003.html>

More information about the ckan-dev mailing list