[ckan-dev] ckan-dev Digest, Vol 62, Issue 1
Bart Praats
bartpraats at hotmail.com
Tue Dec 1 12:39:19 UTC 2015
Hello Ross,
Thank you for your quick response.
This indeed explains my pdf problem!So for the other problem, I asked my employer for some more server information.
Turns out the server is an AWS Amazon server and has a private and public IP, but the public IP is only accessible from our company network. This will probably be the cause of my problems (and the reason the server can not ping it's own public IP). I will need some more researching before I fully understand what this means for my setup.
Kind regards,
Bart Praats
> From: ckan-dev-request at lists.okfn.org
> Subject: ckan-dev Digest, Vol 62, Issue 1
> To: ckan-dev at lists.okfn.org
> Date: Tue, 1 Dec 2015 10:18:19 +0000
>
> Send ckan-dev mailing list submissions to
> ckan-dev at lists.okfn.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> or, via email, send a message with subject or body 'help' to
> ckan-dev-request at lists.okfn.org
>
> You can reach the person managing the list at
> ckan-dev-owner at lists.okfn.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of ckan-dev digest..."
>
>
> Today's Topics:
>
> 1. Stackoverflow Issue (Bart Praats)
> 2. Re: Stackoverflow Issue (Ross Jones)
> 3. Re: Stackoverflow Issue (Matthew Fullerton)
> 4. Re: Stackoverflow Issue (Pedro G Silva)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Tue, 1 Dec 2015 08:52:36 +0100
> From: Bart Praats <bartpraats at hotmail.com>
> To: "ckan-dev at lists.okfn.org" <ckan-dev at lists.okfn.org>
> Subject: [ckan-dev] Stackoverflow Issue
> Message-ID: <DUB130-W38ADB27FB22435D81F6CC4CF0F0 at phx.gbl>
> Content-Type: text/plain; charset="iso-8859-1"
>
>
>
>
>
>
>
> Dear Ckan community,
> I have posted an issue on Stackoverflow which is still not solved: http://stackoverflow.com/questions/33775693/ckan-datapusher-pdf-gives-indexerror-pop-from-empty-list-when-uploading-to-da
> For the "Pop from empty list" I created an issue on the Ckan Github page. But my Datapusher problem is something I am doing wrong, I think, and I can't figure out what.
> I narrowed it down to the code line where all goes wrong:
> https://github.com/ckan/datapusher/blob/master/datapusher/jobs.py#L290
>
> Here my Datapusher gets the url from the Json object. However, my Datapusher and my Ckan instance run on the same server, e.g. they have the same IP address. Only the port differs.
> I don't know if this should give a problem or not. But it should be possible to host both Datapusher and Ckan on the same host machine I suppose? Currently I don't have a domain name yet so I am using my IP to browse to the site. (example: 163.883.716.17) In this example, my Datapusher is reachable on 163.883.716.17:5820. This works. I can reach my Ckan instance and I can reach my Datapusher with a successful Json object returned.
> Now what causes the error is that when I try to reach the public address mentioned before from the command line of my server, I get a timeout error.
> This is also what happens when the before mentioned code is executed. So my question is:
> How can I solve this timeout error? Or what could I have done wrong in ly configurations?
> Thank you for any help!
> Kind regards,
> Bart Praats
>
>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20151201/38e6f5c5/attachment-0001.html>
>
> ------------------------------
>
> Message: 2
> Date: Tue, 1 Dec 2015 08:09:20 +0000
> From: Ross Jones <ross at servercode.co.uk>
> To: CKAN Development Discussions <ckan-dev at lists.okfn.org>
> Subject: Re: [ckan-dev] Stackoverflow Issue
> Message-ID: <5CB0A97A-B4B0-40FC-A248-37B9DC2992F1 at servercode.co.uk>
> Content-Type: text/plain; charset=iso-8859-1
>
> Hi Bart,
>
> I think you have two problems here. The first is that, as far as I am aware, datapusher doesn't support PDF files, and they should not be being sent to the datapusher. The error is unfortunate, and should be fixed, but the PDF shouldn't get that far in the first place.
>
> > Now what causes the error is that when I try to reach the public address mentioned before from the command line of my server, I get a timeout error.
>
> The second seems to be a connectivity problem. I'm confused as to whether you're having problems with:
>
> - datapusher connecting to CKAN
> in which case can you confirm your ckan.ini has the correct value for ckan.site_url ?
>
> - or, accessing publicly accessible internet urls
> in which case there is something wrong with your machine/VM. If you can't curl the URL you want to fetch, there's little change that datapusher will be able to either. Can you post the actual URL?
>
> Cheers,
>
> Ross
>
>
>
> ------------------------------
>
> Message: 3
> Date: Tue, 1 Dec 2015 09:19:33 +0100
> From: Matthew Fullerton <matt.fullerton at gmail.com>
> To: CKAN Development Discussions <ckan-dev at lists.okfn.org>
> Subject: Re: [ckan-dev] Stackoverflow Issue
> Message-ID:
> <CAAm+7TaJ0TA3yY2BbRBWkVYUnhSC428Bj2uofC0MQk84fO9YYg at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> As a minor follow-up to this, is your ckan machine behind a proxy server?
> On 1 Dec 2015 9:09 a.m., "Ross Jones" <ross at servercode.co.uk> wrote:
>
> > Hi Bart,
> >
> > I think you have two problems here. The first is that, as far as I am
> > aware, datapusher doesn't support PDF files, and they should not be being
> > sent to the datapusher. The error is unfortunate, and should be fixed, but
> > the PDF shouldn't get that far in the first place.
> >
> > > Now what causes the error is that when I try to reach the public address
> > mentioned before from the command line of my server, I get a timeout error.
> >
> > The second seems to be a connectivity problem. I'm confused as to whether
> > you're having problems with:
> >
> > - datapusher connecting to CKAN
> > in which case can you confirm your ckan.ini has the correct value for
> > ckan.site_url ?
> >
> > - or, accessing publicly accessible internet urls
> > in which case there is something wrong with your machine/VM. If you can't
> > curl the URL you want to fetch, there's little change that datapusher will
> > be able to either. Can you post the actual URL?
> >
> > Cheers,
> >
> > Ross
> >
> > _______________________________________________
> > ckan-dev mailing list
> > ckan-dev at lists.okfn.org
> > https://lists.okfn.org/mailman/listinfo/ckan-dev
> > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
> >
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20151201/70294c8c/attachment-0001.html>
>
> ------------------------------
>
> Message: 4
> Date: Tue, 1 Dec 2015 10:18:16 +0000
> From: Pedro G Silva <pedro.gracio at deimos.com.pt>
> To: CKAN Development Discussions <ckan-dev at lists.okfn.org>
> Subject: Re: [ckan-dev] Stackoverflow Issue
> Message-ID:
> <CAJG0=QgxJc93hndVGkcDOnPGG4zCVBJ+wOQ4furvmvLx3Bqy8A at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> I have deployed ckan 2.4.0 and I documented everything. The only thing i
> did different was using tomcat server, since i was having similar
> datapusher issues and switching to tomcat solved (dunno why)
>
> *CKAN*
>
>
> sudo apt-get install python-dev postgresql libpq-dev python-pip
> python-virtualenv git-core solr-jetty openjdk-6-jdk
>
> sudo apt-get install python-pastescript
>
> mkdir -p ~/ckan/lib
>
> sudo ln -s ~/ckan/lib /usr/lib/ckan
>
> mkdir -p ~/ckan/etc
>
> sudo ln -s ~/ckan/etc /etc/ckan
>
>
> sudo mkdir -p /usr/lib/ckan/default
>
> sudo chown `whoami` /usr/lib/ckan/default
>
> virtualenv --no-site-packages /usr/lib/ckan/default
>
> . /usr/lib/ckan/default/bin/activate
>
>
> pip install -e 'git+https://github.com/ckan/ckan.git@ckan-2.4.1#egg=ckan'
>
> pip install -r /usr/lib/ckan/default/src/ckan/requirements.txt
>
>
> pip install -r /usr/lib/ckan/default/src/ckan/dev-requirements.txt
>
>
> deactivate
>
> . /usr/lib/ckan/default/bin/activate
>
>
> sudo -u postgres psql -l
>
>
> sudo -u postgres createuser -S -D -R -P ckan_default
>
> sudo -u postgres createdb -O ckan_default ckan_default -E utf-8
>
>
> sudo mkdir -p /etc/ckan/default
>
> sudo chown -R `whoami` /etc/ckan/
>
>
> cd /usr/lib/ckan/default/src/ckan
>
> paster make-config ckan /etc/ckan/default/development.ini
>
>
>
> sudo nano /etc/ckan/default/development.ini
>
> sqlalchemy.url = postgresql://ckan_default:ckan@localhost/ckan_default
>
> ckan.site_id = default
>
> ckan.site_url = http://100.100.0.201:5000
>
>
>
> #sudo nano /etc/default/jetty
>
> #NO_START=0 # (line 4)
>
> #JETTY_HOST=0.0.0.0 # (line 15)
>
> #JETTY_PORT=8983 # (line 18)
>
>
> #sudo service jetty start
>
>
> tomcat
>
> sudo apt-get install tomcat6-admin solr-tomcat
>
>
>
> sudo nano /etc/tomcat6/tomcat-users.xml
>
> <user username="tomcat" password="tomcat" roles="tomcat,manager,admin"/>
>
>
> sudo nano /etc/tomcat6/server.xml
>
> <Connector port="8983
>
>
>
> sudo mv /etc/solr/conf/schema.xml /etc/solr/conf/schema.xml.bak
>
> sudo ln -s /usr/lib/ckan/default/src/ckan/ckan/config/solr/schema.xml
> /etc/solr/conf/schema.xml
>
>
> #sudo service jetty restart
>
> sudo service tomcat6 restart
>
>
>
> sudo -u postgres createuser -S -D -R -P -l datastore_default
>
> sudo -u postgres createdb -O ckan_default datastore_default -E utf-8
>
>
> sudo nano /etc/ckan/default/development.ini
>
> solr_url=http://127.0.0.1:8983/solr
>
> ckan.plugins = stats text_view image_view recline_view datastore
>
> ckan.datastore.write_url = postgresql://ckan_default:ckan@localhost
> /datastore_default
>
> ckan.datastore.read_url = postgresql://datastore_default:ckan@localhost
> /datastore_default
>
>
> cd /usr/lib/ckan/default/src/ckan
>
> paster db init -c /etc/ckan/default/development.ini
>
>
>
> #PERMISSIONS
>
> paster --plugin=ckan datastore set-permissions -c
> /etc/ckan/default/development.ini | sudo -u postgres psql --set
> ON_ERROR_STOP=1
>
>
>
> ln -s /usr/lib/ckan/default/src/ckan/who.ini /etc/ckan/default/who.ini
>
>
> cd /usr/lib/ckan/default/src/ckan
>
>
> paster sysadmin add deimos -c /etc/ckan/default/development.ini
> paster create-test-data -c /etc/ckan/default/development.ini
>
>
>
> sudo nano /etc/ckan/default/development.ini
>
> ckan.storage_path = /var/lib/ckan
>
> ckan.max_resource_size = 10000
>
> ckan.max_image_size = 200
>
>
> sudo mkdir -p /var/lib/ckan/storage
>
> sudo mkdir -p /var/lib/ckan/resources
>
>
> paster serve /etc/ckan/default/development.ini
>
>
> for debughttp://stackoverflow.com/questions/13480800/ckan-development-install-issue-attributeerror-module-object-has-no-attributecd
> /home/bunny/ckan/lib/default/src/ckansudo apt-get install nodejs
> npmnpm install less nodewatch./bin/less
> ORhttp://docs.ckan.org/en/latest/contributing/frontend/index.htmlsudo
> apt-add-repository ppa:chris-lea/node.jssudo apt-get updatesudo
> apt-get install nodejs npmcd /home/bunny/ckan/lib/default/src/ckannpm
> install less at 1.7.5 nodewatch
> screen
>
> . /usr/lib/ckan/default/bin/activate
>
> ./bin/less
>
> //datapusher
>
> sudo apt-get install python-dev python-virtualenv build-essential
> libxslt1-dev libxml2-dev git
>
>
> . /usr/lib/ckan/default/bin/activate
>
> cd ~
>
> git clone https://github.com/ckan/datapusher
>
> cd datapusher
>
>
> add pdftables to requirements
>
> pip install -r requirements.txt
>
> pip install -r requirements-dev.txt
>
> #pip install sqlalchemy-migrate==0.9.6
>
> #pip install SQLAlchemy==0.9.6
>
> #pip install -e .
>
>
>
>
>
> sudo nano /etc/ckan/default/development.ini
>
> [app:main]
>
> ckan.datapusher.url = http://0.0.0.0:8800/
>
> ckan.plugins = datapusher
>
> ckan.storage_path = /var/lib/ckan
>
> ckan.max_resource_size = 5000
>
>
> sudo mkdir -p /var/lib/ckan/storage
>
> sudo mkdir -p /var/lib/ckan/resources
>
>
> sudo service nginx stop
>
> sudo service apache2 restart
>
>
>
> #sudo visudo -f /etc/sudoers#<user> <host> =#bunny bunny = (root)
> NOPASSWD: /bin/mkdir
> sudo chown -R `whoami` /var/lib/ckansudo chmod -R u+rwx /var/lib/ckan
>
>
> . /usr/lib/ckan/default/bin/activate
>
> JOB_CONFIG='/home/bunny/datapusher/deployment/datapusher_settings.py python
> wsgi.py' (NOT WORKING, use nextline instead)
>
> cd datapusher/
>
> python datapusher/main.py deployment/datapusher_settings.py &
>
>
>
>
> paster serve /etc/ckan/default/development.ini
>
> On Tue, Dec 1, 2015 at 8:19 AM, Matthew Fullerton <matt.fullerton at gmail.com>
> wrote:
>
> > As a minor follow-up to this, is your ckan machine behind a proxy server?
> > On 1 Dec 2015 9:09 a.m., "Ross Jones" <ross at servercode.co.uk> wrote:
> >
> >> Hi Bart,
> >>
> >> I think you have two problems here. The first is that, as far as I am
> >> aware, datapusher doesn't support PDF files, and they should not be being
> >> sent to the datapusher. The error is unfortunate, and should be fixed, but
> >> the PDF shouldn't get that far in the first place.
> >>
> >> > Now what causes the error is that when I try to reach the public
> >> address mentioned before from the command line of my server, I get a
> >> timeout error.
> >>
> >> The second seems to be a connectivity problem. I'm confused as to whether
> >> you're having problems with:
> >>
> >> - datapusher connecting to CKAN
> >> in which case can you confirm your ckan.ini has the correct value for
> >> ckan.site_url ?
> >>
> >> - or, accessing publicly accessible internet urls
> >> in which case there is something wrong with your machine/VM. If you
> >> can't curl the URL you want to fetch, there's little change that datapusher
> >> will be able to either. Can you post the actual URL?
> >>
> >> Cheers,
> >>
> >> Ross
> >>
> >> _______________________________________________
> >> ckan-dev mailing list
> >> ckan-dev at lists.okfn.org
> >> https://lists.okfn.org/mailman/listinfo/ckan-dev
> >> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
> >>
> >
> > _______________________________________________
> > ckan-dev mailing list
> > ckan-dev at lists.okfn.org
> > https://lists.okfn.org/mailman/listinfo/ckan-dev
> > Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
> >
> >
>
>
> --
>
>
> [image: DEIMOS Engenharia, SA] <http://www.deimos.pt>
> *Pedro Gracio Silva*
> Aerospace Systems / Production Center
>
> *Av. D. Joao II, Lt 1.17.01, 10 ? 1998-023 Lisboa, Portugal*
> Ph: +351 21 893 3020 ? Fax: +351 21 896 9099
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20151201/80e5c275/attachment.html>
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> ckan-dev mailing list
> ckan-dev at lists.okfn.org
> https://lists.okfn.org/mailman/listinfo/ckan-dev
> Unsubscribe: https://lists.okfn.org/mailman/options/ckan-dev
>
>
> ------------------------------
>
> End of ckan-dev Digest, Vol 62, Issue 1
> ***************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.okfn.org/pipermail/ckan-dev/attachments/20151201/fbe9f264/attachment-0002.html>
More information about the ckan-dev
mailing list