Openshift app not responding
I have a django application running hosted in openshift, at some point in the last few days it has completely stopped responding in the browser.
I can still use ssh, I can deploy it, and the openshift.redhat.com interface does not report any errors in the application, but a request for any page through the browser simply does not respond. The browser just keeps saying "Waiting for appname-user.rhcloud.com".
The logs can provide some insight.
rhc tail appname
==> app-root/logs/python.log <==
Unable to open logs
(98)Address already in use: make_sock: could not bind to address 127.8.172.129:8080
no listening sockets available, shutting down
Unable to open logs
(98)Address already in use: make_sock: could not bind to address 127.8.172.129:8080
no listening sockets available, shutting down
Unable to open logs
... (ad infinitum)
I don't think it stopped responding after deployment, it worked, then it didn't. I don't know what happened.
Would be grateful for any ideas what might be causing this or how to debug / resolve it.
EDIT
Based on a sensible fantasy prediction, I tried a force-stop / start loop and this was logged:
$ rhc app force-stop -a notebook
$ rhc app start -a notebook
$ rhc tail notebook
==> app-root/logs/python.log <==
(98)Address already in use: make_sock: could not bind to address 127.8.172.129:8080
no listening sockets available, shutting down
Unable to open logs
(98)Address already in use: make_sock: could not bind to address 127.8.172.129:8080
no listening sockets available, shutting down
Unable to open logs
[Sun Nov 16 01:26:13 2014] [notice] SELinux policy enabled; httpd running as context unconfined_u:system_r:openshift_t:s0:c4,c359
[Sun Nov 16 01:26:13 2014] [notice] Digest: generating secret for digest authentication ...
[Sun Nov 16 01:26:13 2014] [notice] Digest: done
[Sun Nov 16 01:26:13 2014] [notice] Apache/2.2.15 (Unix) mod_wsgi/3.4 Python/2.7.5 configured -- resuming normal operations
==> app-root/logs/postgresql.log <==
2014-11-16 06:26:05 GMT LOG: trying another address for the statistics collector
2014-11-16 06:26:05 GMT LOG: could not bind socket for statistics collector: Cannot assign requested address
2014-11-16 06:26:05 GMT LOG: disabling statistics collector for lack of working socket
2014-11-16 06:26:05 GMT WARNING: autovacuum not started because of misconfiguration
2014-11-16 06:26:05 GMT HINT: Enable the "track_counts" option.
2014-11-16 06:26:06 GMT LOG: database system was interrupted; last known up at 2014-11-15 07:33:35 GMT
2014-11-16 06:26:06 GMT LOG: database system was not properly shut down; automatic recovery in progress
2014-11-16 06:26:06 GMT LOG: record with zero length at 0/1D48850
2014-11-16 06:26:06 GMT LOG: redo is not required
2014-11-16 06:26:06 GMT LOG: database system is ready to accept connections
I could be wrong, but I still have a feeling that the app is fine, it could be DNS related. I tried the DNS trace of my app url ( http://notebook-davur.rhcloud.com ) and the trace ends up with this nugget:
Sending request to "ns3.p23.dynect.net" (208.78.71.23)
Received authoritative (AA) response:
-> Header: Non-Existent Domain
source to share
I was pretty sure this was not a code issue as the issue did not start immediately after deploying any changes. To make sure the problem was not with the code, I created a new app and pushed the same code to it.
$ rhc app create newappname python-2.7
$ rhc cartridge add postgresql-9.2 -a newappname
Notice the Git URL returned after the command app create
.
$ git remote add newapptest GIT_URL
$ git push newapptest --force
$ open http://newapptest-mydomain.rhcloud.com
This voila! It worked.
Since it was a simple hobby project without any important data in the database, I went ahead and deleted it via the web interface and re executed the above commands, replacing newappname
with originalappname
and instead of adding it as a new git remote
updated remote pointer to now old original / now deleted application.
$ git remote set-url openshift GIT_URL
( Note : all data in the database is deleted by the app, so if the data is valuable, starting with pg_dump and ending with pg_restore)
And my original url responds again. I'm pretty sure it was related to Openshift's own settings, possibly DNS, etc.
source to share