Rethinkdb SIGTERM, shutdown
I am running RethinkDB in Docker. Everything worked REALLY well for a while until we moved to a new datacenter (but I'm not sure if this is due to traffic at all). Here's what's going on.
I run the rethinkdb container and everything works well. After a while (it changes from an hour or more) I see the following in the Docker logs (highlighted in yellow):
I have no idea why it is getting SIGTERM randomly from the system. Any ideas would be greatly appreciated!
EDIT: I am adding a snippet of the log file for SIGTERM. There doesn't seem to be any pattern according to the timestamps.
2015-07-15T16:15:02.888762613 663165.661585s notice: Server got SIGTERM from pid 0, uid 0; shutting down...
2015-07-17T17:02:11.562306701 13322.914561s notice: Server got SIGTERM from pid 0, uid 0; shutting down...
2015-07-19T18:31:12.499022237 96786.220054s notice: Server got SIGTERM from pid 0, uid 0; shutting down...
2015-07-20T13:52:44.493304030 69690.608865s notice: Server got SIGTERM from pid 0, uid 0; shutting down...
EDIT 2: I ran RethinkDB outside of Docker and I can see this in the logs: Error: Workflow could not sync with main process . Not sure if this is bothering anything. It does not affect the RethinkDB instance at all (all clients remain connected).
2015-07-21T06:53:10.663375859 0.116098s info: Automatically using cache size of 10702 MB
2015-07-21T06:53:10.676277261 0.128998s notice: Listening for intracluster connections on port 29015
2015-07-21T06:53:10.684504354 0.137225s notice: Listening for client driver connections on port 28015
2015-07-21T06:53:10.685485550 0.138206s notice: Listening for administrative HTTP connections on port 8080
2015-07-21T06:53:10.686313405 0.139034s notice: Listening on addresses: 127.0.0.1, 172.17.42.1, 192.151.151.122, ::1, fe80::1879:43ff:fe5e:bdb2%34, fe80::62eb:69ff:fe07:d986%2, fe80::b837:f2ff:fecd:d5cd%4
2015-07-21T06:53:10.686316632 0.139037s notice: Server ready, "0aa312e817ef_nrx" 069ac5b3-9f43-4bbe-9022-c1f006790e99
2015-07-21T06:53:11.558116243 1.010837s error: worker process failed to resynchronize with main process
2015-07-21T06:53:11.558122179 1.010843s notice: A newer version of the RethinkDB server is available: 2.0.4. You can read the changelog at <https://github.com/rethinkdb/rethinkdb/releases>.
EDIT 3: I found another issue here which I think might be the real issue. The rethink adapter (in the app) maintains connections to a DB server that exhausts the available file descriptors / ports on the system. Here is an example printout lsof
. NOTE , this is only a short list. There are hundreds / thousands of them that remain open when multiple people use the system
node 11633 [username_ommitted] 201u IPv4 0x53153575d33d64bb 0t0 TCP 192.168.1.142:61041->[RETHINK_IP]:28015 (ESTABLISHED)
node 11633 [username_ommitted] 202u IPv4 0x53153575d33fa65b 0t0 TCP 192.168.1.142:61053->[RETHINK_IP]:28015 (ESTABLISHED)
node 11633 [username_ommitted] 203u IPv4 0x53153575dd6a5d8b 0t0 TCP 192.168.1.142:61043->[RETHINK_IP]:28015 (ESTABLISHED)
node 11633 [username_ommitted] 204u IPv4 0x53153575bff6717b 0t0 TCP 192.168.1.142:61044->[RETHINK_IP]:28015 (ESTABLISHED)
node 11633 [username_ommitted] 206u IPv4 0x53153575d33e54bb 0t0 TCP 192.168.1.142:61049->[RETHINK_IP]:28015 (ESTABLISHED)
node 11633 [username_ommitted] 207u IPv4 0x53153575d33ef4bb 0t0 TCP 192.168.1.142:61050->[RETHINK_IP]:28015 (ESTABLISHED)
node 11633 [username_ommitted] 208u IPv4 0x53153575d33f2a4b 0t0 TCP 192.168.1.142:61051->[RETHINK_IP]:28015 (ESTABLISHED)
node 11633 [username_ommitted] 209u IPv4 0x53153575c333a17b 0t0 TCP 192.168.1.142:61054->[RETHINK_IP]:28015 (ESTABLISHED)
node 11633 [username_ommitted] 210u IPv4 0x53153575d33b47fb 0t0 TCP 192.168.1.142:61056->[RETHINK_IP]:28015 (ESTABLISHED)
node 11633 [username_ommitted] 211u IPv4 0x53153575d33de17b 0t0 TCP 192.168.1.142:61057->[RETHINK_IP]:28015 (ESTABLISHED)
node 11633 [username_ommitted] 212u IPv4 0x53153575d33f065b 0t0 TCP 192.168.1.142:61058->[RETHINK_IP]:28015 (ESTABLISHED)
node 11633 [username_ommitted] 213u IPv4 0x53153575bff67a4b 0t0 TCP 192.168.1.142:61059->[RETHINK_IP]:28015 (ESTABLISHED)
node 11633 [username_ommitted] 216u IPv4 0x53153575dd68f31b 0t0 TCP 192.168.1.142:61062->[RETHINK_IP]:28015 (ESTABLISHED)
node 11633 [username_ommitted] 217u IPv4 0x53153575dd675a4b 0t0 TCP 192.168.1.142:61063->[RETHINK_IP]:28015 (ESTABLISHED)
source to share
I had the same problem and I suppose it is caused by not closing the database connections (as you noticed in yours EDIT 3
). I'm using RethinkDB in an expressjs app and following the middleware example from here , but I always break the request-response loop inside my controllers without being called next()
, which means the closeConnection middleware was never reached.
source to share