How to gracefully restart a NodeJS server on an uncaught exception?
I am using the express-domain-middleware module to transfer my requests across the domain. In my error middleware, I have the following check:
if(err.domain) {
//something should happen here, it completely uncaught
}
else {
//silly kids, I'll log this and send a 500;
The above code snippet is based on this example in the documentation express-domain-middleware
:
//with domain-middleware
app.use(require('express-domain-middleware'));
app.use(app.router);
app.use(function errorHandler(err, req, res, next) {
console.log('error on request %d %s %s', process.domain.id, req.method, req.url);
console.log(err.stack);
res.send(500, "Something bad happened. :(");
if(err.domain) {
//you should think about gracefully stopping & respawning your server
//since an unhandled error might put your application into an unknown state
}
});
app.get('/error', function(req, res, next) {
db.query('SELECT happiness()', process.domain.intercept(function(rows) {
fs.readFile('asldkfjasdf', process.domain.intercept(function(contents) {
process.nextTick(process.domain.intercept(function() {
throw new Error("The individual request will be passed to the express error handler, and your application will keep running.");
}));
}));
}));
});
What I don't want to do if this exception happened outside of the domain is a call process.exit(1)
(I don't think anyone would actually do this, do they?). I also don't want to leave my expression in a dangerous state. I would rather just log the uncaught exception so I can refactor it afterwards, gracefully restart that domain (server?) And get stuff done.
Assumptions
- By using this approach, a larger application will not be compromised by single domain instability. I can restart without interrupting any other connections. ( Edit ): I created a route that specifically throws an unhandled exception and logs into the application in a separate tab. By forcing the exception and restarting Forever, I also killed the session "law" - I'm guessing I did something wrong).
- The user won't know what happened if I just restart the server. I am guessing that they will just see the login screen as soon as they update, without knowing why. If it was a handled exception, I would send a JSON response to the client explaining what happened, or redirect the user to an error page. What is the correct way to handle this scenario when the exception is unhandled?
TL; DR;
- This app runs in Azure on IIS. What command (s) to send in order for the app to restart gracefully? I'm not familiar with that, what kind of behavior takes IIS vs
Forever
. I purposefully ran the application locally withForever
and watched it automatically start when an unhandled exception was raised, but I am developing it on a Mac and the application is running on Windows. Windows solution needed =)
Edit edit . Please correct me if there are misconceptions about what is happening within the domain and exceptions are thrown. I'm here to learn!
source to share
The node program can also be restarted using spawn:
var exec = require('child_process').exec;
spawn('sh', ['-c', 'sleep 2; node /home/username/Desktop/MainNodeApplication.js'], {
detached: true,
stdio: ['ignore', 'ignore', 'ignore']
});
process.exit(0);
The previous code is written to run on a Linux machine with a very simple node script stored in /home/username/Desktop/MainNodeApplication.js. This code spawns another process, detaches it from the parent process, and exits immediately. Meanwhile, the spawned process restarts the node script 2 seconds after we're good and sure the other instance had time to close.
It doesn't get you where you want it, but it will allow you to gracefully restart your code. You can pass its arguments when you start it so that it knows better what state it should appear in, or you can pass it error information to pass to the user. You can do this by simply adding arguments to the call to node in the appearance arguments.
source to share