Clever Geek Handbook

Spark Streaming gets killed every 2 hours

Spark Streaming app gets killed every 2 hours with any logs when I checked worker logs and found the following error:

14/11/15 13:53:24 INFO network.ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(ip-xxxxxxxxx,38863)
14/11/15 13:53:24 INFO network.ConnectionManager: Removing SendingConnection to ConnectionManagerId(ip-xxxxxxxxx,38863)
14/11/15 13:53:24 ERROR network.SendingConnection: Exception while reading SendingConnection to ConnectionManagerI

I also set spark.cleaner.ttl to 10 minutes, but no luck.

+3


source to share


1 answer


The flow receiver may be stopped. are you using something like this in your code? streamContext.stop (true, true);



If the stream sink is stopped and you are listening to the sink, then the context will stop as well.

0


source






More articles:


All Articles