Spark Streaming gets killed every 2 hours
Spark Streaming app gets killed every 2 hours with any logs when I checked worker logs and found the following error:
14/11/15 13:53:24 INFO network.ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(ip-xxxxxxxxx,38863)
14/11/15 13:53:24 INFO network.ConnectionManager: Removing SendingConnection to ConnectionManagerId(ip-xxxxxxxxx,38863)
14/11/15 13:53:24 ERROR network.SendingConnection: Exception while reading SendingConnection to ConnectionManagerI
I also set spark.cleaner.ttl to 10 minutes, but no luck.
+3
Sachin Janani
source
to share
1 answer
The flow receiver may be stopped. are you using something like this in your code? streamContext.stop (true, true);
If the stream sink is stopped and you are listening to the sink, then the context will stop as well.
0
RisJi
source
to share