How to suppress "Stage 2 ===>" from the output console into a spark?
I have a dataframe and trying to get a separate counter and be able to get a great score, but whenever the scala program is executed I get this message ([Stage 2:=============================> (1 + 1) / 2])
, how can I suppress this message in the console.
val countID = dataDF.select(substring(col("dataDF"),5,7).distinct().count()
source to share
You need to install spark.ui.showConsoleProgress
infalse
I found this in the comments of the ticket for adding a progress bar.
https://issues.apache.org/jira/browse/SPARK-4017
I haven't seen it in any of the documentation, but it needs to be added.
source to share
If you want to do it by code. Add the following when creating the SparkContext:
import org.apache.log4j.{Level, Logger}
import org.apache.spark.{SparkConf, SparkContext}
Logger.getRootLogger.setLevel(Level.ERROR) // Disabling "INFO" level logs (these lines must be before to create the SparkContext)
val conf = new SparkConf().set("spark.ui.showConsoleProgress", "false").setAppName("myApp")
val sc = new SparkContext(conf)
UPDATE CONTENTS FOR SPARK2 +:
Using SparkSession, you can suppress these messages by adding the following line ( .config("spark.ui.showConsoleProgress", "false")
) to the declaration:
spark = SparkSession
.builder
.master("local[*]")
.appName("myApp")
.config("spark.ui.showConsoleProgress", "false")
source to share