Failed to serialize task: java.lang.reflect.InvocationTargetException

I am getting an exception:

Task serialization failed: java.lang.reflect.InvocationTargetException

My code:

JDBCRDD jdbcRDD = new JDBCRDD(sc.sc(),Connection, getSchema(url), Table_Name, fields, new Filter[]{}, partitionList.toArray(new JDBCPartition[0])
        );

System.out.println("count before to Java RDD=" + jdbcRDD.cache().count());

JavaRDD<Row> jrdd = jdbcRDD.toJavaRDD();

System.out.println("count=" + jrdd.count());

jrdd.foreachPartition( (Iterator<Row> it)-> { 

                new DataPull().updateDB(it); 
  });

} 

      

The DataPull class is serializable.

But i get

15/05/26 10:20:07 ERROR yarn.ApplicationMaster: User class threw exception: Job aborted due to stage failure: Task serialization failed: java.lang.reflect.InvocationTargetException
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:497)
org.apache.spark.serializer.SerializationDebugger$ObjectStreamClassMethods$.getObjFieldValues$extension(SerializationDebugger.scala:240)
org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visitSerializable(SerializationDebugger.scala:150)
org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visit(SerializationDebugger.scala:99)
org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visitSerializable(SerializationDebugger.scala:158)
org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visit(SerializationDebugger.scala:99)
org.apache.spark.serializer.SerializationDebugger$.find(SerializationDebugger.scala:58)
org.apache.spark.serializer.SerializationDebugger$.improveException(SerializationDebugger.scala:39)
org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:47)
org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:80)
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitMissingTasks(DAGScheduler.scala:837)
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:778)
org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:762)
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1362)
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1354)
org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)

      

Need help solving this problem.

+3


source to share


1 answer


In my case, it caused a fast codec. Try using another one to see if that solves your problem.

You can edit conf / spark-defaults.conf and add for example



spark.io.compression.codec      lzf

      

+2


source







All Articles