Executor cannot select postgres driver in standalone Spark cluster

I presented a replay app to start a 2.1 standalone cluster. The game also adds the postgres application dependency, and the application works with the local spark libraries. But while running on a standalone cluster, it gives me an error:

o.a.s.s.TaskSetManager - Lost task 0.0 in stage 0.0 (TID 1, 172.31.21.3, executor 1): java.lang.ClassNotFoundException: org.postgresql.Driver 

      

I placed the following in the spark-defaults.conf directory

spark.executor.extraClassPath   /home/ubuntu/downloads/postgres/postgresql-9.4-1200-jdbc41.jar 
spark.driver.extraClassPath     /home/ubuntu/downloads/postgres/postgresql-9.4-1200-jdbc41.jar 

      

However, executors cannot select a driver. Am I missing something? Help is needed. Thank you.

+3


source to share


1 answer


put postgresql-9.4-1200-jdbc41.jar in the SPARK_DIRECTORY / jars folder and restart spark cluster.



+2


source







All Articles