Executor cannot select postgres driver in standalone Spark cluster
I presented a replay app to start a 2.1 standalone cluster. The game also adds the postgres application dependency, and the application works with the local spark libraries. But while running on a standalone cluster, it gives me an error:
o.a.s.s.TaskSetManager - Lost task 0.0 in stage 0.0 (TID 1, 172.31.21.3, executor 1): java.lang.ClassNotFoundException: org.postgresql.Driver
I placed the following in the spark-defaults.conf directory
spark.executor.extraClassPath /home/ubuntu/downloads/postgres/postgresql-9.4-1200-jdbc41.jar spark.driver.extraClassPath /home/ubuntu/downloads/postgres/postgresql-9.4-1200-jdbc41.jar
However, executors cannot select a driver. Am I missing something? Help is needed. Thank you.
+3
Rishikesh Teke
source
to share
1 answer
put postgresql-9.4-1200-jdbc41.jar in the SPARK_DIRECTORY / jars folder and restart spark cluster.
+2
learner
source
to share