Using a spark to read from a hive
Problem
I am trying to read from a Hive table but I am getting the following error:
[error] (run-main-0) org.apache.spark.sql.AnalysisException: Table or view not found: tags; line 1 pos 14
I posted hive-site.xml
both in $SPARK_HOME/conf
and in $HIVE_HOME/conf
. Also, I had no problem using sqoop to grab data from mysql and import it into the hive. Is there something wrong with my Scala code? Or is it a configuration error?
Scala Code:
package test1
import java.io.File
import org.apache.spark.sql.Row
import org.apache.spark.sql.SparkSession
case class Movie(movieid: String, title: String, genres: String)
case class Tag(userid: String, title: String, tag: String)
object SparkHiveTest {
def main(args: Array[String]) {
val warehouseLocation = new File("spark-warehouse").getAbsolutePath
val spark = SparkSession
.builder()
.master("local")
.appName("SparkHiveExample")
.config("spark.sql.warehouse.dir", warehouseLocation)
.enableHiveSupport()
.getOrCreate()
spark.sql("SELECT * FROM tags").show()
spark.stop()
}
}
hive site.xml:
<configuration>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://localhost/metastore?createDatabaseIfNotExist=true</value>
<description>metadata is stored in a MySQL server</description>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
<description>MySQL JDBC driver class</description>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>hiveuser</value>
<description>user name for connecting to mysql server</description>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>hivepass</value>
<description>password for connecting to mysql server</description>
</property>
</configuration>
source to share
According to the API doc for HiveContext :
An instance of the Spark SQL launcher that integrates with the data stored in Hive. The configuration for the hive is read from the hive-site.xml on the classpath.
So don't forget to place it hive-site.xml
in your project's resource folder in your IDE.
He solved my problem immediately.
source to share