I have modified the table using hiveql. Then it didn't work to show the table from spark-sql. ERROR: path does not exist

I changed the table by HiveQL.

"ALTER TABLE new_law_area_2 RENAME TO law_area"

      

I then intended to show my table using spark-sql.

"SELECT * FROM law_area LIMIT 10"

      

But it didn't work ... with this error.

org.spark_project.guava.util.concurrent.UncheckedExecutionException:org.apache.spark.sql.AnalysisException

17/04/18 14:17:47 ERROR SparkSQLDriver: Failed in [select * from law_area limit 10]
org.spark_project.guava.util.concurrent.UncheckedExecutionException: org.apache.spark.sql.AnalysisException: Path does not exist: hdfs: //dmlab/apps/hive/warehouse/dimension.db/new_law_area_2;
    at org.spark_project.guava.cache.LocalCache $ LocalLoadingCache.getUnchecked (LocalCache.java:4882)
    at org.spark_project.guava.cache.LocalCache $ LocalLoadingCache.apply (LocalCache.java:4898)
    at org.apache.spark.sql.hive.HiveMetastoreCatalog.lookupRelation (HiveMetastoreCatalog.scala: 128)
    at org.apache.spark.sql.hive.HiveSessionCatalog.lookupRelation (HiveSessionCatalog.scala: 70)
    at org.apache.spark.sql.catalyst.analysis.Analyzer $ ResolveRelations $ .org $ apache $ spark $ sql $ catalyst $ analysis $ Analyzer $ ResolveRelations $$ lookupTableFromCatalog (Analyzer.scala: 457)
    at org.apache.spark.sql.catalyst.analysis.Analyzer $ ResolveRelations $$ anonfun $ apply $ 8.applyOrElse (Analyzer.scala: 479)
    at org.apache.spark.sql.catalyst.analysis.Analyzer $ ResolveRelations $$ anonfun $ apply $ 8.applyOrElse (Analyzer.scala: 464)
    at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan $$ anonfun $ resolveOperators $ 1.apply (LogicalPlan.scala: 61)
    at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan $$ anonfun $ resolveOperators $ 1.apply (LogicalPlan.scala: 61)
    at org.apache.spark.sql.catalyst.trees.CurrentOrigin $ .withOrigin (TreeNode.scala: 70)
    at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperators (LogicalPlan.scala: 60)
    at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan $$ anonfun $ 1.apply (LogicalPlan.scala: 58)
    at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan $$ anonfun $ 1.apply (LogicalPlan.scala: 58)
    at org.apache.spark.sql.catalyst.trees.TreeNode $$ anonfun $ 4.apply (TreeNode.scala: 307)
    at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator (TreeNode.scala: 188)
    at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren (TreeNode.scala: 305)
    at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperators (LogicalPlan.scala: 58)
    at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan $$ anonfun $ 1.apply (LogicalPlan.scala: 58)
    at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan $$ anonfun $ 1.apply (LogicalPlan.scala: 58)
    at org.apache.spark.sql.catalyst.trees.TreeNode $$ anonfun $ 4.apply (TreeNode.scala: 307)
    at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator (TreeNode.scala: 188)
    at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren (TreeNode.scala: 305)
    at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperators (LogicalPlan.scala: 58)
    at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan $$ anonfun $ 1.apply (LogicalPlan.scala: 58)
    at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan $$ anonfun $ 1.apply (LogicalPlan.scala: 58)
    at org.apache.spark.sql.catalyst.trees.TreeNode $$ anonfun $ 4.apply (TreeNode.scala: 307)
    at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator (TreeNode.scala: 188)
    at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren (TreeNode.scala: 305)
    at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperators (LogicalPlan.scala: 58)
    at org.apache.spark.sql.catalyst.analysis.Analyzer $ ResolveRelations $ .apply (Analyzer.scala: 464)
    at org.apache.spark.sql.catalyst.analysis.Analyzer $ ResolveRelations $ .apply (Analyzer.scala: 454)
    at org.apache.spark.sql.catalyst.rules.RuleExecutor $$ anonfun $ execute $ 1 $$ anonfun $ apply $ 1.apply (RuleExecutor.scala: 85)
    at org.apache.spark.sql.catalyst.rules.RuleExecutor $$ anonfun $ execute $ 1 $$ anonfun $ apply $ 1.apply (RuleExecutor.scala: 82)
    at scala.collection.LinearSeqOptimized $ class.foldLeft (LinearSeqOptimized.scala: 124)
    at scala.collection.immutable.List.foldLeft (List.scala: 84)
    at org.apache.spark.sql.catalyst.rules.RuleExecutor $$ anonfun $ execute $ 1.apply (RuleExecutor.scala: 82)
    at org.apache.spark.sql.catalyst.rules.RuleExecutor $$ anonfun $ execute $ 1.apply (RuleExecutor.scala: 74)
    at scala.collection.immutable.List.foreach (List.scala: 381)
    at org.apache.spark.sql.catalyst.rules.RuleExecutor.execute (RuleExecutor.scala: 74)
    at org.apache.spark.sql.execution.QueryExecution.analyzed $ lzycompute (QueryExecution.scala: 69)
    at org.apache.spark.sql.execution.QueryExecution.analyzed (QueryExecution.scala: 67)
    at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed (QueryExecution.scala: 50)
    at org.apache.spark.sql.Dataset $ .ofRows (Dataset.scala: 63)
    at org.apache.spark.sql.SparkSession.sql (SparkSession.scala: 592)
    at org.apache.spark.sql.SQLContext.sql (SQLContext.scala: 699)
    at org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.run (SparkSQLDriver.scala: 62)
    at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd (SparkSQLCLIDriver.scala: 335)
    at org.apache.hadoop.hive.cli.CliDriver.processLine (CliDriver.java:376)
    at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver $ .main (SparkSQLCLIDriver.scala: 247)
    at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main (SparkSQLCLIDriver.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:497)
    at org.apache.spark.deploy.SparkSubmit $ .org $ apache $ spark $ deploy $ SparkSubmit $$ runMain (SparkSubmit.scala: 745)
    at org.apache.spark.deploy.SparkSubmit $ .doRunMain $ 1 (SparkSubmit.scala: 187)
    at org.apache.spark.deploy.SparkSubmit $ .submit (SparkSubmit.scala: 212)
    at org.apache.spark.deploy.SparkSubmit $ .main (SparkSubmit.scala: 126)
    at org.apache.spark.deploy.SparkSubmit.main (SparkSubmit.scala)

How can I solve this problem?

plz ... my all tables didn't work this way ...

I want to use spark-sql

+3


source to share


2 answers


try -



alter table law_area set location 'hdfs://dmlab/apps/hive/warehouse/dimension.db/law_area'

      

+3


source


I had the same problem. I just wanted to add, if DuduMarkovitz's answer doesn't work for you,

ALTER TABLE law_area SET SERDEPROPERTIES ( 'path'='hdfs://dmlab/apps/hive/warehouse/dimension.db/law_area')

      



worked for me.

+1


source







All Articles