Problem importing from sqlserver to hive using apache sqoop?

When I run the following sqoop command from $ SQOOP_HOME / bin it works fine

sqoop import --connect "jdbc:sqlserver://ip_address:port_number;database=database_name;username=sa;password=sa@Admin" --table $SQL_TABLE_NAME --hive-import --hive-home $HIVE_HOME --hive-table $HIVE_TABLE_NAME -m 1 

      

But when I run the same command in a loop for different databases from a bash script like this

while IFS='' read -r line || [[ -n $line ]]; do  
$DATABASE_NAME=$line

sqoop import --connect "jdbc:sqlserver://ip_address:port_number;database=$DATABASE_NAME;username=sa;password=sa@Admin" --table $SQL_TABLE_NAME --hive-import --hive-home $HIVE_HOME --hive-table $HIVE_TABLE_NAME -m 1  

done < "$1"

      

I am passing the database names to my bash script in a text file as a parameter. My hive table is the same as I want to add data from all databases to only one hive table.

For the first two to three databases, it works fine after it starts giving the following errors

15/06/25 11:41:06 INFO mapreduce.Job: Job job_1435124207953_0033 failed with state FAILED due to:  

15/06/25 11:41:06 INFO mapreduce.ImportJobBase: The MapReduce job has already been retired. Performance  

15/06/25 11:41:06 INFO mapreduce.ImportJobBase: counters are unavailable. To get this information,  

15/06/25 11:41:06 INFO mapreduce.ImportJobBase: you will need to enable the completed job store on  

15/06/25 11:41:06 INFO mapreduce.ImportJobBase: the jobtracker with:  
11:41:06 INFO mapreduce.ImportJobBase:mapreduce.jobtracker.persist.jobstatus.active = true  
11:41:06 INFO mapreduce.ImportJobBase: mapreduce.jobtracker.persist.jobstatus.hours = 1  
15/06/25 11:41:06 INFO mapreduce.ImportJobBase: A jobtracker restart is required for these settings  
15/06/25 11:41:06 INFO mapreduce.ImportJobBase: to take effect.  
15/06/25 11:41:06 ERROR tool.ImportTool: Error during import: Import job failed!  

      

I have already restarted my hadoop multi-node cluster by changing the mapred-site.xml file with the above two ie parameters

mapreduce.ImportJobBase:mapreduce.jobtracker.persist.jobstatus.active = true  
 mapreduce.jobtracker.persist.jobstatus.hours = 1  

      

However, I faced the same problem. Since I just started learning sqoop, any help would be appreciated.

+3


source to share





All Articles