Custom SerDe is not supported by Impala, what's the best way to request files in CSV with double quotes?
I have CSV data with each field double-quoted. When I created the catch table serde was used 'com.bizo.hive.serde.csv.CSVSerde' When Impala asks for the previous table, I get an error. SerDe not found.
I added the CSV Serde JAR file to the / usr / lib / impala / lib folder.
Later, after examining the Impala documentation, Impala does not support custom SERDEs. In that case, how can I overcome this problem so that my quoted CSV data is taken. I want to use CSV Serde because it has commas in the values, which is a legal vavlue field.
Thank you so much
source to share
Can you use Hive? If so, here's an approach that might work. CREATE
your table as EXTERNAL TABLE
in Hive and use the SERDE
CREATE statement in the correct place (I think you need something like ROW FORMAT SERDE your_serde_here at the end of the CREATE TABLE statement). Before that, you may need:
ADD JAR 'hdfs:///path/to/your_serde.jar'
Note that the jar needs to be somewhere in the hdfs and the triple /// needed for it to work ...
Then, back in Hive, duplicate the table into another table, which is stored in a format that Impala can easily work with, such as PARQUET. This copying takes place as follows:
CREATE TABLE copy_of_table
STORED AS PARQUET AS
SELECT * FROM your_original_table
Now in Impala run:
INVALIDATE DATA copy_of_table
You should be prepared to work with copy_of_table in Impala.
Let me know if this will work, how could I do something like this in the near future.
source to share
Inside the hive
CREATE TABLE mydb.my_serde_table_impala AS SELECT FROM mydb.my_serde_table
Inside the Impala
INVALIDATE METADATA mydb.my_serde_table_impala
Add these steps to enable first drop the _impala table with any population or swallow files for the serde table.
Impala bypasses MapReduce, unlike Hive. Therefore Impala cannot / does not use SerDe the way MapReduce does.
source to share