Clever Geek Handbook

Find column index by looking up dataset column header in Apache Spark Java

I have a Spark Dataset like below

       0         1                  2          3
    +------+------------+--------------------+---+
    |ItemID|Manufacturer|       Category     |UPC|
    +------+------------+--------------------+---+
    |   804|         ael|Brush & Broom Han...|123|
    |   805|         ael|Wheel Brush Parts...|124|
    +------+------------+--------------------+---+

I need to find the position of a column. Distorting the column header.

    For Example:-

    int position=getColumnPosition("Category");

This should return 2.

Is there a Spark function supported in Datatype Dataset to find the index or any java functions that might run on the Spark dataset. thanks in advance

+3


source to share


1 answer


You need to access the schema and read the field index like this:



int position= df.schema().fieldIndex("Category");
+8


source






More articles:


All Articles