How to keep the same order of schema columns with Spark Dataset map?

I am reading data from a Hive table and then trying to enrich it with an additional column that I got from other columns. But I'm having trouble with Spark changing my schema and ordering all columns by name.

After calling withColumn () and coding it with my enriched class, the schema is correct, but whenever I call map (), the schema and column order changes are wrong. How can I tell Spark to keep the original column order?

session.table("myTable")
    .as(Encoders.bean(Base.class))
    .withColumn("enrichedColumn", lit(""))
    .as(Encoders.bean(Enriched.class))
    .map(enriched -> enriched.enrich(), Encoders.bean(Enriched.class))
    .printSchema();

      

+3


source to share





All Articles