Failed to rename Dataframe column

When displaying table sections using Spark 1.6.3, I am unable to rename the Dataframe column. To reproduce the problem, I create the following partitioned table

sqlContext.sql("create table test (foo string) partitioned by (dt string)")

val df = sqlContext.sparkContext.parallelize(Seq(("foo", 1), ("bar", 2)))
  .toDF("column", "dt")

df.write.mode("overwrite").partitionBy("dt").insertInto("test")

      

        
        
        
      

    

I can get a list of available partitions in a table

scala> sqlContext.sql(s"show partitions test").show

+------+
|result|
+------+
|  dt=1|
|  dt=2|
+------+

      

        
        
        
      

    

Now I would like to rename the column using withColumnRenamed

like this

sqlContext.sql(s"show partitions test").withColumnRenamed("result", "partition").show

      

        
        
        
      

    

What happens with the following error message

org.apache.spark.sql.AnalysisException: Result of result attribute (s) # 835 is missing from result # 826 in statement! Project [result # 835 AS partition # 836];

I can get around this problem by using a simple alias

> sqlContext.sql(s"show partitions test").select($"result".as("partition")).show

+---------+
|partition|
+---------+
|     dt=1|
|     dt=2|
+---------+

      

        
        
        
      

    

But why is this happening in the first place?