Clever Geek Handbook

Why is dropna () not working?

System: Spark 1.3.0 (Anaconda Python dist.) On Cloudera Quickstart VM 5.4

Here's the Spark DataFrame:

from pyspark.sql import SQLContext
from pyspark.sql.types import *
sqlContext = SQLContext(sc)

data = sc.parallelize([('Foo',41,'US',3),
                       ('Foo',39,'UK',1),
                       ('Bar',57,'CA',2),
                       ('Bar',72,'CA',3),
                       ('Baz',22,'US',6),
                       (None,75,None,7)])

schema = StructType([StructField('Name', StringType(), True),
                     StructField('Age', IntegerType(), True),
                     StructField('Country', StringType(), True),
                     StructField('Score', IntegerType(), True)])

df = sqlContext.createDataFrame(data,schema)

data.show() 

Name Age Country Score
Foo  41  US      3    
Foo  39  UK      1    
Bar  57  CA      2    
Bar  72  CA      3    
Baz  22  US      6    
null 75  null    7

However, none of these works!

df.dropna()
df.na.drop()

I am getting this message:

>>> df.show()
Name Age Country Score
Foo  41  US      3    
Foo  39  UK      1    
Bar  57  CA      2    
Bar  72  CA      3    
Baz  22  US      6    
null 75  null    7    
>>> df.dropna().show()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/spark/python/pyspark/sql/dataframe.py", line 580, in __getattr__
    jc = self._jdf.apply(name)
  File "/usr/lib/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line 538, in __call__
  File "/usr/lib/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py", line 300, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o50.apply.
: org.apache.spark.sql.AnalysisException: Cannot resolve column name "dropna" among (Name, Age, Country, Score);
    at org.apache.spark.sql.DataFrame$$anonfun$resolve$1.apply(DataFrame.scala:162)
    at org.apache.spark.sql.DataFrame$$anonfun$resolve$1.apply(DataFrame.scala:162)
    at scala.Option.getOrElse(Option.scala:120)
    at org.apache.spark.sql.DataFrame.resolve(DataFrame.scala:161)
    at org.apache.spark.sql.DataFrame.col(DataFrame.scala:436)
    at org.apache.spark.sql.DataFrame.apply(DataFrame.scala:426)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)
    at py4j.Gateway.invoke(Gateway.java:259)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:207)
    at java.lang.Thread.run(Thread.java:745)

Has anyone else experienced this problem? What's the workaround? Pyspark seems like I'm looking for a column called "na". Any help would be appreciated!

+3


source to share


2 answers


tl; dr Methods na

and dropna

are only available since Spark 1.3.1.

Several mistakes you made:

  • data = sc.parallelize([....('',75,'', 7 )])

    , you should use ''

    to view None

    , however it is just a string and not null

  • na

    and dropna

    - both methods in the dataFrame class, so you have to call it with df

    .



Runnable Code:

data = sc.parallelize([('Foo',41,'US',3),
                       ('Foo',39,'UK',1),
                       ('Bar',57,'CA',2),
                       ('Bar',72,'CA',3),
                       ('Baz',22,'US',6),
                       (None, 75, None, 7)])

schema = StructType([StructField('Name', StringType(), True),
                 StructField('Age', IntegerType(), True),
                 StructField('Country', StringType(), True),
                 StructField('Score', IntegerType(), True)])

df = sqlContext.createDataFrame(data,schema)

df.dropna().show()
df.na.drop().show()
+6


source


I understand the question was asked a year ago, in case of exiting the solution for Scala, below in case someone lands here looking for the same

val data = sc.parallelize(List(("Foo",41,"US",3), ("Foo",39,"UK",1),   
("Bar",57,"CA",2), ("Bar",72,"CA",3), ("Baz",22,"US",6), (None, 75,  
   None, 7)))
val schema = StructType(Array(StructField("Name", StringType, true),   
  StructField("Age", IntegerType, true), StructField("Country", 
  StringType, true), StructField("Score", IntegerType, true)))

val dat = data.map(d => Row(d._1, d._2, d._3, d._4))
val df = sqlContext.createDataFrame(dat, schema)
df.na.drop()


Note: The above solution will still not give the correct result in Scala, not sure if the implementation is different between Scala and python communication. na.drop works if the missing data is null. It does not match “and“ No. ”One alternative in this case is to use the withColumn function to handle missing values ​​of various forms.

-1


source






More articles:


All Articles