Convert Dstream to Spark DataFrame using pyspark

I want to convert to Dstream

in DataFrame

order to apply the same transforms in this DataFrame

and call the model NaiveBayesModel

to predict the target probability, I am using Apache Spark 2.1.1, Dstream

built from socketTextStream

. I tried to call the foreachRDD

function Dstream

but it doesn't work.

def predict(rdd):
    count = rdd.count()
    if(count>0):
        hashingTF = HashingTF(numFeatures=1000)
        features = hashingTF.transform(rdd)
        result = model.transform(features)
        return result.probability
    else:
        print("No data receveid")

model = NaiveBayesModel.load(sc, "ML_models/NaiveClassifier/naiveBayesClassifier-2010-09-10-08-51-25")
lines = ssc.socketTextStream("localhost", 9999)
tweets = lines.map(lambda v: json.loads(v))
text_dstream = tweets.map(lambda tweet: tweet['text'])
df = text_dstream.foreachRDD(lambda rdd: predict(rdd))
ssc.start()             # Start the computation
ssc.awaitTermination()

      

        
        
        
      

    

The following error message appears

AttributeError: 'RDD' object has no attribute '_jdf'

      

        
        
        
      

    

My idea is to convert Dstream

to Spark DataFrame

and apply the transform using:

#Tokenize sentiment text
tokenizer = Tokenizer(inputCol="SentimentText", outputCol="SetimentTextTokenize")
wordsData = tokenizer.transform(df)

hashingTF = HashingTF(inputCol="SetimentTextTokenize", outputCol="rawFeatures", numFeatures=1000)
featurizedData = hashingTF.transform(wordsData)