Clever Geek Handbook

How to use matplotlib to plot sql-pyspark results

I am new to pyspark. I want to plot the result using matplotlib but don't know which function to use. I was looking for a way to convert the sql result to pandas and then I used a graph.

+3


source to share


1 answer


Hi I found a solution for this. I converted sql dataframe to pandas dataframe and then I managed to plot graphs. below is a sample code. of



pyspark.sql import Row
from pyspark.sql import HiveContext
import pyspark
from IPython.display import display
import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline 
sc = pyspark.SparkContext()
sqlContext = HiveContext(sc)
test_list = [(1, 'hasan'),(2, 'nana'),(3, 'dad'),(4, 'mon')]
rdd = sc.parallelize(test_list)
people = rdd.map(lambda x: Row(id=int(x[0]), name=x[1]))
schemaPeople = sqlContext.createDataFrame(people)
# Register it as a temp table
sqlContext.registerDataFrameAsTable(schemaPeople, "test_table")
df1=sqlContext.sql("Select * from test_table")
pdf1=df1.toPandas()
pdf1.plot(kind='barh',x='name',y='id',colormap='winter_r')
+5


source






More articles:


All Articles