spark跟pandas数据转换

因为传统的机器学习是基于sklearn,xgboost,有着丰富分算法库,spark mlib不能满足所有的需求. spark来处理数据预处理和特征工程,sklearn,xgboost来训练. 需要spark和sklearn,xgboost进行数据转化.

pandas dataframe转 spark dataframe,

import pandas as pd
from pyspark.sql import SparkSession


#pandas读取cvs,形成dataframe,
userDF = pd.read_csv("src/main/resources/upload.csv")

#启动spark
spark = SparkSession \
        .builder \
        .appName("Python Spark SQL Hive integration example") \
        .enableHiveSupport() \
        .getOrCreate()

#spark读取pandas dataframe,形成spark dataframe
sparkDF = spark.createDataFrame(userDF)
sparkDF.show()

spark dataframe 转 pandas data,download.py

from pyspark.sql import SparkSession

spark = SparkSession \
        .builder \
        .appName("Python Spark SQL Hive integration example") \
        .enableHiveSupport() \
        .getOrCreate()

spark.sql("CREATE TABLE IF NOT EXISTS user (userid int, name string)")
spark.sql("LOAD DATA LOCAL INPATH 'src/main/resources/user.txt' INTO TABLE user")

userSparkDF = spark.sql("select * from user")


userPandasDF = userSparkDF.toPandas()

print userPandasDF

spark.stop()