pyspark: create more than 1 dataframe fails

I'd like to convert several large Pandas dataframes into Spark dataframes and then manipulate and merge them, as follows:

import pandas as pd from pyspark import SparkContext,SQLContext df1 = pd.read_csv('data1.cat',delim_whitespace=True) df2 = pd.read_csv('data2.cat',delim_whitespace=True) sc = SparkContext() sql = SQLContext(sc) spark_df1 = sql.createDataFrame(df1) spark_df2 = sql.createDataFrame(df2) 

But something goes wrong, and I get the following error:

 File "/home/user/anaconda3/envs/conda_py3.6.8/lib/python3.6/site-packages/pyspark/sql/context.py", line 307, in createDataFrame return self.sparkSession.createDataFrame(data, schema, samplingRatio, verifySchema) File "/home/user/anaconda3/envs/conda_py3.6.8/lib/python3.6/site-packages/pyspark/sql/session.py", line 724, in createDataFrame data = self._convert_from_pandas(data, schema, timezone) File "/home/user/anaconda3/envs/conda_py3.6.8/lib/python3.6/site-packages/pyspark/sql/session.py", line 487, in _convert_from_pandas np_records = pdf.to_records(index=False) File "/home/user/anaconda3/envs/conda_py3.6.8/lib/python3.6/site-packages/pandas/core/frame.py", line 1839, in to_records return np.rec.fromarrays(arrays, dtype={"names": names, "formats": formats}) File "/home/user/.local/lib/python3.6/site-packages/numpy/core/records.py", line 617, in fromarrays descr = sb.dtype(dtype) ValueError: name already used as a name or title 

Is it possible to create multiple Spark dataframes in the same session like this?

1 Answer

The error is thrown because your pandas DataFrame (df1 of df2) has the same column name for 2 or more columns. Either change the columns name or drop it.

Thanks, Sagar