How do I combine two dataframes?

I'm using Pandas data frames. I have a initial data frame, say D. I extract two data frames from it like this:

A = D[D.label == k] B = D[D.label != k] 

I want to combine A and B so I can have them as one DataFrame, something like a union operation. The order of the data is not important. However, when we sample A and B from D, they retain their indexes from D.

6 Answers

Deprecation Notice: DataFrame.append and Series.append were deprecated in v1.4.0

I believe you can use the append method

bigdata = data1.append(data2, ignore_index=True) 

to keep their indexes just don't use the ignore_index keyword...

You can also use pd.concat, which is particularly helpful when you are joining more than two dataframes:

bigdata = pd.concat([data1, data2], ignore_index=True, sort=False) 

Thought to add this here in case someone finds it useful. @ostrokach already mentioned how you can merge the data frames across rows which is

df_row_merged = pd.concat([df_a, df_b], ignore_index=True) 

To merge across columns, you can use the following syntax:

df_col_merged = pd.concat([df_a, df_b], axis=1) 

If you're working with big data and need to concatenate multiple datasets calling concat many times can get performance-intensive.

If you don't want to create a new df each time, you can instead aggregate the changes and call concat only once:

frames = [df_A, df_B] # Or perform operations on the DFs result = pd.concat(frames) 

This is pointed out in the pandas docs under concatenating objects at the bottom of the section):

Note: It is worth noting however, that concat (and therefore append) makes a full copy of the data, and that constantly reusing this function can create a significant performance hit. If you need to use the operation over several datasets, use a list comprehension.

If you want to update/replace the values of first dataframe df1 with the values of second dataframe df2. you can do it by following steps —

Step 1: Set index of the first dataframe (df1)

df1.set_index('id') 

Step 2: Set index of the second dataframe (df2)

df2.set_index('id') 

and finally update the dataframe using the following snippet —

df1.update(df2) 

# collect excel content into list of dataframes data = [] for excel_file in excel_files: data.append(pd.read_excel(excel_file, engine="openpyxl")) # concatenate dataframes horizontally df = pd.concat(data, axis=1) # save combined data to excel df.to_excel(excelAutoNamed, index=False) 

You can try the above when you are appending horizontally! Hope this helps sum1