I have a DataFrame df filled with rows and columns where there are duplicate Id's:
Index Id Type 0 a1 A 1 a2 A 2 b1 B 3 b3 B 4 a1 A ... When I use:
uniqueId = df["Id"].unique() I get a list of unique IDs.
How can I however apply this filtering on the whole DataFrame such that it keeps the structure but that the duplicates (based on "Id") are removed?
1 Answer
It seems you need DataFrame.drop_duplicates with parameter subset which specify where are test duplicates:
#keep first duplicate value df = df.drop_duplicates(subset=['Id']) print (df) Id Type Index 0 a1 A 1 a2 A 2 b1 B 3 b3 B #keep last duplicate value df = df.drop_duplicates(subset=['Id'], keep='last') print (df) Id Type Index 1 a2 A 2 b1 B 3 b3 B 4 a1 A #remove all duplicate values df = df.drop_duplicates(subset=['Id'], keep=False) print (df) Id Type Index 1 a2 A 2 b1 B 3 b3 B 
