I am having a following DataFrame schema:
| str_A | str_B | co_A | co_B | ... | co_X |
where co_A and co_B stand for custom objects
Those objects inherit object and have defined rich comparison methods
FWIW: __hash__ method is also overriden and calls super().__hash__()
What I want to do is to sort by the two custom columns and here is where the confusion comes.
The following code works as expected:
df.sort_values(by=['str_A', 'str_B']) The following code works as well:
df.sort_values(by=['co_A']) but what doesn't work is:
df.sort_values(by=['co_A', 'co_B']) which throws
ValueError: Categorical categories must be unique
And it turns out that some combinations of co_x and str_x do work, others do not.
I thought it might be cause of mixed types and so I dropped the string values and left only the custom objects, the error persists. I am clueless here and I would appreciate any help.
EDIT: I am using pandas version 0.23.0
To generate a simple data which I can reproduce the issue with, the following sample code can be used
def feed(): return type('Feed', (), { attr: names.get_first_name() if attr.startswith('substr') else names.get_last_name() for attr in CPE_VERSION_ATTRIBUTE_LIST }) feed = [feed() for i in range(100)] EDIT2:
I am using jupyter notebook, the CustomObject class is imported from a lib.
What I did is I created another simple class directly in the jupyter with the similar methods as the original one to check whether there is something wrong with the data... and it worked. Which is even more confusing. There must be something wrong with the CustomObject class (I did not write the class, however, I can not find anything that could cause the issue).
The class is supposed to handle version comparisons -- the specification can be found in the gist
EDIT3: Found the root cause of the issue, don't know how to solve it, yet
Having implemented CustomObject like
class CustomObject: def __init__(self, stream: str): if stream is None: raise TypeError() self.stream = stream def __repr__(self): return "{cls!s}(stream={stream!r})".format( cls=self.__class__.__name__, stream=self.stream ) def __str__(self): return "{stream!s}".format( stream=self.stream ) def __lt__(self, other): if other is None: return False return self.stream < other.stream def __gt__(self, other): if other is None: return True return self.stream > other.stream # <<<<<<< This causes the issue # def __eq__(self, other): # if other is None: # return False # return self.stream == other.stream # ======= def __hash__(self): return super().__hash__() It works as expected without the commented code above.
31 Answer
This is an old question, but I ran into the same problem.
Cause
The problem is that your __hash__ function should always return True when two objects are the equal (as defined by your __eq__ function).
Right now this is not the case because your equivalence is based on the stream attribute and your class uses the __hash__ function of object which is (I believe) based on the objects location in memory.
Solution
Defining your hash function as below might do the trick.
def __hash__(self): return hash(self.stream) 
