Given a machine learning model RBF SVC called 'm', I performed a gridSearchCV on gamma value, to optimize recall. I'm looking to answer to this: "The grid search should find the model that best optimizes for recall. How much better is the recall of this model than the precision?"
So I did the gridSearchCV:
grid_values = {'gamma': [0.001, 0.01, 0.05, 0.1, 1, 10, 100]} grid_m_re = GridSearchCV(m, param_grid = grid_values, scoring = 'recall') grid_m_re.fit(X_train, y_train) y_decision_fn_scores_re = grid_m_re.decision_function(X_test) print('Grid best parameter (max. recall): ', grid_m_re.best_params_) print('Grid best score (recall): ', grid_m_re.best_score_) This tell me the best model is for gamma=0.001 and it has a recall score of 1.
I'm wondering how to get the precision for this model to get the trade of of this model, cause the GridSearchCV only has attribute to get what it was optimize for.([Doc sklearn.GridSearchCV][1])
2 Answers
Not sure if there's an easier/more direct way to get this, but this approach also allows you to capture the 'best' model to play around with later:
First do you CV fit on training data:
grid_m_re = GridSearchCV(m, param_grid = grid_values, scoring = 'recall') grid_m_re.fit(X_train, y_train) Once you're done, you can pull out the 'best' model (as determined by your scoring criteria during CV), and then use it however you want:
m_best = grid_m_re.best_estimator_ and in your specific case:
from sklearn.metrics import precision_score y_pred = m_best.predict(X_test) precision_score(y_test, y_pred) You can easily overfit if you don't optimize both, C and gamma at the same time.
if you plot the SVC with C on the X axis, gamma on the y axis and recall as color you get some kind of V-Shape, see here
So if you do grid search, better optimize for both, C and gamma, at the same time.
The problem is that usually you get the best results for small C-Values, and in that area the V-shape has it's pointy end: is not very big and difficult to hit.
I recently used:
make a random grid of 10 points every point contains C, gamma, direction, speed cut the dataset with stratifiedShuffleSplit fit & estimate score with cross validation repeat: kill the worst two points the best two points spawn a kid move every point in its direction with just a little bit of random, fit & estimate score with cross validation (if a point notice it goes downward, turn around and half speed) until break criterion is hit Worked like a charm.
I used the max distance in the feature space divided by four as initial speed, the direction had a maximum random of pi/4
Well, the cross validation was a bit costly.
Cleptocreatively inspired by this paper.
... and another edit:
I used between 10 and 20 cycles in the loop to get the perfect points. If your dataset is too big to do several fits, make a representative subset for the first few trainings...
