Set parameter within a step of a Pipeline (XGB : eval_set)

I am trying to fit an XGBoost model to my data with an early stopping round and therefore an eval_set parameter. However, I am using a pipeline that does preprocessing before the model fitting step. I would want to set the parameter "eval_set" to that particular step and have used the syntax "stepname__eval_set=.." which doesn't seem to work. Here is my code :

XGB=XGBRegressor(n_estimators=10000,learning_rate=0.05,verbose=False) myPip=Pipeline(steps=[("preprocessing",preprocessor), ("model",XGB)]) myPip.fit(X_train2,y_train,model__eval_set=[(X_val2,y_val)],model__early_stopping_rounds=5) 

It returns the following error

ValueError Traceback (most recent call last) C:\Users\PCGZ~1\AppData\Local\Temp/ipykernel_17976/459508294.py in <module> 2 myPip=Pipeline(steps=[("preprocessing",preprocessor), 3 ("model",XGB)]) ----> 4 myPip.fit(X_train2,y_train,model__eval_set=[(X_val2,y_val)],model__early_stopping_rounds=5) 5 y_pred_val=myPip.predict(X_val2) 6 y_pred_train=myPip.predict(X_train2) ~\anaconda3\lib\site-packages\sklearn\pipeline.py in fit(self, X, y, **fit_params) 344 if self._final_estimator != 'passthrough': 345 fit_params_last_step = fit_params_steps[self.steps[-1][0]] --> 346 self._final_estimator.fit(Xt, y, **fit_params_last_step) 347 348 return self ~\anaconda3\lib\site-packages\xgboost\core.py in inner_f(*args, **kwargs) 618 for k, arg in zip(sig.parameters, args): 619 kwargs[k] = arg --> 620 return func(**kwargs) 621 622 return inner_f ~\anaconda3\lib\site-packages\xgboost\sklearn.py in fit(self, X, y, sample_weight, base_margin, eval_set, eval_metric, early_stopping_rounds, verbose, xgb_model, sample_weight_eval_set, base_margin_eval_set, feature_weights, callbacks) 1012 with config_context(verbosity=self.verbosity): 1013 evals_result: TrainingCallback.EvalsLog = {} -> 1014 train_dmatrix, evals = _wrap_evaluation_matrices( 1015 missing=self.missing, 1016 X=X, ~\anaconda3\lib\site-packages\xgboost\sklearn.py in _wrap_evaluation_matrices(missing, X, y, group, qid, sample_weight, base_margin, feature_weights, eval_set, sample_weight_eval_set, base_margin_eval_set, eval_group, eval_qid, create_dmatrix, enable_categorical, feature_types) 497 evals.append(train_dmatrix) 498 else: --> 499 m = create_dmatrix( 500 data=valid_X, 501 label=valid_y, ~\anaconda3\lib\site-packages\xgboost\sklearn.py in _create_dmatrix(self, ref, **kwargs) 932 except TypeError: # `QuantileDMatrix` supports lesser types than DMatrix 933 pass --> 934 return DMatrix(**kwargs, nthread=self.n_jobs) 935 936 def _set_evaluation_result(self, evals_result: TrainingCallback.EvalsLog) -> None: ~\anaconda3\lib\site-packages\xgboost\core.py in inner_f(*args, **kwargs) 618 for k, arg in zip(sig.parameters, args): 619 kwargs[k] = arg --> 620 return func(**kwargs) 621 622 return inner_f ~\anaconda3\lib\site-packages\xgboost\core.py in __init__(self, data, label, weight, base_margin, missing, silent, feature_names, feature_types, nthread, group, qid, label_lower_bound, label_upper_bound, feature_weights, enable_categorical) 741 return 742 --> 743 handle, feature_names, feature_types = dispatch_data_backend( 744 data, 745 missing=self.missing, ~\anaconda3\lib\site-packages\xgboost\data.py in dispatch_data_backend(data, missing, threads, feature_names, feature_types, enable_categorical) 955 return _from_tuple(data, missing, threads, feature_names, feature_types) 956 if _is_pandas_df(data): --> 957 return _from_pandas_df(data, enable_categorical, missing, threads, 958 feature_names, feature_types) 959 if _is_pandas_series(data): ~\anaconda3\lib\site-packages\xgboost\data.py in _from_pandas_df(data, enable_categorical, missing, nthread, feature_names, feature_types) 402 feature_types: Optional[FeatureTypes], 403 ) -> DispatchedDataBackendReturnType: --> 404 data, feature_names, feature_types = _transform_pandas_df( 405 data, enable_categorical, feature_names, feature_types 406 ) ~\anaconda3\lib\site-packages\xgboost\data.py in _transform_pandas_df(data, enable_categorical, feature_names, feature_types, meta, meta_type) 376 for dtype in data.dtypes 377 ): --> 378 _invalid_dataframe_dtype(data) 379 380 feature_names, feature_types = _pandas_feature_info( ~\anaconda3\lib\site-packages\xgboost\data.py in _invalid_dataframe_dtype(data) 268 type_err = "DataFrame.dtypes for data must be int, float, bool or category." 269 msg = f"""{type_err} {_ENABLE_CAT_ERR} {err}""" --> 270 raise ValueError(msg) 271 272 ValueError: DataFrame.dtypes for data must be int, float, bool or category. When categorical type is supplied, The experimental DMatrix parameter`enable_categorical` must be set to `True`. Invalid columns:MSZoning: object, Street: object, LotShape: object, LandContour: object, Utilities: object, LotConfig: object, LandSlope: object, Neighborhood: object, Condition1: object, Condition2: object, BldgType: object, HouseStyle: object, RoofStyle: object, RoofMatl: object, Exterior1st: object, Exterior2nd: object, MasVnrType: object, ExterQual: object, ExterCond: object, Foundation: object, BsmtQual: object, BsmtCond: object, BsmtExposure: object, BsmtFinType1: object, BsmtFinType2: object, Heating: object, HeatingQC: object, CentralAir: object, Electrical: object, KitchenQual: object, Functional: object, GarageType: object, GarageFinish: object, GarageQual: object, GarageCond: object, PavedDrive: object, SaleType: object, SaleCondition: object 

PS : The prepocessing pipeline isn't the issue, since the pipeline worked fine with other models that do not take the eval_set parameter.

Thank you in advance for your kindly help.

1 Answer

I have found "a" solution for this particular problem : which was passing the eval_set parameter (which was unprocessed data) to the model that was fitted using preprocessed data. Trying to evaluate it with unprocessed data that ultimately had a different column structure gave the error shown above. The idea is to perform the pipeline step by step, just like so :

XGB=XGBRegressor(n_estimators=10000,learning_rate=0.05,verbose=False) #This is our original Pipeline myPip=Pipeline(steps=[("preprocessing",preprocessor), ("model",XGB)]) #We fit the preprocessing step on the unprocessed training data myPip[0].fit(X_train2,y_train) #And transform both the training and validation data X_trainXGB=myPip[0].transform(X_train2) X_valXGB=myPip[0].transform(X_val2) #We fit the model on the clean data myPip[1].fit(X_trainXGB,y_train,eval_set=[(X_valXGB,y_val)],early_stopping_rounds=5) #And predict the result using the preprocessed (transformed) validation data y_preds=myPip[1].predict(X_valXGB)