hyperopt对xgboost和lightgbm进行调参

hyperopt简介

hyperopt是一个贝叶斯优化来调整参数的工具, 优化输入参数是的目标函数的值最小, 当模型的参数过多时, 该方法比gridsearchcv要快,并且有比较好的效果， 或者结合使用，对于重要的单个参数使用gridsearchcv暴力穷举缩小主要参数范围， 再使用hyperopt加上其他次重要参数在小范围空间进行精细调参

---

hyperopt测试

安装

pip install hyperopt

寻找函数的最小值x使得函数y=x最小，使用hyperopt下的fmin方法，指定tpe退火算法进行搜索

from hyperopt import fmin, tpe, hp
best = fmin(
    fn=lambda x: x,  # 优化最小化函数
    space=hp.uniform('x', 0, 1),  # x的搜索空间, uniform0-1之间的均匀分布
    algo=tpe.suggest,  # 使用的搜索算法
    max_evals=100)  # 最大评估次数
print(best)

输出结果,结果返回best是一个字典

100%|██████████| 100/100 [00:00<00:00, 153.98trial/s, best loss: 5.599423732018799e-05]
{'x': 0.0011367664025380307}

寻找一个值x使得平方函数最小

best = fmin(
    fn=lambda x: (x-1)**2,
    space=hp.uniform('x', -2, 2),
    algo=tpe.suggest,
    max_evals=100)

print(best)

输出结果为接近1

100%|██████████| 100/100 [00:00<00:00, 379.37trial/s, best loss: 9.565468616456154e-07]
{'x': 0.9990219678626724}

把每次寻找结果画出来,可见随着搜索次数的增加，搜索结果无限逼近1

import matplotlib.pyplot as plt

res = []
for epoch in range(1, 100):
    best = fmin(
        fn=lambda x: (x-1)**2,
        space=hp.uniform('x', -2, 2),
        algo=tpe.suggest,
        max_evals=epoch)
    res.append(best["x"])
plt.plot(res)
plt.show()

hyperopt_test.png

---

hyperopt的使用步骤

（1）定义最小化的目标函数
（2）定义搜索空间
（3）定义存储搜索点数据的数据库
（4）定义搜索算法

---

定义变量搜索空间

对于变量的变化范围与取值概率，常用的有以下几类,实现接口在 from hyperopt import hp下

(1) hp.choice(label, options)
options输入是list或者tuple, 在上限限之间枚举一个值，适合离散变量的参数
(2) hp.randint(label, upper)
upper是一个整数, 返回子啊[0, ipper)下的一个随机整数
(3) hp.uniform(label, low, high)
在一个low和high之间的均匀分布上搜索，在浮点数空间上进行
(4) hp.quniform(label, low, high, q)
在low和high上下区间均匀搜索离散值, q是离散值间隔，搜索round（uniform（low，high）/ q）* q的值，可以简单理解为在low和high之间（附近），均匀搜索间隔为q的整数值，用代码模拟如下

import numpy as np
import pandas as pd

def get_value(low, high, q):
    return round(np.random.uniform(low, high) / q) * q

dic = {}
for i in range(500):
    res = get_value(3, 20, 4)
    dic[res] = dic.get(res, 0) + 1

pd.Series(dic).sort_index().plot(kind="bar")

hp.quniform测试(3, 20, 4).png

dic = {}
for i in range(500):
    res = get_value(3, 20, 3)
    dic[res] = dic.get(res, 0) + 1

pd.Series(dic).sort_index().plot(kind="bar")

hp.quniform测试(3, 20, 3).png

定义存储参数的数据库

使用默认的trials数据库，直接在fmin中定义trials对象, 返回一个代表所有搜索内容的字典列表

trials = hyperopt.Trials()

定义搜索方法

对应algo中的algo参数，支持以下算法：

(1) 随机搜索(hyperopt.rand.suggest)
(2) 模拟退火(hyperopt.anneal.suggest)
(3) TPE算法（hyperopt.tpe.suggest，算法全称为Tree-structured Parzen Estimator Approach）

使用hyperopt对模型进行调参

（1）以一个二分类问题为例， 对原始数据进行特征工程，处理成标准模型数据, 其中preprocessor是一个ColumnTransformer对象，整体封装为一个pipeline，fit_transform得到标准数据

import numpy as np
import pandas as pd
import xgboost as xgb
from xgboost import XGBClassifier
from lightgbm import LGBMClassifier
from sklearn.preprocessing import OneHotEncoder
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import FunctionTransformer
from sklearn.model_selection import cross_val_score
import hyperopt

df = pd.read_csv("data/train.csv")

df = pd.concat([df[df["y"] == "yes"], df[df["y"] == "no"].sample(n=df[df["y"] == "yes"].shape[0] * 3)], ignore_index=True)

continus_cols = ["age", "duration", "campaign", "pdays", "previous", "emp.var.rate", "cons.price.idx", "cons.conf.idx", "euribor3m", 
                 "nr.employed"]

category_cols = ["job", "marital", "education", "default", "housing", "loan", "contact", "month", "day_of_week", "poutcome"]

preprocessor = ColumnTransformer(
            transformers=[
                    ("none", FunctionTransformer(lambda x: x), continus_cols), 
                    ("onehot", OneHotEncoder(handle_unknown='ignore'), category_cols)
                    ])

train = preprocessor.fit_transform(df[continus_cols + category_cols])
train_y = [0 if x == "no" else 1 for x in df["y"].values]

（2）定义优化函数， 定义优化函数为训练集Xgboost 5折交叉验证的1-auc，每一轮将搜索参数字典和auc打印出来

def hyperopt_objective(params):
    
    model = XGBClassifier(
        max_depth=int(params['max_depth']) + 3,
        learning_rate=params['learning_rate'],
        n_estimators=int(params['n_estimators']),
        min_child_weight=int(params['min_child_weight']),
        subsample=params["subsample"],
        colsample_bytree=params["colsample_bytree"],
        gamma=params["gamma"]
    )
     
    res = np.mean(cross_val_score(model, train, train_y, cv=5, n_jobs=-1, scoring='roc_auc'))
    print("*" * 30)
    print(params)
    print("roc_auc: {}".format(res))
    
    return 1 - res

在搜索参数传入模型时进行转化

(1) max_depth浮点数转化为整数， 并且基数 + 3
(2) n_estimators浮点数转化为整数
(3) min_child_weight浮点数转化为整数

（3）定义调参空间， 对xgboost的主要参数指定搜索空间

params_space = {
    'max_depth': hyperopt.hp.randint('max_depth', 12),
    'learning_rate': hyperopt.hp.uniform('learning_rate', 1e-3, 5e-1),
    'n_estimators': hyperopt.hp.quniform("n_estimators", 100, 200, 20),
    'min_child_weight': hyperopt.hp.randint('min_child_weight', 3),
    'subsample': hyperopt.hp.uniform('subsample', 0.6, 1),
    'colsample_bytree': hyperopt.hp.uniform('colsample_bytree', 0.6, 1),
    'gamma': hyperopt.hp.uniform('gamma', 0, 0.3)
}

（4） 定义存储过程参数的默认数据库trials 对象

trials = hyperopt.Trials()

(5) 定义主方法fmin，输出的best是一个参数字典

best = hyperopt.fmin(
    hyperopt_objective,
    space=params_space,
    algo=hyperopt.tpe.suggest,
    max_evals=20,
    trials=trials)

print("最佳参数")
print(best)

搜索过程如下

******************************                                                    
{'colsample_bytree': 0.6587057989285068, 'gamma': 0.08349627219616713, 'learning_rate': 0.1071323085829065, 'max_depth': 11, 'min_child_weight': 2, 'n_estimators': 200.0, 'subsample': 0.8267234620571373}
roc_auc: 0.94656374379679                                                         
******************************                                                    
{'colsample_bytree': 0.637307962287014, 'gamma': 0.05174238990136167, 'learning_rate': 0.47001705750952716, 'max_depth': 6, 'min_child_weight': 0, 'n_estimators': 140.0, 'subsample': 0.8097032550531738}
roc_auc: 0.9418830756277139                                                       
100%|██████████| 20/20 [02:51<00:00,  8.57s/trial, best loss: 0.04881901526421206]
最佳参数
{'colsample_bytree': 0.7786809516292569, 'gamma': 0.1362980300099654, 'learning_rate': 0.08161287652125764, 'max_depth': 2, 'min_child_weight': 0, 'n_estimators': 140.0, 'subsample': 0.8670303638365404}

对于lightgbm，使用sklearn的接口LGBMClassifier，调参部分代码如下

def hyperopt_objective(params):
    
    model = LGBMClassifier(
        max_depth=int(params['max_depth']) + 3,
        learning_rate=params['learning_rate'],
        n_estimators=int(params['n_estimators']),
        subsample=params["subsample"],
        colsample_bytree=params["colsample_bytree"],
        min_data_in_leaf=int(params["min_data_in_leaf"])  # 最小叶子节点样本数
    )
    
    res = np.mean(cross_val_score(model, train, train_y, cv=5, n_jobs=-1, scoring='roc_auc'))
    print("*" * 30)
    print(params)
    print("roc_auc: {}".format(res))
    
    return 1 - res

# 定义调参空间
params_space = {
    'max_depth': hyperopt.hp.randint('max_depth', 12),
    'learning_rate': hyperopt.hp.uniform('learning_rate', 1e-3, 5e-1),
    'n_estimators': hyperopt.hp.quniform("n_estimators", 100, 200, 20),
    'subsample': hyperopt.hp.uniform('subsample', 0.6, 1),
    'colsample_bytree': hyperopt.hp.uniform('colsample_bytree', 0.6, 1),
    'min_data_in_leaf': hyperopt.hp.quniform('min_data_in_leaf', 3, 15, 2)
}

trials = hyperopt.Trials()

best = hyperopt.fmin(
    hyperopt_objective,
    space=params_space,
    algo=hyperopt.tpe.suggest,
    max_evals=20,
    trials=trials)

print("最佳参数")
print(best)

输出如下

******************************                                                    
{'colsample_bytree': 0.9772472660582058, 'learning_rate': 0.38724477853350703, 'max_depth': 9, 'n_estimators': 140.0, 'subsample': 0.8082547449297106}
roc_auc: 0.9378750118665657                                                       
******************************                                                    
{'colsample_bytree': 0.8698414936492014, 'learning_rate': 0.47084277807947017, 'max_depth': 1, 'n_estimators': 140.0, 'subsample': 0.7345708189893152}
roc_auc: 0.9272081797753863                                                       
******************************                                                    
{'colsample_bytree': 0.6787426765354041, 'learning_rate': 0.0882329240953757, 'max_depth': 5, 'n_estimators': 120.0, 'subsample': 0.7412720323927691}
roc_auc: 0.95005503163141                                                         
100%|██████████| 20/20 [00:23<00:00,  1.18s/trial, best loss: 0.04919464019243969]
最佳参数
{'colsample_bytree': 0.8939241012291288, 'learning_rate': 0.03462416674347271, 'max_depth': 10, 'n_estimators': 180.0, 'subsample': 0.7304965473301595}

lightGBM的调参攻略

参数解析

• max_depth: 默认是-1, 即不限制树的最大深度, lightGBM不使用max_depth控制树的复杂度, 使用num_leaves ,如果指定了可以一定程度限制树深, 防止过拟合
• num_leaves: 默认是31, 1 < num_leaves <= 131072, 这个值设置成2的max_depth次方-1, 这个值越大树复杂度越大, 在训练集上的精度越高
• max_bin: 默认是是255. max_bin > 1, 越大训练集精度越高
• learning_rate: 默认是0.1, 低学习率配合高迭代num_iterations,或者n_estimators
• min_data_in_leaf: 叶子节点最小数据量,小于这个量树就不再生长了, 默认是20, 增大这个值可以限制树的生长
• lambda_l1: 默认0
• lambda_l2: 默认0
• dart: 带有dropout的gbdt, 可能可以提高精度

For Faster Speed

• Use bagging by setting bagging_fraction and bagging_freq
• Use feature sub-sampling by setting feature_fraction
• Use small max_bin
• Use save_binary to speed up data loading in future learning
• Use parallel learning, refer to Parallel Learning Guide

For Better Accuracy

• Use large max_bin (may be slower)
• Use small learning_rate with large num_iterations
• Use large num_leaves (may cause over-fitting)
• Use bigger training data
• Try dart

Deal with Over-fitting

• Use small max_bin
• Use small num_leaves
• Use min_data_in_leaf and min_sum_hessian_in_leaf
• Use bagging by set bagging_fraction and bagging_freq
• Use feature sub-sampling by set feature_fraction
• Use bigger training data
• Try lambda_l1, lambda_l2 and min_gain_to_split for regularization
• Try max_depth to avoid growing deep tree

lgb_params = {'num_leaves': 2**7-1,
              'min_data_in_leaf': 25, 
              'objective':'regression_l2',
              'max_depth': -1,
              'learning_rate': 0.1,
              'min_child_samples': 20,
              'boosting': 'gbdt',
              'feature_fraction': 0.6,
              'bagging_fraction': 0.9,
              'bagging_seed': 11,
              'metric': 'mae',
              'seed':1024,
              'lambda_l1': 0.2}