基于Transformer-CatBoost融合模型的不平衡数据分类研究

Research on Imbalanced Data Classification Based on Transformer-CatBoost Fusion Model

摘要: 在二分类问题中，由于数据失衡处理不当、特征繁杂等问题，捕捉特征之间的复杂相关关系成为挑战。为改善这种现象，建立Transformer-CatBoost融合模型，引入Transformer挖掘用户数据的深层信息与CatBoost抗过拟合实现高效分类。首先，为Transformer编码器增加违约分类头，构建BaseTransformer。然后，将M个BaseTransformer集成作为模型的第一层学习器，得到第一层的预测结果与原特征一同输入第二层学习器CatBoost，实现基于Stacking的模型融合。采取多样化的模型评价指标，对比10种数据不平衡处理方法，选择了NCR方法参与实验，随后引入Optuna方法优化模型参数。最后，将模型与各种基准模型比较，借助消融实验证得模型的有效性与可行性，并利用Lending Club数据集证得模型的泛化能力。

Abstract: In binary classification problems, due to problems such as improper handling of data imbalance and complex features, capturing the correlation between features becomes a challenge. To improve this phenomenon, a Transformer-CatBoost fusion model is established, introducing Transformer to mine the deep information of user data and CatBoost to resist overfitting and achieve efficient classification. Firstly, a default classification head is introduced to the encoder of Transformer to build BaseTransformer. Then, M BaseTransformers are integrated as the first layer of the model. The prediction results of first layer together with the original features are input into the second layer CatBoost to build the stacking fusion model. Ten data imbalance processing methods are applied for comparison by using a variety of model evaluation indicators, and the NCR method is selected. Then the Optuna method is introduced to optimize the model parameters. Finally, the model is compared with various benchmark models. The effectiveness and feasibility of the model are verified by ablation experiments. And we prove the generalization ability of the model using the Lending Club dataset.