GBMILs: Gradient Boosting Models for Multiple Instance Learning

Andrei Konstantinov; Lev Utkin; Vladimir Muliukha; Vladimir Zaborovsky

Andrei Konstantinov Peter the Great St.Petersburg Polytechnic University https://orcid.org/0000-0002-1542-6480
Lev Utkin Peter the Great St.Petersburg Polytechnic University https://orcid.org/0000-0002-5637-1420
Vladimir Muliukha Peter the Great St.Petersburg Polytechnic University https://orcid.org/0000-0002-3583-7324
Vladimir Zaborovsky Peter the Great St.Petersburg Polytechnic University https://orcid.org/0000-0003-2284-9833

Keywords: Multiple Instance Learning, Gradient Boosting Machine, Attention Mechanism, Neural Networks

Abstract

An approach based on using the gradient boosting machine for solving the Multiple Instance Learning (MIL) problem under condition of small tabular data is proposed. The MIL deals with labeled objects called bags which consist of several parts of the objects called instances with unknown labels and each bag label depends on the instance labels. Three modifications of the approach are developed and studied. They are determined by different aggregation functions which combine the intermediate predictions of the instance classes in each bag and allow gradient-based optimization through them. The modifications are based on the following aggregation functions: the Hard Max Aggregation, the Simple Attention Aggregation, and the Ensemble of gradient boosting machines in fusion with the Attention Neural Networks. The former two modifications can use an arbitrary decision tree gradient boosting model, which allows iterative training on loss gradients. The later modification simultaneously optimizes an ensemble of parallel gradient boosting models and the parameters of neural network. Numerical experiments with tabular datasets illustrate the proposed modifications of the approach and their superiority comparing to available MIL approaches accompanied by accuracy improvement up to 8%.