ANN, SVM, XGB and KPLS models are ensembles of several submodels, whereas MLR models are singletons. A predicted property value combines the predicted values from each of a number of separately trained submodels, all of which use the same inputs, parameters and training pool. They differ in which observations from that training pool are assigned to the training set and which are assigned to the verification set. For ANNEs, the submodels are individual neural networks that all have the same inputs and number of neurons - i.e., have the same "architecture.
Building a consensus prediction from an ensemble of well-trained submodels almost always improves the “signal-to-noise” ratio, because the submodels are all "aiming" at the same target (the signal), but the degree to which they miss that target (the noise) differs, with the noise tending to cancel out in the consensus prediction. This presumes that the component models are substantially different from each other. We accomplish this in part by assigning random initial weights to each submodel but also by splitting the training pool into a training and verification set in a ratio of about 2:1, then shuffling the assignments to generate different splits ("folds") for each model.
Which submodels get included in the ensemble?
By default, ADMET Modeler trains more submodels than the default 33 that are required for each ensemble - 165 for ANNEs (Network multiplier = 5) and 40 for SVM and KPLS ensembles. Once trained, models are ranked based on the sum of their verification set performance and the absolute difference between their training and verification set performance. The latter component helps eliminate over-trained submodels. RMSE is used as the performance metric for regression, whereas Youden's index is used for classification. Those submodels that perform best are incorporated into the ensemble model and their statistics are recorded in the associated .rms file.
Ensemble predictions
Predictions from selected submodels are simply averaged for regression models, but classification submodel outputs can be combined either by voting or by averaging.
SVM ensemble classification outputs are determined by which class gets the most votes from the individual SVM submodels that make up the ensemble.
In the voting method for binary classification ANNEs, each network sets a threshold independently that yields the best Youden's index for its training set and votes for a positive classification if the output exceeds its threshold. If the number of networks voting "positive" exceeds a threshold fraction of the number of networks in the ensemble (50% by default), then the ensemble classification is "positive"; otherwise it is "negative." Those positive vote tallies are also used for performing confidence analysis.
Outputs from the individual ANNs are continuous variables. In the averaging method for binary classification models, they are averaged and compared to a threshold that gives the best Youden statistic across the full training pool. If the average exceeds that threshold, the ensemble classifies it as "positive"; otherwise it is classified as "negative." Since a voting tally is needed for confidence analysis, a "vote" for each network is determined by comparing the individual ANN output to the overall ensemble threshold. The ANN's vote is "positive" if its own output exceeds that threshold. This means that an ensemble classification may be "positive" even when a minority of ANNs vote that way in terms of CFA; see Clark et al. (2014) for a detailed explanation of this rather subtle point.
Each network in a multi-class ANNE produces an output for each class. When averaging is used, the network output for each class is averaged across all networks, and the class with the highest score "wins" - i.e., that class becomes the ensemble prediction. In the case of simple multiclass voting, each network casts a vote for the class that receives the highest score for that network. If one class receives a simple majority, it wins. Otherwise the list of contending classes is reduced by applying the method of plurality with elimination. If that fails to break the tie, the winning class is the one among those remaining in contention that receives the highest overall score summed over all networks.