3.4.6. Finding the Best Ensemble

3.4.6. Finding the Best Ensemble

In most cases, identifying the best model is a somewhat subjective matter. The result depends on what criteria have been applied and how the comparisons have been made. ADMET Modeler allows you to choose your own criteria and provides two methods - Dot Product and Borda Rank - for performing the comparison. Details of how the comparisons are made are provided elsewhere.

Experienced modelers rely on multiple statistics as well as visualizations of predictions for both the training pool and test set data. The default settings for regression models include root mean squared error (RMSE), Spearman rank correlation coefficient (SRCC) and the coefficient of determination (Q-squared), whereas they include Sensitivity, Specificity and the Matthews Correlation Coefficient for binary classification models. Models for endpoints having more than two classes use Youden and Matthews Correlation Coefficient by default. The statistics can be enabled and disabled based on user preferences. Users can also adjust the number of “best” ensembles to highlight in the Model Performance Grid as well as display a text viewer containing a sorted list of ranks for all models in the grid.

The Dot Product Method works as follows:

1. All statistics are scaled to [0.5,1.0] across the entire model grid.

2. The dot product for each model is calculated as depicted below.

3. Models are sorted by rank and the top performing ones are highlighted in the Model Performance Grid.

The dot product process is illustrated schematically below using sensitivity (Se), specificity (Sp) and Matthews correlation coefficient (MCC) as performance criteria with (top) and without (bottom) taking performance on the verification set into account.

To apply the Borda Rank Method:

1. Calculate rank order for each statistic for each data set: training, verification and test.

2. Sum ranks across statistics and sets. See Figure 3-4g.

3. Rank models by ascending Borda Count.

The process is illustrated below for the simplified case in which only Youden's indexes J are considered and verify statistics are ignored.

The test set is not used in the model building process, so its performance is expected to be a more robust metric than the performance of the training set and verification set. The default Dot Product Method settings takes the product of the test set metric with the geometric mean of the training and verification sets, which keeps the weight of the test set consistent whether or not verification statistics are included. The training and verification set performances are pooled when CFA threshold refinement is applied to classification models or averaging is used. The geometric mean for dot product calculation is used to make the ranks obtained relatively insensitive to whether or not the verification set is included. Similarly, the training (and, if used, the verification set) ranks are averaged before being added to the test rank when Borda Ranks are calculated. Performance on the test set is always included, except in cases where models are trained on all of the data - i.e., where the test set size is 0%.

An autoscaled [0.5, 1] model complexity metric is also considered by default. This gives weight to the models in the Model Performance Grid based on the number of adjustable weights used in each architecture.

The Dot Product and Borda Rank methods give model rankings reasonably similar to rankings made by experts and in empirical tests utilizing varied subsets of well-known data. Models ranked highly by these methodologies regularly identify models that perform better on external data sets than do lower-ranked models.