Classification Confidence | ADMET Predictor Documentation

For classification models, the number of votes is a natural measure of ensemble uncertainty. Verma et al. (2015) estimated classification uncertainty by taking the fraction of agreement within the ensemble to be a direct measure of the probability that a given classification is correct. If 7 of 10 models agree on the classification for a particular compound, for example, then one's confidence that the ensemble classification is correct - i.e., its predictive value - is 70% ,

When the K individual networks in an ensemble are statistically independent and all have the same probability p of misclassifying a dubious compound, the likelihood of making an erroneous classification is expected to follow the binomial distribution defined by:

where ε(k) is the probability of exactly k networks getting the classification wrong. As noted above, the networks have all been trained using the same training pool and are trying to predict the same endpoint, so they are not independent. On the other hand, they have been trained on random training/verification splits of the training pool and start from random initial weights, so the probability of misclassification does vary somewhat from network to network for any given compound. The net result is that the observed distribution of classification error is not as sharp as expected from a simple binomial distribution - in statistical parlance, it is "overdisperse."

We have found that a better approach is to fit the distribution of predictions across vote tallies k to one function and the distribution of errors to another. It turns out that [beta binomial (BB) distributions ]work well for modeling such repeated non-independent trials:

where K is the number of networks in the ensemble. The shape of the distribution is controlled by the parameters α and β. The distribution is concave down (like the binomial) when both parameters are greater than one and concave up (bowl shaped) when either parameter is less than 1. It is flat (a uniform distribution) when α = β = 1. In all cases, the distribution mean equals α/(α + β). It approaches the binomial distribution with p = α/(α + β) when α + β becomes large. Such would be the case if the networks in the ensemble were independent and the likelihood of misclassification was uniform.

The confidence analysis procedure can be summarized as follows, where the "vote tally” is the number of networks in the ensemble that classify a given observation as a positive:

Fit the distribution of training pool predictions as a function of the vote tally k to a beta binomial φ(k)
Fit the distribution of training pool errors with respect to the vote tally k to a second beta binomial, ε(k)
Calculate MR, the overall misclassification rate for the training pool.
The contingent uncertainty distribution - i.e., the probability that a classification based on a tally of k positive votes is wrong - can then be estimated as u(k) = MR×ε(k)/φ(k).
Calculate the estimated confidence as 1- u(k). Note that this corresponds to the positive or negative predictive value for a given prediction, depending on the output classification.

The confidence estimate shown in the ADMET Predictor spreadsheet is offset in parentheses for selected classification models, with a maximum value of 99% and minimum that is typically near 50%. When individual confidence estimates cannot be made reliably - e.g., because too few errors were observed - the class confidences (overall positive and negative predictive values) are provided in the model tooltip. A detailed derivation along with technical details of how CFA has been implemented and validation of its utility has been published (Clark 2014).

Split beta binomial error models

Many ADMET data sets are unbalanced either in terms of the class populations or in terms of their relative information content. In either case, the assumption that it is appropriate to pool positive and negative errors can break down. Modeler tests for this possibility by also carrying out a split error CFA in which beta binomials ε1(k) and ε0(k) are fit to predictions for the positive and negative observations separately across the full range of vote tallies. The split uncertainty function is then given by u(k) = MR₁×ε₁(k)/φ(k) for k below the tally threshold k* and by u(k) = MR₀×ε₀(k)/φ(k) for k above k*, where MR₁ and MR₀ are the misclassification rates for positive and negative observations, respectively.

The decision threshold k* is set to a half-integral number of votes (16.5 by default), so there can be no predictions or errors at that value. Instead the values of u(k) immediately above and below are connected by a straight line so that that a cross-over uncertainty value can be estimated. There are theoretical reasons why this crossover should always lie near 0.50 for robust error models (Clark 2014); our experience indicates that deviations beyond ±0.05 are usually cause for concern.

After both unified and split error CFAs have been run, the observed error rates in the training pool are compared to the predictive uncertainty profiles across the middle of the range of possible tallies. Whichever CFA has the smaller sum of squared deviations between (1/3)×K and (2/3)×K (K = 33 by default) is retained and associated with the ANNE from which it was derived.

Threshold refinement

In early versions of ADMET Modeler, classification using the voting method was based strictly on simple majority rule: if more than half of the networks in an ensemble classified a compound as a "positive," then the overall ensemble prediction was positive. It turns out, however, that this rule is often suboptimal, especially for data sets that are strongly unbalanced. Shifting the voting threshold to equal the mean of ε(k) reduces the number of overall errors. In the usual unbalanced case where positives are in the minority, the number of false positives decreases more than the number of false negatives increases, which improves the odds that a given positive prediction will be correct - i.e., the positive predictive value.

Such a shift tends to ensure that the minimum confidence for any prediction does not fall much below 50%. This is optimal when the cost of positive and negative errors is comparable, so it is enabled by default. In cases where the costs of the different types of error are strongly skewed, however (e.g., in silico screening for mutagenicity), the cost of false alarms is low relative to the cost of missed positives, and turning Refine threshold OFF may be preferable.

Note that only the parameters from the unified CFA error distribution are used to refine thresholds.

Interaction with threshold averaging

The technique was initially developed for voting-based classification models, wherein each network determines its vote by applying its own classification threshold to its individual output. Models built with Use averaging method turned ON, in contrast, make their predictions based on the average output of the networks that make up the ensemble. The "vote" for each network when using the averaging method is based on the comparing the ANN output for each network to the ensemble threshold θ for each individual model.

Averaging helps adjust the ensemble decision threshold to accommodate imbalances in the data set. As a consequence, it turns out that adjusting thresholds to the geometric mean of the original average threshold and α/(α + β) is more appropriate than using the mean of ε(k) itself. It also gives some networks more influence over the predictive outcome than others for a particular compound: a few networks whose outputs lie far below the threshold can have more influence on the ultimate prediction than a larger number of networks whose outputs lie just above the threshold - or vice versa.

Our experience has shown that combining averaging with shifting the weighting of positive and negative observations from the 50:50 default can be problematic, especially when data sets are very unbalanced. We recommend that Refine threshold be turned OFF in such situations.

In CFA model construction, on the other hand, each network gets one vote regardless of how strongly it affects the average output. This difference broadens the spread of predictions and errors around the nominal voting threshold for the averaging method, which is defined by θ × K. The discontinuity in uncertainty around the voting threshold is generally "softened" as a result (Clark 2014).