3.4.5. Uncertainty Analysis

Being able to quantify predictive uncertainty is critical for the effective use of QSARs in regulatory decision-making (Kruhlak 2011). The ability to identify particularly reliable and unreliable predictions would be useful, as would being able to confidently assess the statistical significance of the difference between pairs of predictions. Research to date has focused on analyzing overall predictivity, but the reliability of predictive classifications for individual structures has not been as thoroughly explored.

ADMET Predictor has reported aggregate performance statistics across training pools and test sets for its models for many years. The key statistics for classification models are sensitivity, specificity and MCC, whereas the root mean squared error (RMSE) and mean absolute error (MAE) are more important for regression models. In averaging predictive uncertainty across all predictions, these measures implicitly assume that all uncertainties are the same, which is rarely if ever completely true (Beck 2000). For one thing, kinds of compounds that are better represented in the training pool are likely to be more accurately predicted. It is hard to know exactly what "represented" means in this context, however, and how much difference it makes for any particular compound. Existing methods for assessing uncertainty are complex and typically can only use data from the test set (e.g., Clark 2009), which limits their usefulness. In the absence of other information, however, such aggregate statistics are the best one can do.

We do have other information for the ensemble models in ADMET Predictor, however. In particular, the variation in individual network predictions provides useful insight into the uncertainty of the ensemble output. When the networks in a well-trained ensemble all agree on a prediction it is likely to be correct, regardless of the nature of the model. Conversely, the odds of a compound being misclassified or that a quantitative prediction is substantially different from the experimental value are higher when the network outputs disagree. Moreover, the network outputs are more likely to disagree if the combination of input descriptor values is very different from anything present in the data set, which provides a useful complement to our range-based applicability domain determination.

It is reasonable to expect that predictive uncertainty and ensemble consistency are monotonically related to each other; the same holds true for the percentage of agreement and predictive confidence. In practice things turn out to be more complicated than has generally been appreciated, however. Researchers at Simulations Plus, Inc., have taken a unique but very practical approach to relating the uncertainty of an ensemble prediction to the spread of the network outputs. The relationship has the same general form for each model, but differs in its parametric details.

The confidence analysis methods used for classification and regression in ADMET Predictor differ somewhat in detail, but both grew out of a few shared insights about predictions obtained from well-trained ensemble models:

Predictions for different compounds are likely to have different uncertainties, i.e., they will be heteroscedastic.
The individual network predictions are correlated across compounds rather than being statistically independent.

The individual networks in an ANNE do start out pretty independent, since the training pool observations are partitioned between the training and verification sets by random shuffle and each network's weights are initialized to random values. Subsequent optimization of each network's weights, however, drives their outputs towards more or less similar values as each network learns the underlying patterns in the data. If that fails to happen, the consolidated ensemble output will usually not be an accurate prediction of the classification or property value and the model will be discarded.

The lack of independence suggests that the uncertainty statistics involved will follow generalized, typically overdisperse distributions of the kind more often encountered in biometrics than those encountered for the independent, uniformly distributed samples typical of most chemical analyses.

Finally, Bayes theorem states that the probability p that an event y will occur given that event x has occurred is given by:

p(y|x) = p(y)×p(x|y)/p(x)

where p(y|x) ("probability of y given x") is the contingent probability. This relationship leads to similar relationships for both classification and regression uncertainties:

u(pred) = (mean err) × g(err|prop )/f(prop)

where u is the predictive uncertainty, pred is the predicted value, err is the error and prop is some property of the ensemble. Estimating uncertainty accurately across a range of ensemble predictions requires knowing the general form of the contingent probability distribution g of the error across prop and the general form of the determining variable distribution f. It turns out that beta binomial distributions work well for classification model uncertainty and that gamma distributions work well for regression model uncertainty when the ensemble properties are number of positive network outputs ("votes") and the standard deviation of the predictions (s_pred), respectively.