3.2.7. Uncertainty and Model Selection
Associating uncertainty estimates with a classification or regression model makes it more useful because doing so lets you distinguish between predictions that are very reliable and those that are less so. This has important implications for which follow-up actions are appropriate: a positive toxicity prediction with an associated confidence of 95%, for example, suggests that if a negative in vitro test result was obtained for the compound, that test should be re-run. Conversely, the combination of a confident negative prediction with a positive test result would call for a retest along with a careful analysis of the compound to make sure the structure is correct and that it has not degraded.
A classification for which the estimated confidence lies below 55-60%, on the other hand, should be considered "indeterminate" and not in contradiction of a nominally discordant test result or of a confident prediction from a complementary model. Confidence values as low as 40% are not necessarily inappropriate, however. They typically arise in areas where the cost of one type of error is much higher than the other or in cases where there are systematic categorical biases in the input data.
Both factors contribute to the relatively high number of low confidence positive predictions from mutagenicity models. On the one hand, classification thresholds for those models must be set low enough to reach a sensitivity of at least 0.80. That is the minimum acceptable to regulatory agencies, but using it necessarily increases the number of false positives. On the other hand, many of the nominal "positives" in the data upon which these models are built reflect tests of industrial grade chemicals carried out for the EPA or NIHS. In many of those cases, these results are due to one or more contaminants rather than the molecule whose structure was input for the test sample. Those are not, strictly speaking, functional errors but they affect the predictivity of the model as though they were.
Regression uncertainties are also useful in validating model performance, since they allow you to distinguish marginal disagreements with literature values from substantive ones. If the estimated standard deviations of the predictive error represented by the error bars in performance plots overlap, the difference in the predictions is unlikely to be significant. Conversely, other things being more or less equal, it makes sense to prioritize the compound predicted to have better activity over one predicted to perform more poorly if their error bars do not overlap.
The failure to get a good uncertainty model for either a classification or a regression model may indicate overtraining or a bias in test set selection. In such cases, it is best to use an alternative model with somewhat inferior nominal performance statistics if one is available. Exceptional performance statistics associated with a weak uncertainty model are probably too good to be true.