3.1. How to Build a Good Model

3.1. How to Build a Good Model

Part 2 of this manual deals with ADMET Properties, and its first chapter (2.1) addresses the question, "What makes a good model?" The answer is, of course, relevant to model construction as well and that chapter is recommended reading for ADMET Modeler users. Nonetheless, the focus there is primarily on the global property models offered in ADMET Predictor. ADMET Modeler users, on the other hand, are more likely to be concerned with building local models using proprietary data related to biological activity. Though the emphases are slightly different, the main steps are the same for both kinds of models:

1. Find a good endpoint to work on.

2. Get good data.

3. Curate, curate, and then curate some more.

4. Prepare for validation.

5. Select a model type to build.

6. Explore the model space.

7. Pick your final model.

8. Evaluate external model performance.

9. Put the model to work.

This chapter provides an overview of the model building process. Though it alludes to specific ADMET Predictor functionality that is useful at one step or another, it is not intended to be either self-contained or exhaustive. Chapters 3.2-3.3 focus on the mechanics of steps 4 through 8; for a deeper understanding of that part of the process, the reader is referred to Modeler Theory and Details. The preliminary considerations are critical, however, if you want to finish up with a robust and useful model.

Step 1: Finding a Good Endpoint

Your dependent variable should be well-defined and of practical use. It is important, for example, to have data available that covers a dynamic range that includes most typical values you hope to predict and that the measurements involved will be made under similar conditions. It is easy to fall into the trap of building a model just because lots of data is available, even though much of that data is irrelevant or inappropriate.

The structural class generation tool in the MedChem Studio module was originally developed for evaluating high-throughput screening (HTS) data. It is well-suited for qualitatively assessing how consistent endpoint values are within structural classes as well as revealing differences - i.e., broad structure-activity and structure-property relationships as well as pairwise relationships such as "activity cliffs" (see Chapter 4.4). Such an examination of the data is critical when trying to combine data sets and when dealing with data from cell-based and phenotypic screens.

The majority of compounds exhibit no detectable effect by or on enzyme or receptor activities. Such inert compounds do not contribute to regression models and so fall outside their applicability domains; the prediction is implicitly contingent on the compound actually being a substrate or inhibitor. Creating a classification filter to distinguish actives from inactives can be challenging but should always be considered in such cases.

Step 2: Getting Good Data

Consistency is important but there is statistical power in numbers. Pooling data from diverse data sets risks adding noise and undue bias along with information. Only using data from a single laboratory almost always increases precision but generally comes with some loss in generalizability. This is a point where the criteria for global commercial models diverge from those for local "in house models." The former needs to place a greater emphasis on covering a large area of chemical space but, if it is going to be useful across a range of users, also needs to be robust against modest variations in how the endpoint data was obtained.

Caco-2 bilayer permeability is an important example of an endpoint that is not robust. The cell line is heterogeneous in origin and very sensitive to growth conditions, which makes results from this assay notoriously variable from laboratory to laboratory. This variability has kept Simulations Plus from building a global model for Caco-2 permeability, but a local model can be of great internal value even though its scope is restricted to a company's own analytical methods and chemistry.

Global models provide a valuable complement to such local models, but it is difficult for an individual company to justify the investment in curation necessary to create global models as good as those offered by ADMET Predictor. When there is a desire to adjust one of our global property models to better reflect the particular chemistry space in which a company works, consider building a local model of the difference between your internal measurements and our predictions. A prediction for a new compound can then be obtained by adding the predicted difference back to ADMET Predictor's estimate.

Step 3: Curation, Curation and More Curation

ADMET Modeler operates on single molecules, so the first step in curating data is to extract the active moiety from salts and other multicomponent compounds. This can be accomplished [automatically ]or manually. Following that, it is good practice to standardize substructural representations (e.g., how nitro groups are drawn).

Removing duplicate entries typically follows. ADMET Predictor will identify duplicate entries based on shared name or structure or based on tautomeric equivalence. Usually these are matters of simple redundancy and they are readily resolved by averaging endpoint values or their geometric mean, if you expect to use the log as the dependent variable. For classification models, eliminating all but one example that represents a consensus of the replicates serves the same purpose. It is not at all unknown, however, to encounter cases where the same name is attached to structures that are not tautomers of each other. This happens fairly often with natural products, metabolites, and compounds where one or more words from a multipart name have been dropped. Esters are particularly prone to getting confused with salts. Such errors can require considerable detective work to sort out (Waldman 2015).

ADMET Predictor allows you to standardize tautomers. Standardization uses a rule-based system that strikes a balance between consistency and accuracy. The microstate analysis tool available with the S+pKa model can be used to resolve most tautomeric ambiguities that are not handled automatically.

Step 4: Preparing for validation.

It may seem that validation is something that can be postponed until after a model has been built, but that is not the case. The key step in validation is setting aside a test set before model construction begins and to do so without allowing the endpoint data to bias how compounds get allocated between the training pool and the test set. This set cannot be used in any way to influence the training of the ensembles. It is only to be used at the end of the process to compare models (typically ensemble models) to see which ones provide the best prediction for compounds not used to train them.

Our preferred approach is to construct the training pool by drawing representative points from clusters generated by Kohonen mapping (Section 3.4.2) or, for large data sets, by K-means clustering (Lloyd 1982). Remaining compounds are assigned to the test set. Sometimes, however, these methods yield better performance statistics for the test set than for the training set, which is suspicious. In that event, it may be better to try several random partitions of the data set. Stratified sampling, where the rows are sorted by the dependent variable and each quantile is sampled, is an alternative strategy that has the advantage of providing uniform coverage.

Step5: Selecting a Model Type to Build

ADMET Modeler supports several advanced methods for building regression and classification models, but our commercial models are artificial neural net ensembles (ANNEs). Voting is the default method for determining the consensus output from classification ensembles, although averaging is often a better alternative for unbalanced data sets - i.e., ones in which the number of positives is much smaller or much larger than the number of negatives. We recommend including confidence analysis (CFA) threshold refinement, especially when the voting method is used.

Support vector machines (SVM), kernel partial least squares (KPLS), and multiple linear regression methods are also supported, though they are generally more prone to overtraining than ANNE, at least in our hands. This may reflect our use of the early stopping technique, which is currently only available for ANNE.

Starting with version 13, ADMET Predictor is capable of running eXtreme Gradient Boosting (XGBoost) ensemble models.

Step 6: Exploring the Model Space

It is possible to run a single huge ANNE experiment using input and neuron step sizes of 1, then to pick the best performing model from among hundreds. That is generally not a good strategy.

Instead, you should explore the model space thoroughly enough to be sure that you haven't missed a golden opportunity but not so thoroughly that you get lost. If you look hard enough for the best-looking model possible, you will eventually find it. The odds are good, however, that such a model is fortuitous and will not stand up well in general use if it is that hard to find.

Step 7: Picking Your Model.

Picking the "best" model is a challenging multiobjective optimization problem that is sensitive to the needs and desires of the modeler and end users. We provide an automated ensemble selection tool that applies multiple criteria simultaneously.

It is not uncommon to see models that exhibit outstanding performance on the training pool but yield mediocre to relatively poor test set statistics. Such models are usually overtrained and, as the test set performance suggests, do not generalize well. Our experience has shown that models that perform about equally well on the training pool and test sets should be preferred.

The early stopping technique we use with ANNEs helps avoid the generation of overtrained models and helps take the interplay between model complexity and quality of fit into account. All other things being equal, a model with more inputs or interaction terms (e.g., neurons for ANNs) is less desirable than a simpler one.

ADMET Modeler provides another, complementary tool to aid in assessment of ANNE classification models: confidence analysis (CFA). Besides providing confidence measures for individual predictions based on the degree of agreement among networks in the ensemble, it provides an independent check on whether the nominal performance statistics may be unduly optimistic (CFA in Model Selection).

Step 8: Evaluate External Model Performance.

If at all possible, it is a good idea to apply your final model to a totally external validation set to see how generalizable the predictions are. Keep in mind that literature values or categorizations may themselves be incorrect. If drawn from literature compilations, they may also have been determined under different conditions. Nonetheless, if the performance statistics obtained differ markedly from those seen for your test set, it is cause for concern.

Interpretation is also complicated if the balance between positives and negatives in the external validation set is very different from that for the data set used to build the model. Published data sets are often artificially balanced because many modeling methods do not handle class imbalances well. Fortunately, the positive and negative outputs in our ANNE models are counterweighted (by default) based on class sizes so that the relative expected cost of positive and negative errors is the same. Sensitivity and specificity tend to be naturally balanced as a result, even when the data set itself is not balanced.

Note that when the number of positive and negative examples in a data set is the same, maximizing Youden's index will minimize the false rate; this is not true for unbalanced data sets.

Step 9: Putting the Model to Work

ADMET Modeler is designed to make building reliable models straightforward, but the overarching goal is to build useful models. That was one of the driving forces behind pulling advanced chemistry tools into ADMET Predictor from MedChem Studio. These include R-table analysis to visualize which parts of chemistry space are covered by existing analogs as well as transformation and combinatorial explosion tools for populating unfilled regions of chemistry space that lie nearby. Matched molecular pair analysis and an interactive sketcher (MedChem Designer) are also available to help guide analog design and synthesis.