The training pool is the subset of compounds that will be used to train the ANN, SVM, KPLS, or MLR models. The test set is a group of compounds that are set aside before training begins, and is not involved in the training in any way. Once a model has been fully trained, the test set is used to assess the model's predictive performance. This provides an objective estimate of the overall predictive accuracy of each ensemble that the program generates.
Partitioning the data into a training pool and a test set necessarily involves trade-offs. On one hand, maximizing the size and diversity of the training pool provides as much information as possible to the model building. On the other hand, maximizing the size and diversity of the test set gives the most robust measure of how well the model is going to perform when applied to compounds outside the training set.
For ANN, SVM and KPLS models, the training pool is further subdivided into a training set and a verification set, with each individual model in the ensemble "seeing" a different split ("fold"). The verification set is involved in building individual models, albeit only indirectly: for early stopping in ANNEs and for parameter setting in SVM, KPLS, and some ANNE classification models. More critically, predictive performance on the verification set is used to determine which models get included in the final ensemble. Hence verification statistics are a better indicator of what to expect for compounds from outside the data set than training set statistics, but not as good as test set performance statistics.
The Minimum test set size field (1) allows you to set the minimum test set size as a percentage of the total number of observations. This is a minimum; Kohonen mapping is focused on selecting an informative training pool and generally produces a test set that is larger - often considerably larger - than the specified minimum.
Five methods are provided for splitting a data set into a training pool and a test set. Using a Kohonen Self-Organizing Map (2) is the method we generally prefer, but it can be slow for large data sets. In fact, if you are working with more than 10000 data points and you opt to use Kohonen mapping, you will be prompted to confirm that you really want to do so. The Reuse an existing Kohonen map option saves time - and also lets you keep your test set the same while varying your random number seed; it will be selected by default if you open a project that used Kohonen mapping to select the test set. Automatically generated maps for a data set of N compounds use an m×m Kohonen map, where m is the largest integer not greater than the square root of N. The Manual with... option lets you specify a different size for the map if you wish to override the default setting. A smaller map will tend to collapse singletons together and reduce the size of the training pool, whereas a larger map will tend to populate more cells and increase the size of the training pool.
If Kohonen mapping fails to find the minimum number of test set points, enough additional points are selected from the training pool at random and moved to the test set to meet the minimum test set size requirement.
Utilizing K-means clustering (3) (Lloyd 1982) has many of the advantages of Kohonen mapping but is better suited to large data sets.
If Stratified sampling... (4) is selected, the data is sorted using the specified attribute and partitioned as evenly as possible into N×(min_test/100) quantiles, where N is the total number of observations and min_test is the specified Minimum test set size (1). An observation (the Median or one drawn at Random) is then taken from each quantile and put into the test set; remaining observations are assigned to the training pool. Sorting is typically done on the dependent variable for regression models. Doing so on classification models will result in an independent random sample from each class. Sorting on molecular weight or N_Atoms is useful for avoiding artifacts from "lumpy" size distributions.
Observations can also be partitioned at Random (5) or Manually (6). Selecting the latter will replace the Kohonen panel on the right side of the dialog with the sorted values from the selected attribute - here, the compound Identifier. Selected values are assigned to the test set. Note that manual selection overrides the Minimum test set size setting.