Mallows's Cp addresses the issue of overfitting, in which model selection statistics such as the residual sum of squares always get smaller as more variables are added to a model. Thus, if we aim to select the model giving the smallest residual sum of squares, the model including all variables would always be selected. Instead, the Cp statistic calculated on a sample of data estimates the sum squared prediction error (SSPE) as its population target
where is the fitted value from the regression model for the ith case, E(Yi | Xi) is the expected value for the ith case, and σ2 is the error variance (assumed constant across the cases). The mean squared prediction error (MSPE) will not automatically get smaller as more variables are added. The optimum model under this criterion is a compromise influenced by the sample size, the effect sizes of the different predictors, and the degree of collinearity between them.
If Pregressors are selected from a set of K > P, the Cp statistic for that particular set of regressors is defined as:
An alternate version of Cp can also be defined as:[5]
where
RSS is the residual sum of squares on a training set of data
p is the number of predictors
and refers to an estimate of the variance associated with each response in the linear model (estimated on a model containing all predictors)
Note that this version of the Cp does not give equivalent values to the earlier version, but the model with the smallest Cp from this definition will also be the same model with the smallest Cp from the earlier definition.
Limitations
The Cp criterion suffers from two main limitations[6]
the Cp approximation is only valid for large sample size;
the Cp cannot handle complex collections of models as in the variable selection (or feature selection) problem.[6]
Practical use
The Cp statistic is often used as a stopping rule for various forms of stepwise regression. Mallows proposed the statistic as a criterion for selecting among many alternative subset regressions. Under a model not suffering from appreciable lack of fit (bias), Cp has expectation nearly equal to P; otherwise the expectation is roughly P plus a positive bias term. Nevertheless, even though it has expectation greater than or equal to P, there is nothing to prevent Cp < P or even Cp < 0 in extreme cases. It is suggested that one should choose a subset that has Cp approaching P,[7] from above, for a list of subsets ordered by increasing P. In practice, the positive bias can be adjusted for by selecting a model from the ordered list of subsets, such that Cp < 2P.
Since the sample-based Cp statistic is an estimate of the MSPE, using Cp for model selection does not completely guard against overfitting. For instance, it is possible that the selected model will be one in which the sample Cp was a particularly severe underestimate of the MSPE.
Model selection statistics such as Cp are generally not used blindly, but rather information about the field of application, the intended use of the model, and any known biases in the data are taken into account in the process of model selection.
^Gilmour, Steven G. (1996). "The interpretation of Mallows's Cp-statistic". Journal of the Royal Statistical Society, Series D. 45 (1): 49–56. JSTOR2348411.
^Boisbunon, Aurélie; Canu, Stephane; Fourdrinier, Dominique; Strawderman, William; Wells, Martin T. (2013). "AIC, Cp and estimators of loss for elliptically symmetric distributions". arXiv:1308.2766 [math.ST].
Judge, George G.; Griffiths, William E.; Hill, R. Carter; Lee, Tsoung-Chao (1980). The Theory and Practice of Econometrics. New York: Wiley. pp. 417–423. ISBN978-0-471-05938-7.