---
title: "Cross-validation of regression models"
author: "John Fox and Georges Monette"
date: "`r Sys.Date()`"
package: cv
output: 
  rmarkdown::html_vignette:
  fig_caption: yes
bibliography: ["cv.bib"]
csl: apa.csl
vignette: >
  %\VignetteIndexEntry{Cross-validation of regression models}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  message = TRUE,
  warning = TRUE,
  fig.align = "center",
  fig.height = 6,
  fig.width = 7,
  fig.path = "fig/",
  dev = "png",
  comment = "#>" #,
)

# save some typing
knitr::set_alias(w = "fig.width",
                 h = "fig.height",
                 cap = "fig.cap")

# colorize text: use inline as `r colorize(text, color)`
colorize <- function(x, color) {
  if (knitr::is_latex_output()) {
    sprintf("\\textcolor{%s}{%s}", color, x)
  } else if (knitr::is_html_output()) {
    sprintf("<span style='color: %s;'>%s</span>", color,
      x)
  } else x
}


.opts <- options(digits = 5)
```

## Cross-validation

Cross-validation (CV) is an essentially simple and intuitively reasonable approach to estimating the predictive accuracy of regression models. CV is developed in many standard sources on regression modeling and "machine learning"---we particularly recommend @JamesEtAl:2021 [Secs. 5.1, 5.3]---and so we will describe the method only briefly here before taking up computational issues and some examples.

Validating research by replication on independently collected data is a common scientific norm. Emulating this process in a single study by data-division is less common: The data are randomly divided into two, possibly equal-size, parts; the first part is used to develop and fit a statistical model; and then the second part is used to assess the adequacy of the model fit to the first part of the data. Data-division, however, suffers from two problems: (1) Dividing the data decreases the sample size and thus increases sampling error; and (2), even more disconcertingly, particularly in smaller samples, the results can vary substantially based on the random division of the data: See @Harrell:2015 [Sec. 5.3] for this and other remarks about data-division and cross-validation.

Cross-validation speaks to both of these issues. In CV, the data are randomly divided as equally as possible into several, say $k$, parts, called "folds." The statistical model is fit $k$ times, leaving each fold out in turn. Each fitted model is then used to predict the response variable for the omitted fold, computing some CV criterion or "cost" measure, such as the mean-squared error of prediction. The CV criterion is then averaged over the $k$ folds. In the extreme $k = n$, the number of cases in the data, thus omitting individual cases and refitting the model $n$ times---a procedure termed "leave-one-out (LOO) cross-validation."

Because the $k$ models are each fit to $n - 1$ cases, LOO CV produces a nearly unbiased estimate of prediction error. The $n$ regression models are highly statistical dependent, however, based as they are on nearly the same data, and so the resulting estimate of prediction error has relatively large variance. In contrast, estimated prediction error for $k$-fold CV with $k = 5$ or $10$ (commonly employed choices) are somewhat biased but have smaller variance. It is also possible to correct $k$-fold CV for bias (see below).

## Examples

### Polynomial regression for the `Auto` data

The data for this example are drawn from the **ISLR2** package for R, associated with @JamesEtAl:2021, and the presentation here is close (though not identical) to that in the original source [@JamesEtAl:2021 Secs. 5.1, 5.3], and it demonstrates the use of the `cv()` function in the **cv** package.[^boot] 

[^boot]: @JamesEtAl:2021 use the `cv.glm()` function in the **boot** package [@CantyRipley2022; @DavisonHinkley1997]. Despite its name, `cv.glm()` is an independent function and not a method of a `cv()` generic function.

The `Auto` dataset contains information about 392 cars:

```{r Auto}
data("Auto", package="ISLR2")
head(Auto)
dim(Auto)
```
With the exception of `origin` (which we don't use here), these variables are largely self-explanatory, except possibly for units of measurement: for details see `help("Auto", package="ISLR2")`.

We'll focus here on the relationship of `mpg` (miles per gallon) to `horsepower`, as displayed in the following scatterplot:

```{r mpg-horsepower-scatterplot}
#| out.width = "100%",
#| fig.height = 6,
#| fig.cap = "`mpg` vs `horsepower` for the `Auto` data"
plot(mpg ~ horsepower, data=Auto)
```
The relationship between the two variables is monotone, decreasing, and nonlinear. Following @JamesEtAl:2021, we'll consider approximating the relationship by a polynomial regression, with the degree of the polynomial $p$ ranging from 1 (a linear regression) to 10.[^log-trans] Polynomial fits  for $p = 1$ to $5$ are shown in the following figure:

[^log-trans]: Although it serves to illustrate the use of CV, a polynomial is probably not the best choice here. Consider, for example the scatterplot for log-transformed `mpg` and `horsepower`, produced by `plot(mpg ~ horsepower, data=Auto, log="xy")` (execution of which is left to the reader).

```{r mpg-horsepower-scatterplot-polynomials}
#| out.width = "100%",
#| fig.height = 5,
#| fig.cap = "`mpg` vs `horsepower` for the `Auto` data"
plot(mpg ~ horsepower, data=Auto)
horsepower <- with(Auto, 
                   seq(min(horsepower), max(horsepower), 
                       length=1000))
for (p in 1:5){
  m <- lm(mpg ~ poly(horsepower,p), data=Auto)
  mpg <- predict(m, newdata=data.frame(horsepower=horsepower))
  lines(horsepower, mpg, col=p + 1, lty=p, lwd=2)
}
legend("topright", legend=1:5, col=2:6, lty=1:5, lwd=2,
       title="Degree", inset=0.02)
```
The linear fit is clearly inappropriate; the fits for $p = 2$ (quadratic) through $4$ are very similar; and the fit for $p = 5$ probably over-fits the data by chasing one or two relatively high `mpg` values at the right.

The following graph shows two measures of estimated squared error as a function of polynomial-regression degree: The mean-squared error (MSE), defined as $\mathsf{MSE} = \sum (y_i - \widehat{y}_i)^2/n$, and the usual unbiased estimated error variance, defined as $\widehat{\sigma}^2 = \sum (y_i - \widehat{y}_i)^2/(n - p - 1)$. The former necessarily declines with $p$ (or, more strictly, can't increase with $p$), while the latter gets slightly larger for the largest values of $p$, with the "best" value, by a small margin, for $p = 7$.

```{r mpg-horsepower-MSE-se2}
#| out.width = "100%",
#| fig.height = 5,
#| fig.cap = "Estimated squared error as a function of polynomial degree, $p$"
library("cv") # for mse() and other functions

se <- mse <- numeric(10)
for (p in 1:10){
  m <- lm(mpg ~ poly(horsepower, p), data=Auto)
  mse[p] <- mse(Auto$mpg, fitted(m))
  se[p] <- summary(m)$sigma
}

plot(c(1, 10), range(mse, se^2), type="n",
     xlab="Degree of polynomial, p",
     ylab="Estimated Squared Error")
lines(1:10, mse, lwd=2, lty=1, col=2, pch=16, type="b")
lines(1:10, se^2, lwd=2, lty=2, col=3, pch=17, type="b")
legend("topright", inset=0.02,
       legend=c(expression(hat(sigma)^2), "MSE"),
       lwd=2, lty=2:1, col=3:2, pch=17:16)
```
The code for this graph uses the `mse()` function from the **cv** package to compute the MSE for each fit.

#### Using `cv()`

The generic `cv()` function has an `"lm"` method, which by default performs $k = 10$-fold CV:
```{r cv-lm-1}
m.auto <- lm(mpg ~ poly(horsepower, 2), data=Auto)
summary(m.auto)

cv(m.auto)
```
The `"lm"` method by default uses `mse()` as the CV criterion and the Woodbury matrix identity to update the regression with each fold deleted without having literally to refit the model. Computational details are discussed in the final section of this vignette. The function reports the CV estimate of MSE, a biased-adjusted estimate of the MSE (the bias adjustment is explained in the final section), and the MSE is also computed for the original, full-sample regression. Because the division of the data into 10 folds is random, `cv()` explicitly (randomly) generates and saves a seed for R's pseudo-random number generator, to make the results replicable. The user can also specify the seed directly via the `seed` argument to `cv()`.

To perform LOO CV, we can set the `k` argument to `cv()` to the number of cases in the data, here `k=392`, or, more conveniently, to `k="loo"` or `k="n"`:

```{r cv.lm-2`}
cv(m.auto, k="loo")
```
For LOO CV of a linear model, `cv()` by default uses the hatvalues from the model fit to the full data for the LOO updates, and reports only the CV estimate of MSE. Alternative methods are to use the Woodbury matrix identity or the "naive" approach of literally refitting the model with each case omitted. All three methods produce exact results for a linear model (within the precision of floating-point computations):
```{r cv.lm-3}
cv(m.auto, k="loo", method="naive")

cv(m.auto, k="loo", method="Woodbury")
```
The `"naive"` and `"Woodbury"` methods also return the bias-adjusted estimate of MSE and the full-sample MSE, but bias isn't an issue for LOO CV.

This is a small regression problem and all three computational approaches are essentially instantaneous, but it is still of interest to investigate their relative speed. In this comparison, we include the `cv.glm()` function from the **boot** package, which takes the naive approach, and for which we have to fit the linear model as an equivalent Gaussian GLM. We use the `microbenchmark()` function from the package of the same name for the timings [@Mersmann:2023]:
```{r cv.lm.timings, cache=TRUE}
m.auto.glm <- glm(mpg ~ poly(horsepower, 2), data=Auto)
boot::cv.glm(Auto, m.auto.glm)$delta

microbenchmark::microbenchmark(
  hatvalues = cv(m.auto, k="loo"),
  Woodbury = cv(m.auto, k="loo", method="Woodbury"),
  naive = cv(m.auto, k="loo", method="naive"),
  cv.glm = boot::cv.glm(Auto, m.auto.glm),
  times=10
)
```
On our computer, using the hatvalues is about an order of magnitude faster than employing Woodbury matrix updates, and more than two orders of magnitude faster than refitting the model.[^microbenchmark]

[^microbenchmark]: Out of impatience, we asked `microbenchmark()` to execute each command only 10 times rather than the default 100. With the exception of the last columns, the output is self-explanatory. The last column shows which methods have average timings that are statistically distinguishable. Because of the small number of repetitions (i.e., 10), the `"hatvalues"` and `"Woodbury"` methods aren't distinguishable, but the difference between these methods persists when we perform more repetitions---we invite the reader to redo this computation with the default `times=100` repetitions.

#### Comparing Competing Models

The `cv()` function also has a method that can be applied to a list of regression models for the same data, composed using the `models()` function. For $k$-fold CV, the same folds are used for the competing models, which reduces random error in their comparison. This result can also be obtained by specifying a common seed for R's random-number generator while applying `cv()` separately to each model, but employing a list of models is more convenient for both $k$-fold and LOO CV (where there is no random component to the composition of the $n$ folds).

We illustrate with the polynomial regression models of varying degree for the `Auto` data (discussed previously), beginning by fitting and saving the 10 models:
```{r polyomial-models}
for (p in 1:10){
  assign(paste0("m.", p),
         lm(mpg ~ poly(horsepower, p), data=Auto))
}
objects(pattern="m\\.[0-9]")
summary(m.2) # for example, the quadratic fit
```
We then apply `cv()` to the list of 10 models (the `data` argument is required):
```{r polynomial-regression-CV}
# 10-fold CV
cv.auto.10 <- cv(models(m.1, m.2, m.3, m.4, m.5,
                     m.6, m.7, m.8, m.9, m.10),
              data=Auto, seed=2120)
cv.auto.10[1:2] # for the linear and quadratic models

# LOO CV
cv.auto.loo <- cv(models(m.1, m.2, m.3, m.4, m.5,
                        m.6, m.7, m.8, m.9, m.10),
                 data=Auto, k="loo")
cv.auto.loo[1:2] # linear and quadratic models
```
Because we didn't supply names for the models in the calls to the `models()` function, the names `model.1`, `model.2`, etc., are supplied by the function.

Finally, we extract and graph the adjusted MSEs for $10$-fold CV and the MSEs for LOO CV:
```{r polynomial-regression-CV-graph}
#| out.width = "100%",
#| fig.height = 5,
#| fig.cap = "Cross-validated 10-fold and LOO MSE as a function of polynomial degree, $p$"
cv.mse.10 <- sapply(cv.auto.10, function(x) x[["adj CV crit"]])
cv.mse.loo <- sapply(cv.auto.loo, function(x) x[["CV crit"]])
plot(c(1, 10), range(cv.mse.10, cv.mse.loo), type="n",
     xlab="Degree of polynomial, p",
     ylab="Cross-Validated MSE")
lines(1:10, cv.mse.10, lwd=2, lty=1, col=2, pch=16, type="b")
lines(1:10, cv.mse.loo, lwd=2, lty=2, col=3, pch=17, type="b")
legend("topright", inset=0.02,
       legend=c("10-Fold CV", "LOO CV"),
       lwd=2, lty=2:1, col=3:2, pch=17:16)
```

Alternatively, we can use the `plot()` method for `"cvModList"` objects to compare the models, though with separate graphs for 10-fold and LOO CV:

```{r, polynomial-regression-CV-graph-2, fig.show="hold"}
#| out.width = "45%",
#| fig.height = 6,
#| fig.cap = "Cross-validated 10-fold and LOO MSE as a function of polynomial degree, $p$"
plot(cv.auto.10, main="Polynomial Regressions, 10-Fold CV",
     axis.args=list(labels=1:10), xlab="Degree of Polynomial, p")
plot(cv.auto.loo, main="Polynomial Regressions, LOO CV",
     axis.args=list(labels=1:10), xlab="Degree of Polynomial, p")
```

In this example, 10-fold and LOO CV produce generally similar results, and also results that are similar to those produced by the estimated error variance $\widehat{\sigma}^2$ for each model, reported above (except for the highest-degree polynomials, where the CV results more clearly suggest over-fitting).

### Logistic regression for the `Mroz` data

The `Mroz` data set from the **carData** package [associated with @FoxWeisberg:2019] has been used by several authors to illustrate binary logistic regression; see, in particular @FoxWeisberg:2019. The data were originally drawn from the U.S. Panel Study of Income Dynamics and pertain to married women. Here are a few cases in the data set:
```{r Mroz-data}
data("Mroz", package="carData")
head(Mroz, 3)
tail(Mroz, 3)
```
The response variable in the logistic regression is `lfp`, labor-force participation, a factor coded `"yes"` or `"no"`. The remaining variables are predictors:

* `k5`, number of children 5 years old of younger in the woman's household;
* `k618`, number of children between 6 and 18 years old;
* `age`, in years;
* `wc`, wife's college attendance, `"yes"` or `"no"`;
* `hc`, husband's college attendance;
* `lwg`, the woman's log wage rate if she is employed, or her *imputed* wage rate, if she is not [a variable that @FoxWeisberg:2019 show is problematically defined]; and
* `inc`, family income, in $1000s, exclusive of wife's income.

We use the `glm()` function to fit a binary logistic regression to the `Mroz` data:
```{r Mroz-logistic-regresion}
m.mroz <- glm(lfp ~ ., data=Mroz, family=binomial)
summary(m.mroz)

BayesRule(ifelse(Mroz$lfp == "yes", 1, 0), 
          fitted(m.mroz, type="response"))
```
In addition to the usually summary output for a GLM, we show the result of applying the `BayesRule()` function from the **cv** package to predictions derived from the fitted model. Bayes rule, which predicts a "success" in a binary regression model when the fitted probability of success [i.e., $\phi = \Pr(y = 1)$] is $\widehat{\phi} \ge .5$ and a "failure" if $\widehat{\phi} \lt .5$.[^BayesRule] The first argument to `BayesRule()` is the binary {0, 1} response, and the second argument is the predicted probability of success. `BayesRule()` returns the proportion of predictions that are *in error*, as appropriate for a "cost" function.

[^BayesRule]: `BayesRule()` does some error checking; `BayesRule2()` is similar, but omits the error checking, and so can be faster for large problems.

In this example, the fitted logistic regression incorrectly predicts 31% of the responses; we expect this estimate to be optimistic given that the model is used to "predict" the data to which it is fit.

The `"glm"` method for `cv()` is largely similar to the `"lm"` method, although the default algorithm, selected explicitly by `method="exact"`, refits the model with each fold removed (and is thus equivalent to `method="naive"` for `"lm"` models). For generalized linear models, `method="Woodbury"` or (for LOO CV) `method="hatvalues"` provide approximate results (see the last section of the vignette for details):

```{r cv-Mroz-10-fold}
cv(m.mroz, criterion=BayesRule, seed=248)

cv(m.mroz, criterion=BayesRule, seed=248, method="Woodbury")
```
To ensure that the two methods use the same 10 folds, we specify the seed for R's random-number generator explicitly; here, and as is common in our experience, the `"exact"` and `"Woodbury"` algorithms produce nearly identical results. The CV estimates of prediction error are slightly higher than the estimate based on all of the cases.

Here are results of applying LOO CV to the Mroz model, using both the exact and the approximate methods:
```{r cv-Mroz-LOO}
cv(m.mroz, k="loo", criterion=BayesRule)

cv(m.mroz, k="loo", criterion=BayesRule, method="Woodbury")

cv(m.mroz, k="loo", criterion=BayesRule, method="hatvalues")
```
To the number of decimal digits shown, the three methods produce identical results for this example.

As for linear models, we report some timings for the various `cv()` methods of computation in LOO CV as well as for the `cv.glm()` function from the **boot** package (which, recall, refits the model with each case removed, and thus is comparable to `cv()` with `method="exact"`):
```{r glm.timings, cache=TRUE}
microbenchmark::microbenchmark(
  hatvalues=cv(m.mroz, k="loo", criterion=BayesRule, method="hatvalues"),
  Woodbury=cv(m.mroz, k="loo", criterion=BayesRule, method="Woodbury"),
  exact=cv(m.mroz, k="loo", criterion=BayesRule),
  cv.glm=boot::cv.glm(Mroz, m.mroz,
               cost=BayesRule),
  times=10)
```
There is a substantial time penalty associated with exact computations.

## Cross-validating mixed-effects models

The fundamental analogy for cross-validation is to the collection of new data. That is, predicting the response in each fold from the model fit to data in the other folds is like using the model fit to all of the data to predict the response for new cases from the values of the predictors for those new cases. As we explained, the application of this idea to independently sampled cases is straightforward---simply partition the data into random folds of equal size and leave each fold out in turn, or, in the case of LOO CV, simply omit each case in turn.

In contrast, mixed-effects models are fit to *dependent* data, in which cases as clustered, such as hierarchical data, where the clusters comprise higher-level units (e.g., students clustered in schools), or longitudinal data, where the clusters are individuals and the cases repeated observations on the individuals over time.[^crossed-effects] 

[^crossed-effects]: There are, however, more complex situations that give rise to so-called *crossed* (rather than *nested*) random effects. For example, consider students within classes within schools. In primary schools, students typically are in a single class, and so classes are nested within schools. In secondary schools, however, students typically take several classes and students who are together in a particular class may not be together in other classes; consequently, random effects based on classes within schools are crossed. Although the `lmer()` function in the **lme4** package is capable of modeling both nested and crossed random effects, the `cv()` methods for mixed models in the **cv** package pertain only to nested random effects. It may be possible to cross-validate mixed models with crossed random effects, but we don't know how to do it.

We can think of two approaches to applying cross-validation to clustered data:

1. Treat CV as analogous to predicting the response for one or more cases in a *newly observed cluster*. In this instance, the folds comprise one or more whole clusters; we refit the model with all of the cases in clusters in the current fold removed; and then we predict the response for the cases in clusters in the current fold. These predictions are based only on fixed effects because the random effects for the omitted clusters are presumably unknown, as they would be for data on cases in newly observed clusters.

2. Treat CV as analogous to predicting the response for a newly observed case in an *existing cluster*. In this instance, the folds comprise one or more individual cases, and the predictions can use both the fixed and random effects.

### Example: The High-School and Beyond data

Following their use by @RaudenbushBryk:2002, data from the 1982 *High School and Beyond* (HSB) survey have become a staple of the literature on mixed-effects models. The HSB data are used by @FoxWeisberg:2019 [Sec. 7.2.2] to illustrate the application of linear mixed models to hierarchical data, and we'll closely follow their example here.

The HSB data are included in the `MathAchieve` and `MathAchSchool` data sets in the **nlme** package  [@PinheiroBates:2000]. `MathAchieve` includes individual-level data on 7185  students in 160 high schools, and `MathAchSchool` includes school-level data:
```{r HSB-data}
data("MathAchieve", package="nlme")
dim(MathAchieve)
head(MathAchieve, 3)
tail(MathAchieve, 3)

data("MathAchSchool", package="nlme")
dim(MathAchSchool)
head(MathAchSchool, 2)
tail(MathAchSchool, 2)
```
The first few students are in school number 1224 and the last few in school 9586. 

We'll use only the `School`, `SES` (students' socioeconomic status), and `MathAch` (their score on a standardized math-achievement test) variables in the `MathAchieve` data set, and `Sector` (`"Catholic"` or `"Public"`) in the `MathAchSchool` data set.

Some data-management is required before fitting a mixed-effects model to the HSB data, for which we use the **dplyr** package [@WickhamEtAl:2023]:
```{r HSB-data-management, cache=TRUE}
library("dplyr")
MathAchieve %>% group_by(School) %>%
  summarize(mean.ses = mean(SES)) -> Temp
Temp <- merge(MathAchSchool, Temp, by="School")
HSB <- merge(Temp[, c("School", "Sector", "mean.ses")],
             MathAchieve[, c("School", "SES", "MathAch")], by="School")
names(HSB) <- tolower(names(HSB))

HSB$cses <- with(HSB, ses - mean.ses)
```
In the process, we created two new school-level variables: `meanses`, which is the average SES for students in each school; and `cses`, which is school-average SES centered at its mean. For details, see @FoxWeisberg:2019 [Sec. 7.2.2].

Still following Fox and Weisberg, we proceed to use the `lmer()` function in the **lme4** package [@BatesEtAl:2015] to fit a mixed model for math achievement to the HSB data:
```{r HSB-lmer, cache=TRUE}
library("lme4")
hsb.lmer <- lmer(mathach ~ mean.ses*cses + sector*cses
                   + (cses | school), data=HSB)
summary(hsb.lmer, correlation=FALSE)
```

We can then cross-validate at the cluster (i.e., school) level,
```{r HSB-lmer-CV-cluster, cache=TRUE}
cv(hsb.lmer, k=10, clusterVariables="school", seed=5240)
```
or at the case (i.e., student) level,
```{r HSB-lmer-CV-case, cache=TRUE}
cv(hsb.lmer, seed=1575)
```
For cluster-level CV, the `clusterVariables` argument tells `cv()` how the clusters are defined. Were there more than one clustering variable, say classes within schools, these would be provided as a character vector of variable names: `clusterVariables = c("school", "class")`. For cluster-level CV, the default is `k = "loo"`, that is, leave one cluster out at a time; we instead specify `k = 10` folds of clusters, each fold therefore comprising $160/10 = 16$ schools. 

If the `clusterVariables` argument is omitted, then case-level CV is employed, with `k = 10` folds as the default, here each with $7185/10 \approx 719$ students. Notice that one of the 10 models refit with a fold removed failed to converge. Convergence problems are common in mixed-effects modeling. The apparent issue here is that an estimated variance component is close to or equal to 0, which is at a boundary of the parameter space. That shouldn't disqualify the fitted model for the kind of prediction required for cross-validation.

There is also a `cv()` method for linear mixed models fit by the `lme()` function in the **nlme** package, and the arguments for `cv()` in this case are the same as for a model fit by `lmer()` or `glmer()`. We illustrate with the mixed model fit to the HSB data:
```{r hsb-lme, cache=TRUE}
library("nlme")
hsb.lme <- lme(mathach ~ mean.ses*cses + sector*cses,
                 random = ~ cses | school, data=HSB,
               control=list(opt="optim"))
summary(hsb.lme)

cv(hsb.lme, k=10, clusterVariables="school", seed=5240)

cv(hsb.lme, seed=1575)
```
We used the same random-number generator seeds as in the previous example cross-validating the model fit by `lmer()`, and so the same folds are employed in both cases.[^optimizer] The estimated covariance components and fixed effects in the summary output differ slightly between the `lmer()` and `lme()` solutions, although both functions seek to maximize the REML criterion. This is, of course, to be expected when different algorithms are used for numerical optimization.  To the precision reported, the cluster-level CV results for the `lmer()` and `lme()` models are identical, while the case-level CV results are very similar but not identical.

[^optimizer]: The observant reader will notice that we set the argument `control=list(opt="optim")` in the call to `lme()`, changing the optimizer employed from the default `"nlminb"`. We did this because with the default optimizer, `lme()` encountered the same convergence issue as `lmer()`, but rather than issuing a warning, `lme()` failed, reporting an error. As it turns out, setting the optimizer to `"optim"` avoids this problem.

### Example: Contrived Hierarchical Data

We introduce an artificial data set that exemplifies aspects of cross-validation particular to hierarchical models. Using this data set, we show that model comparisons employing cluster-based and those employing case-based cross-validation may not agree on a "best" model. Furthermore, commonly used measures of fit, such as mean-squared error, do not necessarily become smaller as models become larger, even when the models are nested, and even when the measure of fit is computed for the whole data set.

Consider a researcher studying improvement in a skill, yodeling, for example, among students enrolled in a four-year yodeling program. The plan is to measure each student's skill level at the beginning of the program and every year thereafter until the end of the program, resulting in five annual measurements for each student. It turns out that yodeling appeals to students of all ages, and students enrolling in the program range in age from 20 to 70. Moreover, participants' untrained yodeling skill is similar at all ages, as is their rate of progress with training. All students complete the four-year program.

The researcher, who has more expertise in yodeling than in modeling, decides to model the response, $y$, yodeling skill, as a function of age, $x$, reasoning that students get older during their stay in the program, and (incorrectly) that age can serve as a proxy for elapsed time. The researcher knows that a mixed model should be used to account for clustering due to the expected similarity of measurements taken from each student.

We start by generating the data, using parameters consistent with the description above and meant to highlight the issues that arise in cross-validating mixed-effects models:[^1]

[^1]: We invite the interested reader to experiment with varying the parameters of our example.

```{r data}
# Parameters:
set.seed(9693) 
Nb <- 100     # number of groups
Nw <- 5       # number of individuals within groups
Bb <- 0       # between-group regression coefficient on group mean
SDre <- 2.0   # between-group SD of random level relative to group mean of x
SDwithin <- 0.5  # within group SD
Bw <- 1          # within group effect of x
Ay <- 10         # intercept for response
Ax <- 20         # starting level of x
Nx <- Nw*10      # number of distinct x values

Data <- data.frame(
  group = factor(rep(1:Nb, each=Nw)),
  x = Ax + rep(1:Nx, length.out = Nw*Nb)
) |>
  within(
    {
      xm  <- ave(x, group, FUN = mean) # within-group mean
      y <- Ay +
        Bb * xm +                    # contextual effect
        Bw * (x - xm) +              # within-group effect
        rnorm(Nb, sd=SDre)[group] +  # random level by group
        rnorm(Nb*Nw, sd=SDwithin)    # random error within groups
    }
  )
```

Here is a scatterplot of the data for a representative group of 10 (without loss of generality, the first 10) of 100 students, showing the 95% concentration ellipse for each cluster:[^2]

[^2]: We find it convenient to use the **lattice** [@Sarkar:2008] and **latticeExtra** [@SarkarAndrews:2022] packages for this and other graphs in this section.

```{r plot1}
#| out.width = "100%",
#| fig.height = 6,
#| fig.cap = "Hierarchical data set, showing the first 10 of 100 students."
library("lattice")
library("latticeExtra")
plot <- xyplot(y ~ x, data=Data[1:Nx, ], group=group,
               ylim=c(4, 16),
               par.settings=list(superpose.symbol=list(pch=1, cex=0.7))) +
    layer(panel.ellipse(..., center.cex=0))
plot # display graph
```

The between-student effect of age is 0 but the within-student effect is 1. Due to the large variation in ages between students, the least-squares regression of yodeling skill on age (for the 500 observations among all 100 students) produces an estimated slope close to 0 (though with a small $p$-value), because the slope is heavily weighted toward the between-student effect:

```{r}
summary(lm(y ~ x, data=Data))
```

The initial mixed-effects model that we fit to the data is a simple random-intercepts model:
```{r include=FALSE, echo=FALSE}
library(lme4) # necessary for some reason to knit vignette in RStudio, harmless otherwise
```
```{r}
# random intercept only:
mod.0 <- lmer(y ~ 1 + (1 | group), Data)
summary(mod.0)
```

We will shortly consider three other, more complex, mixed models; because of data-management considerations, it is convenient to fit them now, but we defer discussion of these models:

```{r}
# effect of x and random intercept:
mod.1 <- lmer(y ~ x + (1 | group), Data)

# effect of x, contextual (student) mean of x, and random intercept:
mod.2 <- lmer(y ~ x + xm + (1 | group), Data)
        # equivalent to y ~ I(x - xm) + xm + (1 | group)

# model generating the data (where Bb = 0)
mod.3 <- lmer(y ~ I(x - xm) + (1 | group), Data)
```

We proceed to obtain predictions from the random-intercept model (`mod.0`) and the other models (`mod.1`, `mod.2`, and `mod.3`) based on fixed effects alone, as would be used for cross-validation based on clusters (i.e., students), and for fixed and random effects---so-called best linear unbiased predictions or BLUPs---as would be used for cross-validation based on cases (i.e., occasions within students):

```{r}
Data <- within(Data, {
  fit_mod0.fe <- predict(mod.0, re.form = ~ 0) # fixed effects only
  fit_mod0.re <- predict(mod.0) # fixed and random effects (BLUPs)
  fit_mod1.fe <- predict(mod.1, re.form = ~ 0)
  fit_mod1.re <- predict(mod.1)
  fit_mod2.fe <- predict(mod.2, re.form = ~ 0)
  fit_mod2.re <- predict(mod.2)
  fit_mod3.fe <- predict(mod.3, re.form = ~ 0)
  fit_mod3.re <- predict(mod.3)
})
```

We then prepare the data for plotting:

```{r}
Data_long <- reshape(Data[1:Nx, ], direction = "long", sep = ".", 
              timevar = "effect", varying = grep("\\.", names(Data[1:Nx, ])))
Data_long$id <- 1:nrow(Data_long)
Data_long <- reshape(Data_long, direction = "long", sep = "_", 
              timevar = "modelcode",  varying = grep("_", names(Data_long)))
Data_long$model <- factor(
  c("~ 1", "~ 1 + x", "~ 1 + x + xm", "~ 1 + I(x - xm)")
  [match(Data_long$modelcode, c("mod0", "mod1", "mod2", "mod3"))]
)
```

Predictions based on the random-intercept model `mod.0` for the first 10 students are shown in the following graph:

```{r plot-fits-mod0}
#| out.width = "100%",
#| fig.height = 6,
#| fig.cap = "Predictions from the random intercept model."
(plot +
  xyplot(fit ~ x, subset(Data_long, modelcode == "mod0" & effect == "fe"),
         groups=group, type="l", lwd=2) +
  xyplot(fit ~ x, subset(Data_long, modelcode == "mod0" &  effect == "re"),
         groups=group, type="l", lwd=2, lty=3)
) |> update(
  main="Model: y ~ 1 + (1 | group)",
  key=list(
    corner=c(0.05, 0.05),
    text=list(c("fixed effects only","fixed and random")),
    lines=list(lty=c(1, 3))))
```

The fixed-effect predictions for the various individuals are identical---the estimated fixed-effects intercept or estimated general mean of $y$---while the BLUPs are the sums of the fixed-effects intercept and the random intercepts, and are only slightly shrunken towards the general mean. Because in our artificial data there is no population relationship between age and skill, the fixed-effect-only predictions and the BLUPs are not very different.

Our next model, `mod.1`, includes a fixed intercept and fixed effect of `x` along with a random intercept:

```{r}
summary(mod.1)
```

Predictions from this model appear in the following graph:

```{r plot-fits-mod1}
#| out.width = "100%",
#| fig.height = 6,
#| fig.cap = "Predictions from the model with random intercepts and $x$ as a fixed-effect predictor."
(plot +
  xyplot(fit ~ x, subset(Data_long, modelcode == "mod1" & effect == "fe"),
         groups=group, type="l", lwd=2) +
  xyplot(fit ~ x, subset(Data_long, modelcode == "mod1" & effect == "re"),
         groups=group, type="l", lwd=2, lty=3)
) |> update(
  main="Model: y ~ 1 + x + (1 | group)",
  ylim=c(-15, 35),
  key=list(
    corner=c(0.95, 0.05),
    text=list(c("fixed effects only","fixed and random")),
    lines=list(lty=c(1, 3))))
```

The BLUPs fit the observed data very closely, but predictions based on the fixed effects alone, with a common intercept and slope for all clusters, are very poor---indeed, much worse than the fixed-effects-only predictions based on the simpler random-intercept model, `mod.0`. We therefore anticipate (and show later in this section) that case-based cross-validation will prefer `mod1` to `mod0`, but that cluster-based cross-validation will prefer `mod0` to `mod1`.

Our third model, `mod.2`, includes the contextual effect of $x$---that is, the cluster mean `xm`---along with $x$ and the intercept in the fixed-effect part of the model, and a random intercept:

```{r}
summary(mod.2)
```

This model is equivalent to fitting `y ~ I(x - xm) + xm + (1 | group)`, which is the model that generated the data once the coefficient of the contextual predictor `xm` is set to 0 (as it is in `mod.3`, discussed below).

Predictions from model `mod.2` appear in the following graph:
```{r plot-fits-mod2}
#| out.width = "100%",
#| fig.height = 6,
#| fig.cap = "Predictors from the model with random intercepts, $x$, and the group (student) mean of $x$ as predictors."
(plot +
  xyplot(fit ~ x, subset(Data_long, modelcode == "mod2" & effect == "fe"),
         groups=group, type="l", lwd=2) +
  xyplot(fit ~ x, subset(Data_long, modelcode == "mod2" & effect == "re"),
         groups=group, type="l", lwd=2, lty=3)
) |> update(
  main="Model: y ~ 1 + x + xm + (1 | group)",
  ylim=c(4, 16),
  key=list(
    corner=c(0.05, 0.05),
    text=list(c("fixed effects only","fixed and random")),
    lines=list(lty=c(1, 3))))
```

Depending on the estimated variance parameters of the model, a mixed model like `mod.2` will apply varying degrees of shrinkage to the random-intercept BLUPs that correspond to variation in the heights of the parallel fitted lines for the individual students. In our contrived data, the `mod.2` applies little shrinkage, allowing substantial variability in the heights of the fitted lines, which closely approach the observed values for each student. The fit of the mixed model `mod.2` is consequently similar to that of a fixed-effects model with age and a categorical predictor for individual students (i.e., treating students as a factor, and not shown here). 

The  mixed model `mod.2` therefore fits individual observations well, and we anticipate a favorable assessment using individual-based cross-validation. In contrast, the large variability in the BLUPs results in larger residuals for predictions based on fixed effects alone, and so we expect that cluster-based cross-validation won't show an advantage for model `mod.2` compared to the smaller model `mod.0`, which includes only fixed and random intercepts.

Had the mixed model applied considerable shrinkage, then neither cluster-based nor case-based cross-validation would show much improvement over the random-intercept-only model. In our experience, the degree of shrinkage does not vary smoothly as parameters are changed but tends to be "all or nothing," and near the tipping point, the behavior of estimates can be affected considerably by the choice of algorithm used to fit the model.

Finally, `mod.3` directly estimates the model used to generate the data. As mentioned, it is a constrained version of `mod.2`, with the coefficient of `xm` set to 0, and with `x` expressed as a deviation from the cluster mean `xm`:

```{r}
summary(mod.3)
```

The predictions from `mod.3` are therefore similar to those from `mod.2`:

```{r plot-fits-mod3}
#| out.width = "100%",
#| fig.height = 6,
#| fig.cap = "Predictions from the estimated model generating the data."
(plot +
  xyplot(fit ~ x, subset(Data_long, modelcode == "mod3" & effect == "fe"),
         groups=group, type="l", lwd=2) +
  xyplot(fit ~ x, subset(Data_long, modelcode == "mod3" & effect == "re"),
         groups=group, type="l", lwd=2, lty=3)
) |> update(
  main="Model: y ~ 1 + I(x - xm) + (1 | group)",
  ylim=c(4, 16),
  key=list(
    corner=c(0.05, 0.05),
    text=list(c("fixed effects only","fixed and random")),
    lines=list(lty=c(1, 3))))
```

We next carry out case-based cross-validation, which, as we have explained, is based on both fixed and predicted random effects (i.e., BLUPs), and cluster-based cross-validation, which is based on fixed effects only. In order to reduce between-model random variability in comparisons of models, we apply `cv()` to the list of models created by the `models()` function (introduced previously), performing cross-validation with the same folds for each model:

```{r cross-validation-clusters}
#| out.width = "100%",
#| fig.height = 6,
#| fig.cap = "10-fold cluster-based cross-validation comparing random intercept models with varying fixed effects."
modlist <- models("~ 1"=mod.0, "~ 1 + x"=mod.1, 
                  "~ 1 + x + xm"=mod.2, "~ 1 + I(x - xm)"=mod.3)
cvs_clusters <- cv(modlist, data=Data, cluster="group", k=10, seed=6449)
plot(cvs_clusters, main="Model Comparison, Cluster-Based CV")
```

```{r cross-validation-cases}
#| out.width = "100%",
#| fig.height = 6,
#| fig.cap = "10-fold case-based cross-validation comparing random intercept models with varying fixed effects."
cvs_cases <- cv(modlist, data=Data, seed=9693)
plot(cvs_cases, main="Model Comparison, Case-Based CV")
```

In summary, model `mod.1`, with $x$ alone and without the contextual mean of $x$, is assessed as fitting very poorly by cluster-based CV, but relatively much better by case-based CV. Model `mod.2`, which includes both $x$ and its contextual mean, produces better results using both cluster-based and case-based CV. The data-generating model, `mod.3`, which includes the fixed effect of `x - xm` in place of separate terms in `x` and `xm`, isn't distinguishable from model `mod.2`, which includes `x` and `xm` separately, even though `mod.2` has an unnecessary parameter (recall that the population coefficient of `xm` is 0 when `x` is expressed as deviations from the contextual mean). These conclusions are consistent with our observations based on graphing predictions from the various models, and they illustrate the desirability of assessing mixed-effect models at different hierarchical levels.

## Replicating cross-validation

Assuming that the number of cases $n$ is a multiple of the number of folds $k$---a slightly simplifying assumption---the number of possible partitions of cases into folds is $\frac{n!}{[(n/k)!]^k}$, a number that grows very large very quickly. For example, for $n = 10$ and $k = 5$, so that the folds are each of size $n/k = 2$, there are $113,400$ possible partitions; for $n=100$ and $k=5$, where $n/k = 20$, still a small problem, the number of possible partitions is truly astronomical, $1.09\times 10^{66}$.

Because the partition into folds that's employed is selected randomly, the resulting CV criterion estimates are subject to sampling error. (An exception is LOO cross-validation, which is not at all random.) To get a sense of the magnitude of the sampling error, we can repeat the CV procedure with different randomly selected partitions into folds. All of the CV functions in the **cv** package are capable of repeated cross-validation, with the number of repetitions controlled by the `reps` argument, which defaults to `1`.

Here, for example, is 10-fold CV for the Mroz logistic regression, repeated 5 times:

```{r mroz-reps}
cv(m.mroz, criterion=BayesRule, seed=248, reps=5, 
   method="Woodbury")
```
When `reps` > `1`, the result returned by `cv()` is an object of class `"cvList"`---literally a list of `"cv"` objects. The results are reported for each repetition and then averaged across repetitions, with the standard deviations of the CV criterion and the biased-adjusted CV criterion given in parentheses. In this example, there is therefore little variation across repetitions, increasing our confidence in the reliability of the results. 

Notice that the seed that's set in the `cv()` command pertains to the first repetition and the seeds for the remaining repetitions are then selected pseudo-randomly.[^replicates] Setting the first seed, however, makes the entire process easily replicable, and the seed for each repetition is stored in the corresponding element of the `"cvList"` object (which isn't, however, saved in the example).

[^replicates]: Because of the manner in which the computation is performed, the order of the replicates in the `"cvList"` object returned by `cv()` isn't the same as the order in which the replicates are computed. Each element of the result, however, is a `"cv"` object with the correct random-number seed saved, and so this technical detail can be safely ignored. The individual `"cv"` objects are printed in the order in which they are stored rather than the order in which they are computed.

It's also possible to replicate CV when comparing competing models via the `cv()` method for `"modList"` objects. Recall our comparison of polynomial regressions of varying degree fit to the `Auto` data; we performed 10-fold CV for each of 10 models. Here, we replicate that process 5 times for each model and graph the results:
```{r model-comparison-with-reps}
#| out.width = "100%",
#| fig.height = 5,
#| fig.cap = " Replicated cross-validated 10-fold CV as a function of polynomial degree, $p$"
cv.auto.reps <- cv(models(m.1, m.2, m.3, m.4, m.5,
                        m.6, m.7, m.8, m.9, m.10),
                 data=Auto, seed=8004, reps=5)
plot(cv.auto.reps)
```
The graph shows both the average CV criterion and its range for each of the competing models.

## Cross-validating model selection

### A preliminary example

As @HastieTibshiraniFriedman:2009 [Sec. 7.10.2: "The Wrong and Right Way to Do Cross-validation"] explain, if the whole data are used to select or fine-tune a statistical model, subsequent cross-validation of the model is intrinsically misleading, because the model is selected to fit the whole data, including the part of the data that remains when each fold is removed.

The following example is similar in spirit to one employed by @HastieTibshiraniFriedman:2009. Suppose that we randomly generate $n = 1000$ independent observations for a response variable variable $y \sim N(\mu = 10, \sigma^2 = 0)$, and independently sample $1000$ observations for $p = 100$ "predictors," $x_1, \ldots, x_{100}$, each from $x_j \sim N(0, 1)$. The response has nothing to do with the predictors and so the population linear-regression model $y_i = \alpha + \beta_1 x_{i1} + \cdots + \beta_{100} x_{i,100} + \varepsilon_i$ has $\alpha = 10$ and all $\beta_j = 0$. 

```{r generate-selection-data}
set.seed(24361) # for reproducibility
D <- data.frame(
  y = rnorm(1000, mean=10),
  X = matrix(rnorm(1000*100), 1000, 100)
)
head(D[, 1:6])
```
Least-squares provides accurate estimates of the regression constant $\alpha = 10$ and the error variance $\sigma^2 = 1$ for the "null model" including only the regression constant; moreover, the omnibus $F$-test of the correct null hypothesis that all of the $\beta$s are 0 for the "full model" with all 100 $x$s is associated with a large $p$-value:
```{r omnibus-F}
m.full <- lm(y ~ ., data=D)
m.null <- lm(y ~ 1, data=D)
anova(m.null, m.full)

summary(m.null)
```
Next, using the `stepAIC()` function in the **MASS** package [@VenablesRipley:2002], let us perform a forward stepwise regression to select a "best" model, starting with the null model, and using AIC as the model-selection criterion (see the help page for `stepAIC()` for details):[^selection-order]

[^selection-order]: It's generally advantageous to start with the largest model, here the one with 100 predictors, and proceed by backward elimination. In this demonstration, however, where all of the $\beta$s are really 0, the selected model will be small, and so we proceed by forward selection from the null model to save computing time.

```{r forward-selection}
m.select <- MASS::stepAIC(m.null,
                          direction="forward", trace=FALSE,
                          scope=list(lower=~1, upper=formula(m.full)))
summary(m.select)
mse(D$y, fitted(m.select))
```
The resulting model has 15 predictors, a very modest $R^2 = .044$, but a small $p$-value for its omnibus $F$-test (which, of course, is entirely spurious because the same data were used to select and test the model). The MSE for the selected model is smaller than the true error variance $\sigma^2 = 1$, as is the estimated error variance for the selected model, $\widehat{\sigma}^2 = 0.973^2 = 0.947$.

If we cross-validate the selected model, we also obtain an optimistic estimate of its predictive power:
```{r cv-selectedModel}
cv(m.select, seed=2529)
```

The `cvSelect()` function in the **cv** package allows us to cross-validate the whole model-selection procedure. The first argument to `cvSelect()` is a model-selection function capable of refitting the model with a fold omitted and returning a CV criterion. The `selectStepAIC()` function, also in **cv** and based on `stepAIC()`, is suitable for use with `cvSelect()`:
```{r cvSelect-artificial-data, cache=TRUE}
cv.select <- cvSelect(selectStepAIC, data=D, seed=3791,
                      model=m.null, direction="forward",
                      scope=list(lower=~1, 
                                 upper=formula(m.full)))
cv.select
```

The other arguments to `cvSelect()` are:

* `data`, the data set to which the model is fit;
* `seed`, an optional seed for R's pseudo-random-number generator; as for `cv()`, if the seed isn't supplied by the user, a seed is randomly selected and saved;
* additional arguments required by the model-selection function, here the starting `model` argument, the `direction` of model selection, and the `scope` of models considered (from the model with only a regression constant to the model with all 100 predictors).


By default, `cvSelect()` performs 10-fold CV, and produces an estimate of MSE for the model-selection procedure even *larger* than the true error variance, $\sigma^2 = 1$.

Also by default, when the number of folds is 10 or fewer, `cvSelect()` saves the coefficients of the selected models. In this example, the `compareFolds()` function reveals that the variables retained by the model-selection process in the several folds are quite different:
```{r compare-selected-models}
compareFolds(cv.select)
```
### Mroz's logistic regression revisited

For a contrasting example we apply model selection to Mroz's logistic regression for married women's labor-force participation. First, recall the logistic regression model that we fit to the `Mroz` data:
```{r recall-Mroz-regression}
summary(m.mroz)
```

Applying stepwise model selection Mroz's logistic regression, using BIC as the model-selection criterion (via the argument `k=log(nrow(Mroz))` to `stepAIC()`) selects 5 of the 7 original predictors:
```{r mroz-selection}
m.mroz.sel <- MASS::stepAIC(m.mroz, k=log(nrow(Mroz)),
                            trace=FALSE)
summary(m.mroz.sel)
BayesRule(Mroz$lfp == "yes",
          predict(m.mroz.sel, type="response"))
```
The Bayes rule applied to the selected model misclassifies 32% of the cases in the `Mroz` data. 

Cross-validating the selected model produces a similar, slightly larger, estimate of misclassification, about 33%:
```{r cv-mroz-regression}
cv(m.mroz.sel, criterion=BayesRule, seed=345266)
```
Is this estimate of predictive performance optimistic?

We proceed to apply the model-selection procedure by cross-validation, producing more or less the same result:
```{r cv-mroz-selection}
m.mroz.sel.cv <- cvSelect(selectStepAIC, Mroz, 
                          seed=6681,
                          criterion=BayesRule,
                          model=m.mroz,
                          AIC=FALSE)
m.mroz.sel.cv
```
Setting `AIC=FALSE` in the call to `cvSelect()` uses the BIC rather than the AIC as the model-selection criterion. As it turns out, exactly the same predictors are selected when each of the 10 folds are omitted, and the several coefficient estimates are very similar, as we show using `compareFolds()`:
```{r compare-selected-models-mroz}
compareFolds(m.mroz.sel.cv)
```
In this example, therefore, we appear to obtain a realistic estimate of model performance directly from the selected model, because there is little added uncertainty induced by model selection.

### Cross-validating choice of transformations in regression

The **cv** package also provides a `cvSelect()` procedure, `selectTrans()`, for choosing transformations of the predictors and the response in regression. 

Some background: As @Weisberg:2014 [Sec. 8.2] explains, there are technical advantages to having (numeric) predictors in linear regression analysis that are themselves linearly related. If the predictors *aren't* linearly related, then the relationships between them can often be straightened by power transformations. Transformations can be selected after graphical examination of the data, or by analytic methods. Once the relationships between the predictors are linearized, it can be advantageous similarly to transform the response variable towards normality.

Selecting transformations analytically raises the possibility of automating the process, as would be required for cross-validation. One could, in principle, apply graphical methods to select transformations for each fold, but because a data analyst couldn't forget the choices made for previous folds, the process wouldn't really be applied independently to the folds.

To illustrate, we adapt an example appearing in several places in @FoxWeisberg:2019 (for example in Chapter 3 on transforming data), using data on the prestige and other characteristics of 102 Canadian occupations circa 1970. The data are in the `Prestige` data frame in the **carData** package:
```{r Prestige-data}
data("Prestige", package="carData")
head(Prestige)
summary(Prestige)
```
The variables in the `Prestige` data set are:

* `education`: average years of education for incumbents in the occupation, from the 1971 Canadian Census.
* `income`: average dollars of annual income for the occupation, from the Census.
* `women`: percentage of occupational incumbents who were women, also from the Census.
* `prestige`: the average prestige rating of the occupation on a 0--100 "thermometer" scale, in a Canadian social survey conducted around the same time.
* `type`, type of occupation, and `census`, the Census occupational code, which are not used in our example.

The object of a regression analysis for the `Prestige` data (and their original purpose) is to predict occupational prestige from the other variables in the data set.

A scatterplot matrix (using the `scatterplotMatrix()` function in the **car** package) of the numeric variables in the data reveals that the distributions of `income` and `women` are positively skewed, and that some of the relationships among the three predictors, and between the predictors and the response (i.e., `prestige`), are nonlinear:
```{r scatterplot-matrix}
#| out.width = "100%",
#| fig.height = 6,
#| fig.cap = "Scatterplot matrix for the `Prestige` data."
library("car")
scatterplotMatrix(~ prestige + income + education + women,
                  data=Prestige, smooth=list(spread=FALSE))
```
The `powerTransform()` function in the **car** package transforms variables towards multivariate normality by a generalization of Box and Cox's maximum-likelihood-like approach [@BoxCox:1964]. Several "families" of power transformations can be used, including the original Box-Cox family, simple powers (and roots), and two adaptations of the Box-Cox family to data that may include negative values and zeros: the Box-Cox-with-negatives family and the Yeo-Johnson family; see @Weisberg:2014 [Chap. 8], and @FoxWeisberg:2019 [Chap. 3] for details. Because `women` has some zero values, we use the Yeo-Johnson family:
```{r power-transform-Prestige}
trans <- powerTransform( cbind(income, education, women) ~ 1,
                         data=Prestige, family="yjPower")
summary(trans)
```
We thus have evidence of the desirability of transforming `income` (by the $1/3$ power) and `women` (by the $0.16$ power---which is close to the "0" power, i.e., the log transformation), but not `education`. Applying the "rounded" power transformations makes the predictors better-behaved:
```{r transformed-predictors}
#| out.width = "100%",
#| fig.height = 6,
#| fig.cap = "Scatterplot matrix for the `Prestige` data with the predictors transformed."
P <- Prestige[, c("prestige", "income", "education", "women")]
(lambdas <- trans$roundlam)
names(lambdas) <- c("income", "education", "women")
for (var in c("income", "education", "women")){
  P[, var] <- yjPower(P[, var], lambda=lambdas[var])
}
summary(P)

scatterplotMatrix(~ prestige + income + education + women,
                  data=P, smooth=list(spread=FALSE))
```
Comparing the MSE for the regressions with the original and transformed predictors shows a advantage to the latter:
```{r prestige-regressions}
m.pres <- lm(prestige ~ income + education + women, data=Prestige)
m.pres.trans <- lm(prestige ~ income + education + women, data=P)
mse(Prestige$prestige, fitted(m.pres))
mse(P$prestige, fitted(m.pres.trans))
```
Similarly, component+residual plots for the two regressions, produced by the `crPlots()` function in the **car** package, suggest that the partial relationship of `prestige` to `income` is more nearly linear in the transformed data, but the transformation of `women` fails to capture what appears to be a slight quadratic partial relationship; the partial relationship of `prestige` to `education` is close to linear in both regressions:
```{r CR-plots-untransformed}
#| fig.cap = "Component+residual plots for the `Prestige` regression with the original predictors."
crPlots(m.pres)
```
```{r CR-plots-transformed}
#| fig.cap = "Component+residual plots for the `Prestige` regression with transformed predictors."
crPlots(m.pres.trans)
```

Having transformed the predictors towards multinormality, we now consider whether there's evidence for transforming the response (using `powerTransform()` for Box and Cox's original method), and we discover that there's not:
```{r transform-response}
summary(powerTransform(m.pres.trans))
```

The `selectTrans()` function in the **cv** package automates the process of selecting predictor and response transformations. The function takes a `data` set and "working" `model` as arguments, along with the candidate `predictors` and `response` for transformation, and the transformation `family` to employ. If the `predictors` argument is missing then only the response is transformed, and if the `response` argument is missing, only the supplied predictors are transformed. The default `family` for transforming the predictors is `"bcPower"`---the original Box-Cox family---as is the default `family.y` for transforming the response; here we specify `family="yjPower` because of the zeros in `women`. `selectTrans()` returns the result of applying a lack-of-fit criterion to the model after the selected transformation is applied, with the default `criterion=mse`:
```{r selectTrans}
selectTrans(data=Prestige, model=m.pres,
            predictors=c("income", "education", "women"),
            response="prestige", family="yjPower")
```
`selectTrans()` also takes an optional `indices` argument, making it suitable for doing computations on a subset of the data (i.e., a CV fold), and hence for use with `cvSelect()` (see `?selectTrans` for details):

```{r cv-select-transformations}
cvs <- cvSelect(selectTrans, data=Prestige, model=m.pres, seed=1463,
                predictors=c("income", "education", "women"),
                response="prestige",
                family="yjPower")
cvs

cv(m.pres, seed=1463) # untransformed model with same folds

compareFolds(cvs)
```
The results suggest that the predictive power of the transformed regression is reliably greater than that of the untransformed regression (though in both case, the cross-validated MSE is considerably higher than the MSE computed for the whole data). Examining the selected transformations for each fold reveals that the predictor `education` and the response `prestige` are never transformed; that the $1/3$ power is selected for `income` in all of the folds; and that the transformation selected for `women` varies narrowly across the folds between the $0$th power (i.e., log) and the $1/3$ power.

## Parallel computations

The CV functions in the **cv** package are all capable of performing parallel computations by setting the `ncores` argument (specifying the number of computer cores to be used) to a number > `1` (which is the default). Parallel computation can be advantageous for large problems, reducing the execution time of the program.

To illustrate, let's time model selection in Mroz's logistic regression, repeating the computation as performed previously and then doing it in parallel using 2 cores:
```{r parallel-computation}
system.time(m.mroz.sel.cv <- cvSelect(selectStepAIC, Mroz,
                          seed=6681,
                          criterion=BayesRule,
                          model=m.mroz,
                          AIC=FALSE))

system.time(m.mroz.sel.cv.p <- cvSelect(selectStepAIC, Mroz,
                          seed=6681,
                          criterion=BayesRule,
                          model=m.mroz,
                          AIC=FALSE,
                          ncores=2))
all.equal(m.mroz.sel.cv, m.mroz.sel.cv.p)
```

In this small problem, the parallel computation is actually *slower*, because there is an overhead cost to parallelization, but we can see that it produces the same result as before.

## Computational notes

### Efficient computations for linear and generalized linear models

The most straightforward way to implement cross-validation in R for statistical modeling functions that are written in the canonical manner is to use `update()` to refit the model with each fold removed. This is the approach taken in the default method for `cv()`, and it is appropriate if the cases are independently sampled. Refitting the model in this manner for each fold is generally feasible when the number of folds in modest, but can be prohibitively costly for leave-one-out cross-validation when the number of cases is large.

The `"lm"` and `"glm"` methods for `cv()` take advantage of computational efficiencies by avoiding refitting the model with each fold removed. Consider, in particular, the weighted linear model $\mathbf{y}_{n \times 1} = \mathbf{X}_{n \times p}\boldsymbol{\beta}_{p \times 1} + \boldsymbol{\varepsilon}_{n \times 1}$, where $\boldsymbol{\varepsilon} \sim \mathbf{N}_n \left(\mathbf{0}, \sigma^2 \mathbf{W}^{-1}_{n \times n}\right)$. Here, $\mathbf{y}$ is the response vector,  $\mathbf{X}$ the model matrix, and $\boldsymbol{\varepsilon}$ the error vector, each for $n$ cases, and $\boldsymbol{\beta}$ is the vector of $p$ population regression coefficients. The errors are assumed to be multivariately normally distributed with 0 means and covariance matrix $\sigma^2 \mathbf{W}^{-1}$, where $\mathbf{W} = \mathrm{diag}(w_i)$ is a diagonal matrix of inverse-variance weights. For the linear model with constant error variance, the weight matrix is taken to be $\mathbf{W} = \mathbf{I}_n$, the order-$n$ identity matrix.

The weighted-least-squares (WLS) estimator of $\boldsymbol{\beta}$ is [see, e.g., @Fox:2016, Sec. 12.2.2] [^WLS]
$$
\mathbf{b}_{\mathrm{WLS}} = \left( \mathbf{X}^T \mathbf{W} \mathbf{X} \right)^{-1} 
  \mathbf{X}^T \mathbf{W} \mathbf{y}
$$ 

[^WLS]: This is a definitional formula, which assumes that the model matrix $\mathbf{X}$ is of full column rank, and which can be subject to numerical instability when $\mathbf{X}$ is ill-conditioned. `lm()` uses the singular-value decomposition of the model matrix to obtain computationally more stable results.

Fitted values are then $\widehat{\mathbf{y}} = \mathbf{X}\mathbf{b}_{\mathrm{WLS}}$.

The LOO fitted value for the $i$th case can be efficiently computed by $\widehat{y}_{-i} = y_i - e_i/(1 - h_i)$ where $h_i = \mathbf{x}^T_i \left( \mathbf{X}^T \mathbf{W} \mathbf{X} \right)^{-1} \mathbf{x}_i$ (the so-called "hatvalue"). Here, $\mathbf{x}^T_i$ is the $i$th row of $\mathbf{X}$, and $\mathbf{x}_i$ is the $i$th row written as a column vector. This approach can break down when one or more hatvalues are equal to 1, in which case the formula for $\widehat{y}_{-i}$ requires division by 0.

To compute cross-validated fitted values when the folds contain more than one case, we make use of the Woodbury matrix identify [@Wikipedia-Woodbury:2023],
$$
\left(\mathbf{A}_{m \times m} + \mathbf{U}_{m \times k} 
\mathbf{C}_{k \times k} \mathbf{V}_{k \times m} \right)^{-1} = \mathbf{A}^{-1} - \mathbf{A}^{-1}\mathbf{U} \left(\mathbf{C}^{-1} + 
\mathbf{VA}^{-1}\mathbf{U} \right)^{-1} \mathbf{VA}^{-1}
$$
where $\mathbf{A}$ is a nonsingular order-$n$ matrix. We apply this result by letting
\begin{align*}
	\mathbf{A} &= \mathbf{X}^T \mathbf{W} \mathbf{X} \\
	\mathbf{U} &= \mathbf{X}_\mathbf{j}^T \\
	\mathbf{V} &= - \mathbf{X}_\mathbf{j} \\
	\mathbf{C} &= \mathbf{W}_\mathbf{j} \\
\end{align*}
where the subscript $\mathbf{j} = (i_{j1}, \ldots, i_{jm})^T$ represents the vector of indices for the cases in the $j$th fold, $j = 1, \ldots, k$. The negative sign in $\mathbf{V} = - \mathbf{X}_\mathbf{j}$ reflects the *removal*, rather than addition, of the cases in $\mathbf{j}$. 

Applying the Woodbury identity isn't quite as fast as using the hatvalues, but it is generally much faster than refitting the model. A disadvantage of the Woodbury identity, however, is that it entails explicit matrix inversion and thus may be numerically unstable. The inverse of $\mathbf{A} = \mathbf{X}^T \mathbf{W} \mathbf{X}$ is available directly in the `"lm"` object, but the second term on the right-hand side of the Woodbury identity requires a matrix inversion with each fold deleted. (In contrast, the inverse of each $\mathbf{C} = \mathbf{W}_\mathbf{j}$ is straightforward because $\mathbf{W}$ is diagonal.)

The Woodbury identity also requires that the model matrix be of full rank. We impose that restriction in our code by removing redundant regressors from the model matrix for all of the cases, but that doesn't preclude rank deficiency from surfacing when a fold is removed. Rank deficiency of $\mathbf{X}$ doesn't disqualify cross-validation because all we need are fitted values under the estimated model.

`glm()` computes the maximum-likelihood estimates for a generalized linear model by iterated weighted least squares [see, e.g., @FoxWeisberg:2019, Sec. 6.12]. The last iteration is therefore just a WLS fit of the "working response" on the model matrix using "working weights." Both the working weights and the working response at convergence are available from the information in the object returned by `glm()`. 

We then treat re-estimation of the model with a case or cases deleted as a WLS problem, using the hatvalues or the Woodbury matrix identity. The resulting fitted values for the deleted fold aren't exact---that is, except for the Gaussian family, the result isn't identical to what we would obtain by literally refitting the model---but in our (limited) experience, the approximation is very good, especially for LOO CV, which is when we would be most tempted to use it. Nevertheless, because these results are approximate, the default for the `"glm"` `cv()` method is to perform the exact computation, which entails refitting the model with each fold omitted.

### Computation of the bias-corrected CV criterion

Let $\mathrm{CV}(\mathbf{y}, \widehat{\mathbf{y}})$ represent a cross-validation cost criterion, such as mean-squared error, computed for all of the $n$ values of the response $\mathbf{y}$ based on fitted values $\widehat{\mathbf{y}}$ from the model fit to all of the data. We divide the $n$ cases into $k$ folds of approximately $n_j \approx n/k$ cases each, where $n = \sum n_j$. As above, let $\mathbf{j}$ denote the indices of the cases in the $j$th fold.

Now define $\mathrm{CV}_j = \mathrm{CV}(\mathbf{y}, \widehat{\mathbf{y}}^{(j)})$ and $\mathrm{CV}_j^{(j)} = \mathrm{CV}(\mathbf{y}_{\mathbf{j}}, \widehat{\mathbf{y}}^{(j)}_{\mathbf{j}})$. The superscript $(j)$ on $\widehat{\mathbf{y}}^{(j)}$ and $\widehat{\mathbf{y}}^{(j)}_{\mathbf{j}}$ represents fitted values computed from the model with fold $j$ omitted, respectively for all of the cases (i.e., $\mathrm{CV}_j$) and for the cases only in the $j$th fold ($\mathrm{CV}_j^{(j)}$).

Then the cross-validation criterion is the weighted average of the $\mathrm{CV}_j^{(j)}$
$$
\mathrm{CV} = \frac{1}{n} \sum_{j = 1}^{k} n_j \mathrm{CV}_j^{(j)}
$$
Following @DavisonHinkley1997[pp. 293--295], the bias-adjusted cross-validation criterion is
$$
\mathrm{CV}_{\mathrm{adj}} = \mathrm{CV} + \mathrm{CV}(\mathbf{y}, \widehat{\mathbf{y}}) - \frac{1}{n} \sum_{j=1}^{k} n_j \mathrm{CV}_j
$$

```{r coda, include = FALSE}
options(.opts)
```

## References





