Patrick Breheny’s Lecture Notes

Let \(M\) denotes a generic measure of accuracy, \(y\) denote the observed outcomes (for survival, this includes t and d), \(y^*\) denotes future outcomes, and \(f(X)\) denotes a model’s predictions.

Because of this phenomenon of overfitting, the quantity \[M\{f(X), y\} − M\{f(X), y^*\}\] is almost always positive; this quantity is known as the optimism of the model, and it tends to be more severe for complex models than simple models.

ESL

\[op \equiv {Err}_{in} - \overline{err}\] where \(op\) is optimism, \({Err}_{in}\) is in-sample error, \(\overline{err}\) is training error.