Adjusted R²

Last updated 2020-04-08 Read time 2 min

Summary

Adjusted R² compensates the plain R² for the number of features, reducing optimism when models grow more complex.
Compute both R² and adjusted R² in Python to see how they diverge as features are added.
Understand the limitations with small sample sizes and how to combine adjusted R² with other criteria.

1. Definition #

$$ \mathrm{Adjusted}\;R^2 = 1 - (1 - R^2)\frac{n - 1}{n - p - 1} $$

$n$: number of samples, $p$: number of predictors.
As $p$ increases, the denominator shrinks; unless the new feature improves fit, adjusted R² decreases.

2. Python example #

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
import numpy as np
from sklearn.datasets import make_regression
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

X, y = make_regression(
    n_samples=1_000,
    n_features=10,
    n_informative=6,
    noise=5.0,
    random_state=0,
)

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=0
)

model = LinearRegression()
model.fit(X_train, y_train)

r2 = model.score(X_test, y_test)
n, p = X_test.shape
adj_r2 = 1 - (1 - r2) * (n - 1) / (n - p - 1)

print(f"R2 = {r2:.3f}")
print(f"Adjusted R2 = {adj_r2:.3f}")

Ensure $n - p - 1 > 0$; otherwise shrink the feature set or enlarge the validation sample.

3. When to use it #

Feature selection: discourages unnecessary variables; if adjusted R² drops, the new feature adds noise.
Small samples: values fluctuate more; cross-validate to confirm improvements.
Model comparison: among models fitted to the same dataset, adjusted R² rewards parsimonious solutions.

4. Relation to other metrics #

Metric	Strength	Caveat
R²	Intuitive variance explanation	Always increases with more features
Adjusted R²	Accounts for feature count	Unstable for tiny sample sizes
AIC / BIC	Likelihood + penalty	Need correct likelihood assumptions

Summary #

Adjusted R² tempers R² by penalising complexity, revealing whether added features truly help.
Use it alongside R², AIC/BIC, or cross-validation to form a well-rounded view of model quality.
Pay attention to the sample-to-feature ratio; the metric only behaves well when the denominator stays positive.