まとめ
- Adjusted R² compensates the plain R² for the number of features, reducing optimism when models grow more complex.
- Compute both R² and adjusted R² in Python to see how they diverge as features are added.
- Understand the limitations with small sample sizes and how to combine adjusted R² with other criteria.
1. Definition #
$$ \mathrm{Adjusted};R^2 = 1 - (1 - R^2)\frac{n - 1}{n - p - 1} $$
- \(n\): number of samples, \(p\): number of predictors.
- As \(p\) increases, the denominator shrinks; unless the new feature improves fit, adjusted R² decreases.
2. Python example #
import numpy as np
from sklearn.datasets import make_regression
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
X, y = make_regression(
n_samples=1_000,
n_features=10,
n_informative=6,
noise=5.0,
random_state=0,
)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=0
)
model = LinearRegression()
model.fit(X_train, y_train)
r2 = model.score(X_test, y_test)
n, p = X_test.shape
adj_r2 = 1 - (1 - r2) * (n - 1) / (n - p - 1)
print(f"R2 = {r2:.3f}")
print(f"Adjusted R2 = {adj_r2:.3f}")
Ensure \(n - p - 1 > 0\); otherwise shrink the feature set or enlarge the validation sample.
3. When to use it #
- Feature selection: discourages unnecessary variables; if adjusted R² drops, the new feature adds noise.
- Small samples: values fluctuate more; cross-validate to confirm improvements.
- Model comparison: among models fitted to the same dataset, adjusted R² rewards parsimonious solutions.
4. Relation to other metrics #
| Metric | Strength | Caveat |
|---|---|---|
| R² | Intuitive variance explanation | Always increases with more features |
| Adjusted R² | Accounts for feature count | Unstable for tiny sample sizes |
| AIC / BIC | Likelihood + penalty | Need correct likelihood assumptions |
Summary #
- Adjusted R² tempers R² by penalising complexity, revealing whether added features truly help.
- Use it alongside R², AIC/BIC, or cross-validation to form a well-rounded view of model quality.
- Pay attention to the sample-to-feature ratio; the metric only behaves well when the denominator stays positive.