Ridge and Lasso regression
Ridge and Lasso regression are linear regression models that are designed to prevent models from overfitting by penalizing the coefficients of the model. Techniques such as these that attempt to prevent overfitting and increase generalizability are called regularization. On this page, we will use python to run least squares, ridge, and lasso regression and compare their coefficients.
Import required modules
import numpy as np
import matplotlib.pyplot as plt
import japanize_matplotlib
from sklearn.datasets import make_regression
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
Generate regression data for experiments
Artificially generate regression data using sklearn.datasets.make_regression. Specify two features (n_informative = 2) as necessary for prediction. Otherwise, they are redundant features that are not useful for prediction.
n_features = 5
n_informative = 2
X, y = make_regression(
n_samples=500,
n_features=n_features,
n_informative=n_informative,
noise=0.5,
random_state=777,
)
X = np.concatenate([X, np.log(X + 100)], 1) # Add redundant features
y_mean = y.mean(keepdims=True)
y_std = np.std(y, keepdims=True)
y = (y - y_mean) / y_std
Compare least squares, ridge regression, and lasso regression models
Let’s train each model with the same data and see the differences. Normally, the coefficient alpha of the regularization term should be changed, but in this case, it is fixed.
lin_r = make_pipeline(StandardScaler(with_mean=False), LinearRegression()).fit(X, y)
rid_r = make_pipeline(StandardScaler(with_mean=False), Ridge(alpha=2)).fit(X, y)
las_r = make_pipeline(StandardScaler(with_mean=False), Lasso(alpha=0.1)).fit(X, y)
Compare the values of the coefficients for each model
Plot the absolute value of each coefficient. It can be seen from the graph that for linear regression, the coefficients are almost never zero. We can also see that Lasso regression has coefficients = 0 for all but the features needed for prediction.
Since there were two useful features (n_informative = 2), we can see that linear regression has unconditional coefficients even on features that are not useful, while Lasso-Ridge regression could not be narrowed down to two features. Still, it resulted in many unwanted features having coefficients of zero.
feat_index = np.array([i for i in range(X.shape[1])])
plt.figure(figsize=(12, 4))
plt.bar(
feat_index - 0.2,
np.abs(lin_r.steps[1][1].coef_),
width=0.2,
label="LinearRegression",
)
plt.bar(feat_index, np.abs(rid_r.steps[1][1].coef_), width=0.2, label="RidgeRegression")
plt.bar(
feat_index + 0.2,
np.abs(las_r.steps[1][1].coef_),
width=0.2,
label="LassoRegression",
)
plt.xlabel(r"$\beta$")
plt.ylabel(r"$|\beta|$")
plt.grid()
plt.legend()
