Validation curve

Last updated 2020-02-26 Read time 3 min

Summary

Validation curves visualise how training and validation scores change when a single hyperparameter varies.
Use validation_curve to sweep a regularisation coefficient, plot both curves, and spot the sweet spot.
Learn how to interpret the graph when tuning hyperparameters and what caveats to keep in mind.

Cross-Validation — understanding this concept first will make learning smoother

sklearn.model_selection.validation_curve

1. What is a validation curve? #

A validation curve plots a given hyperparameter on the x-axis and both training/validation scores on the y-axis. Typical interpretation:

Training high, validation low → the model is overfitting; increase regularisation or decrease model capacity.
Both scores low → underfitting; relax regularisation or choose a more expressive model.
Both scores high and close → near an optimal setting; confirm with additional metrics.

While a learning curve analyses “sample size vs. score”, a validation curve analyses “hyperparameter vs. score”.

2. Python example (SVC with `C`) #

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
from __future__ import annotations

import matplotlib.pyplot as plt
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import validation_curve
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

def plot_validation_curve_for_svc() -> None:
    """Plot and save the validation curve for the regularisation parameter of SVC."""
    features, labels = make_classification(
        n_samples=1200,
        n_features=20,
        n_informative=5,
        n_redundant=2,
        n_repeated=0,
        n_classes=2,
        weights=[0.6, 0.4],
        flip_y=0.02,
        class_sep=1.2,
        random_state=42,
    )

    model = make_pipeline(StandardScaler(), SVC(kernel="rbf", gamma="scale"))
    param_range = np.logspace(-3, 2, 10)

    train_scores, valid_scores = validation_curve(
        estimator=model,
        X=features,
        y=labels,
        param_name="svc__C",
        param_range=param_range,
        scoring="roc_auc",
        cv=5,
        n_jobs=None,
    )

    train_mean = train_scores.mean(axis=1)
    train_std = train_scores.std(axis=1)
    valid_mean = valid_scores.mean(axis=1)
    valid_std = valid_scores.std(axis=1)

    plt.figure(figsize=(7, 5))
    plt.semilogx(param_range, train_mean, label="Train score", color="#1d4ed8")
    plt.fill_between(
        param_range,
        train_mean - train_std,
        train_mean + train_std,
        alpha=0.2,
        color="#1d4ed8",
    )
    plt.semilogx(param_range, valid_mean, label="Validation score", color="#ea580c")
    plt.fill_between(
        param_range,
        valid_mean - valid_std,
        valid_mean + valid_std,
        alpha=0.2,
        color="#ea580c",
    )
    plt.title("Validation Curve for SVC (ROC-AUC)")
    plt.xlabel("Regularisation parameter C")
    plt.ylabel("Score")
    plt.ylim(0.5, 1.05)
    plt.legend(loc="best")
    plt.grid(alpha=0.3)

plot_validation_curve_for_svc()

Validation curve for SVC — Lower C over-regularises, higher C overfits. The peak around C ≈ 1 gives the best validation score.

3. Reading the graph #

Left side (small C): strong regularisation causes underfitting; both scores are low.
Right side (large C): weak regularisation leads to high training score but falling validation score (overfitting).
Middle peak: training and validation curves converge, indicating a good trade-off.

4. Applying it in practice #

Pre-tune exploration: identify a promising hyperparameter range before running expensive searches (grid, random, Bayesian).
Check variance: look at the shaded error bands (standard deviation) to judge stability, especially with small datasets.
Prioritise multiple parameters: create validation curves for key parameters to decide which ones deserve deeper search.
Combine with learning curves: understand “what hyperparameter works” and “whether more data helps” simultaneously.

Validation curves make the direction of hyperparameter tuning intuitive and support decision-making across the team. Keeping these plots for each production model streamlines discussions about next steps.

1. What is a validation curve? #

2. Python example (SVC with C) #

3. Reading the graph #

4. Applying it in practice #

2. Python example (SVC with `C`) #