Gaussian Mixture Models (GMM)

Intuition #

This method should be interpreted through its assumptions, data conditions, and how parameter choices affect generalization.

Detailed Explanation #

Mathematics #

The density of \(\mathbf{x}\) is

with mixture weights \(\pi_k\) (non-negative and summing to 1). EM alternates:

• E-step: compute responsibilities \(\gamma_{ik}\). $$ \gamma_{ik} = \frac{\pi_k \, \mathcal{N}(\mathbf{x}_i \mid \boldsymbol{\mu}_k, \boldsymbol{\Sigma}_k)} {\sum_{j=1}^K \pi_j \, \mathcal{N}(\mathbf{x}_i \mid \boldsymbol{\mu}_j, \boldsymbol{\Sigma}_j)}. $$
• M-step: re-estimate \(\pi_k, \boldsymbol{\mu}_k, \boldsymbol{\Sigma}k\) using \(\gamma{ik}\) as weights.

The log-likelihood increases monotonically and converges to a local optimum.

Python walkthrough #

We fit a GMM to synthetic 2D blobs, plot the hard assignments, and report mixture weights and the responsibility matrix shape.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77

from __future__ import annotations

import matplotlib.pyplot as plt
import numpy as np
from numpy.typing import NDArray
from sklearn.datasets import make_blobs
from sklearn.mixture import GaussianMixture

def run_gmm_demo(
    n_samples: int = 600,
    n_components: int = 3,
    cluster_std: list[float] | tuple[float, ...] = (1.0, 1.4, 0.8),
    covariance_type: str = "full",
    random_state: int = 7,
    n_init: int = 8,
) -> dict[str, object]:
    """Fit a Gaussian mixture model and visualise hard labels with component centres."""
    features, _ = make_blobs(
        n_samples=n_samples,
        centers=n_components,
        cluster_std=cluster_std,
        random_state=random_state,
    )

    gmm = GaussianMixture(
        n_components=n_components,
        covariance_type=covariance_type,
        random_state=random_state,
        n_init=n_init,
    )
    gmm.fit(features)

    hard_labels = gmm.predict(features)
    responsibilities = gmm.predict_proba(features)
    log_likelihood = float(gmm.score(features))
    weights = gmm.weights_

    fig, ax = plt.subplots(figsize=(6.2, 5.2))
    scatter = ax.scatter(
        features[:, 0],
        features[:, 1],
        c=hard_labels,
        cmap="viridis",
        s=30,
        edgecolor="white",
        linewidth=0.2,
        alpha=0.85,
    )
    ax.scatter(
        gmm.means_[:, 0],
        gmm.means_[:, 1],
        marker="x",
        c="red",
        s=140,
        linewidth=2.0,
        label="Component centre",
    )
    ax.set_title("Gaussian mixture clustering (hard labels shown)")
    ax.set_xlabel("feature 1")
    ax.set_ylabel("feature 2")
    ax.grid(alpha=0.2)
    handles, _ = scatter.legend_elements()
    labels = [f"cluster {idx}" for idx in range(n_components)]
    ax.legend(handles, labels, title="predicted label", loc="upper right")
    fig.tight_layout()
    plt.show()

    return {
        "log_likelihood": log_likelihood,
        "weights": weights.tolist(),
        "responsibilities_shape": responsibilities.shape,
    }

metrics = run_gmm_demo()
print(f"log-likelihood: {metrics['log_likelihood']:.3f}")
print("mixture weights:", metrics["weights"])
print("responsibility matrix shape:", metrics["responsibilities_shape"])

FAQ #

What is a Gaussian Mixture Model? #

A Gaussian Mixture Model (GMM) is a probabilistic model that assumes data is generated from a mixture of \(K\) Gaussian (normal) distributions with unknown parameters. Each component has its own mean, covariance, and weight. Unlike k-means, which assigns each point to exactly one cluster, GMM produces soft assignments — a probability for each point belonging to each component.

How does a GMM differ from k-means? #

GMM is better when clusters overlap, have different sizes, or non-spherical shapes. k-means is faster and simpler when clusters are well-separated and roughly equal.

How do you choose the number of components K? #

Fit GMMs for a range of \(K\) values and compare using information criteria:

• BIC (Bayesian Information Criterion): penalises model complexity more heavily; generally preferred for cluster selection.
• AIC (Akaike Information Criterion): less penalty for complexity; can overfit with many components.

Choose the \(K\) where BIC reaches a minimum or shows an elbow. Also run multiple initialisations (n_init) to avoid local optima.

What covariance types are available in scikit-learn? #

GaussianMixture supports four covariance structures:

• full: each component has its own unconstrained covariance matrix — most flexible but most parameters.
• tied: all components share one covariance matrix — fewer parameters.
• diag: each component has a diagonal covariance (independent features, different scales per component).
• spherical: each component has a single variance value — closest to k-means.

Start with full when data is small; use diag or spherical when the feature dimension is high or data is limited.

What is the EM algorithm used in GMM? #

EM (Expectation-Maximisation) iterates two steps until convergence:

1. E-step: given the current parameters, compute the responsibility \(\gamma_{ik}\) — the posterior probability that sample \(i\) belongs to component \(k\).
2. M-step: update the mixture weights, means, and covariances using the responsibilities as soft weights.

The log-likelihood is guaranteed to increase (or stay the same) at each iteration, converging to a local optimum. Multiple restarts help find better solutions.

References #