D7.6 · LESSON 6 OF 6

Latent spaces and evaluation

Explain the different training objectives of autoencoders, VAEs, GANs and diffusion models.

THE IDEA

Walking the latent space, and judging the results

A latent space is useful when nearby codes decode to similar outputs. Interpolation checks this: take two codes, move in a straight line z = (1−t)·z₀ + t·z₁ and decode each point. A smooth, plausible blend suggests the space is organised; garbage in the middle suggests it is not.

Judging generated samples is harder than it looks. Showing the prettiest examples hides problems. Measure separately:

Fidelity: does each sample look like real data? (Here: average distance from a sample to its nearest real point.)
Coverage / diversity: does the set include everything the real data contains, and do the samples differ from each other?
Usefulness: does it help the task you built it for?

interpolation t0.25

z₀ = [0, 0] z₁ = [4, 2] z = (1−t)·z₀ + t·z₁ = [1, 0.5] → decode(z) is the in-between sample

Green discs: real data [0 … 5]. Orange dots: generated samples [2.1, 2.0, 1.9, 2.05].

fidelity (average distance to nearest real): 0.063 coverage (real points within 0.5 of a sample): 0.167 diversity (average gap between samples): 0.108

Real data [0 … 5] (green) against four samples near 2. They are all realistic and almost identical.

Predict first. Can a model generate high-quality but nearly identical outputs?

Chapter context

Generative modeling tries to represent or sample from a data distribution. Different model families construct that distribution differently; a plausible-looking sample alone does not tell us how the model was trained or how well it covers the data.

TRY IT YOURSELF

Part 1. Do it hands-on

Interpolate by hand, compute coverage and then craft a sample set that scores well on fidelity and coverage together. Then write the metrics in Python.

interpolation t0.25

z₀ = [0, 0] z₁ = [4, 2] z = (1−t)·z₀ + t·z₁ = [1, 0.5] → decode(z) is the in-between sample

Green discs: real data [0 … 5]. Orange dots: generated samples (your six numbers).

Enter six numbers.

z at t = 0.25:

coverage of [2.1, 2.0, 1.9, 2.05]

your six samples:

To do: Interpolate between latents z₀ = [0, 0] and z₁ = [4, 2] at t = 0.25: z = (1−t)·z₀ + t·z₁. Type both numbers
To do: Real data = [0, 1, 2, 3, 4, 5]. A model produced [2.1, 2.0, 1.9, 2.05], all very realistic. Coverage = fraction of real points with a generated sample within 0.5. Type it (3 decimals)
To do: Write six generated numbers with coverage at least 0.8 (radius 0.5) AND average distance to the nearest real point at most 0.3

Part 2. Do it in code

lerp(z0, z1, t): the point a fraction t of the way from z0 to z1: (1 − t)·z0 + t·z1 (arrays). t must be within [0, 1] (ValueError otherwise).
interpolate(z0, z1, steps): `steps` evenly spaced points from z0 to z1 inclusive, as an array of shape (steps, d). steps must be at least 2 (ValueError).
fidelity(gen, real): the average distance from each generated number to its nearest real number (smaller is better).
coverage(gen, real, r): the fraction of real numbers that have a generated number within r.
diversity(gen): the mean absolute difference over all pairs of generated numbers (0 if fewer than two).

import numpy as np

def lerp(z0, z1, t):
    pass

def interpolate(z0, z1, steps):
    pass

def fidelity(gen, real):
    pass

def coverage(gen, real, r):
    pass

def diversity(gen):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Can a model generate high-quality but nearly identical outputs?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

A VAE uses $z=\mu+\sigma\odot\epsilon$ , with $\epsilon\sim\mathcal N(0,I)$ . A standard diffusion forward marginal is

x_t=\sqrt{\bar\alpha_t}x_0+\sqrt{1-\bar\alpha_t}\epsilon.

Here alpha-bar summarizes retained signal. This is a noising equation, not a complete sampling algorithm.

A VAE optimizes an evidence lower bound:

\operatorname{ELBO}=E_{q_\phi(z\mid x)}[\log p_\theta(x\mid z)]-D_{KL}(q_\phi(z\mid x)\Vert p(z)).

For diagonal Gaussian q and standard normal p, its KL is $\frac12\sum_j(\mu_j^2+\sigma_j^2-1-\log\sigma_j^2)$ . The original GAN minimax objective is

\min_G\max_D E_{x\sim p_{data}}[\log D(x)]+E_{z\sim p(z)}[\log(1-D(G(z)))].

Practical generator losses often use a non-saturating variant. A common diffusion noise-prediction loss is $E\|\epsilon-\epsilon_\theta(x_t,t)\|^2$ under a specified noising schedule.

Worked example

For a scalar diffusion example, take x0=2, alpha-bar=0.64 and epsilon=−1. The signal term is 0.8×2=1.6 and the noise term is 0.6×(−1)=−0.6, giving xt=1.0. The clean value cannot be inferred from xt alone without additional assumptions or a learned model.

For q=N(1,1) against a standard normal prior, the one-dimensional VAE KL is 0.5×(1²+1−1−ln 1)=0.5 nats. This is only the regularization term, not the complete VAE objective.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import sqrt, isclose
x0, alpha, noise = 2,.64,-1
xt=sqrt(alpha)*x0+sqrt(1-alpha)*noise
assert isclose(xt,1)
assert 2+.5*(-1) == 1.5
from math import log
assert isclose(.5*(1**2+1**2-1-log(1**2)),.5)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

A VAE's approximate posterior is not identical to its prior. GAN loss is not a universal sample-quality score. A diffusion noising equation is not sufficient to implement the reverse model.