D7.1 · LESSON 1 OF 6

Autoencoders

Explain the different training objectives of autoencoders, VAEs, GANs and diffusion models.

THE IDEA

Compress, then rebuild

An autoencoder has two halves. The encoder squeezes an input into a short code (the latent vector), and the decoder tries to rebuild the original from that code. It is trained to make the rebuilt copy match the input. Nobody labels anything: the input is its own target.

The squeeze is the point. If the code were as long as the input the network could copy it. With a code of one number for a two-number point, it has to find what actually varies. Here the four points lie on a line, so a single number (the position along the line) is enough to rebuild them exactly, if the encoder points along that line.

code direction (degrees)20

black = original points, blue = what the decoder rebuilds from ONE number each encoder: z = x·cos θ + y·sin θ decoder: (x̂, ŷ) = z·(cos θ, sin θ) total squared error at 20°: 5.3582

Rotate the code direction. Black dots are the data, blue dots what the decoder rebuilds. The dashed lines are the errors.

Predict first. You train an autoencoder and then feed the decoder random code vectors. Can they decode poorly?

NumPy

_, _, Vt = np.linalg.svd(X, full_matrices=False)
W = Vt[:1]                  # encoder direction (the best 1-number code)
Z = X @ W.T                 # encode
X_hat = Z @ W               # decode

Chapter context

Generative modeling tries to represent or sample from a data distribution. Different model families construct that distribution differently; a plausible-looking sample alone does not tell us how the model was trained or how well it covers the data.

TRY IT YOURSELF

Part 1. Do it hands-on

Find the code direction by hand, then fit a linear autoencoder with the SVD and measure its error as the code grows.

code direction (degrees)90

black = original points, blue = what the decoder rebuilds from ONE number each encoder: z = x·cos θ + y·sin θ decoder: (x̂, ŷ) = z·(cos θ, sin θ) total squared error at 90°: 15

error at 0° code of (3, 3) at 45° x of decode(z = 10), 2 decimals

To do: A 1-number code along the x-axis (angle 0°) can only keep the x part of each point. Type the total squared reconstruction error for the four points
To do: Rotate the code direction until the reconstruction error is zero
To do: With the 45° code, what code (one number) does the point (3, 3) get? Type it with three decimals
To do: Feed the decoder a random code z = 10 (the training codes go up to 4.24). Type the x coordinate it decodes to (2 decimals)

Part 2. Do it in code

fit_linear_ae(X, k): X has shape (n, d). Return the encoder matrix W of shape (k, d) whose rows are the top-k right singular vectors of X (np.linalg.svd, full_matrices=False). There is no bias, so the code is z = X @ W.T.
encode(X, W) returns X @ W.T. decode(Z, W) returns Z @ W (the decoder is the transpose of the encoder).
reconstruction_error(X, W): the mean over all entries of (X − decode(encode(X, W), W))².
latent_range(Z): return (min, max) of every column of Z as two arrays.

import numpy as np

def fit_linear_ae(X, k):
    pass

def encode(X, W):
    pass

def decode(Z, W):
    pass

def reconstruction_error(X, W):
    pass

def latent_range(Z):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Can arbitrary random latent vectors decode poorly in a basic autoencoder?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

A VAE uses $z=\mu+\sigma\odot\epsilon$ , with $\epsilon\sim\mathcal N(0,I)$ . A standard diffusion forward marginal is

x_t=\sqrt{\bar\alpha_t}x_0+\sqrt{1-\bar\alpha_t}\epsilon.

Here alpha-bar summarizes retained signal. This is a noising equation, not a complete sampling algorithm.

A VAE optimizes an evidence lower bound:

\operatorname{ELBO}=E_{q_\phi(z\mid x)}[\log p_\theta(x\mid z)]-D_{KL}(q_\phi(z\mid x)\Vert p(z)).

For diagonal Gaussian q and standard normal p, its KL is $\frac12\sum_j(\mu_j^2+\sigma_j^2-1-\log\sigma_j^2)$ . The original GAN minimax objective is

\min_G\max_D E_{x\sim p_{data}}[\log D(x)]+E_{z\sim p(z)}[\log(1-D(G(z)))].

Practical generator losses often use a non-saturating variant. A common diffusion noise-prediction loss is $E\|\epsilon-\epsilon_\theta(x_t,t)\|^2$ under a specified noising schedule.

Worked example

For a scalar diffusion example, take x0=2, alpha-bar=0.64 and epsilon=−1. The signal term is 0.8×2=1.6 and the noise term is 0.6×(−1)=−0.6, giving xt=1.0. The clean value cannot be inferred from xt alone without additional assumptions or a learned model.

For q=N(1,1) against a standard normal prior, the one-dimensional VAE KL is 0.5×(1²+1−1−ln 1)=0.5 nats. This is only the regularization term, not the complete VAE objective.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import sqrt, isclose
x0, alpha, noise = 2,.64,-1
xt=sqrt(alpha)*x0+sqrt(1-alpha)*noise
assert isclose(xt,1)
assert 2+.5*(-1) == 1.5
from math import log
assert isclose(.5*(1**2+1**2-1-log(1**2)),.5)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

A VAE's approximate posterior is not identical to its prior. GAN loss is not a universal sample-quality score. A diffusion noising equation is not sufficient to implement the reverse model.