D7.4 · LESSON 4 OF 6

GANs

Explain the different training objectives of autoencoders, VAEs, GANs and diffusion models.

THE IDEA

Two networks, one game

A generative adversarial network trains two models against each other. The generator turns random noise into samples. The discriminator looks at a sample and outputs the probability that it is real. The discriminator is trained to tell real from fake; the generator is trained so that the discriminator says "real" about its samples.

Green discs: where the real data lives (two clumps). Dots: generated samples. The toy discriminator answers 1 when a sample is within 1 of any green clump.

D(real)0.9D(fake)0.2

discriminator loss = −ln D(real) − ln(1 − D(fake)) = 0.329 generator loss = −ln D(fake) = 1.609 When D wins, G's loss is large; when G fools D, D's loss grows.

Can the generator fool D and still miss data?

generated sample 13generated sample 22.5

toy discriminator scores: 1, 1 clumps covered: 1 of 2

Top: the two losses for chosen discriminator outputs. Bottom: generated samples against two real clumps.

Predict first. A generator fools one discriminator completely. Does that prove it covers the whole data distribution?

Chapter context

Generative modeling tries to represent or sample from a data distribution. Different model families construct that distribution differently; a plausible-looking sample alone does not tell us how the model was trained or how well it covers the data.

TRY IT YOURSELF

Part 1. Do it hands-on

Compute both losses, then create a healthy generator and a mode-collapsed one that still fools the toy discriminator. Then implement the losses and coverage in Python.

Green discs: where the real data lives (two clumps). Dots: generated samples. The toy discriminator answers 1 when a sample is within 1 of any green clump.

Task 3: cover both clumps

generated sample 10generated sample 20

toy discriminator scores: 0, 0 clumps covered: 0 of 2

Task 4: fool the discriminator but cover only one

generated sample 10generated sample 20

toy discriminator scores: 0, 0 clumps covered: 0 of 2

discriminator loss for D(real)=0.9, D(fake)=0.2 generator loss

To do: The discriminator says D(real) = 0.9 and D(fake) = 0.2. Its loss is −ln D(real) − ln(1 − D(fake)). Type it (3 decimals)
To do: The generator's (non-saturating) loss is −ln D(fake) with D(fake) = 0.2. Type it
To do: The real data has two clumps, at −3 and +3. Place two generated samples so BOTH clumps are covered (within 1)
To do: Now fool the toy discriminator completely (both samples scored 1) while covering only ONE clump

Part 2. Do it in code

d_loss(d_real, d_fake): discriminator loss averaged over the batch: mean(−log(d_real) − log(1 − d_fake)). Inputs are arrays of probabilities strictly between 0 and 1 (ValueError otherwise).
g_loss(d_fake): non-saturating generator loss mean(−log(d_fake)).
d_accuracy(d_real, d_fake): fraction of all samples classified correctly if D > 0.5 means 'real'.
coverage(samples, modes, radius): fraction of the real modes that have at least one sample within `radius`.

import numpy as np

def d_loss(d_real, d_fake):
    pass

def g_loss(d_fake):
    pass

def d_accuracy(d_real, d_fake):
    pass

def coverage(samples, modes, radius):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Does fooling one discriminator prove complete data-distribution coverage?

AYesBNo

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

A VAE uses $z=\mu+\sigma\odot\epsilon$ , with $\epsilon\sim\mathcal N(0,I)$ . A standard diffusion forward marginal is

x_t=\sqrt{\bar\alpha_t}x_0+\sqrt{1-\bar\alpha_t}\epsilon.

Here alpha-bar summarizes retained signal. This is a noising equation, not a complete sampling algorithm.

A VAE optimizes an evidence lower bound:

\operatorname{ELBO}=E_{q_\phi(z\mid x)}[\log p_\theta(x\mid z)]-D_{KL}(q_\phi(z\mid x)\Vert p(z)).

For diagonal Gaussian q and standard normal p, its KL is $\frac12\sum_j(\mu_j^2+\sigma_j^2-1-\log\sigma_j^2)$ . The original GAN minimax objective is

\min_G\max_D E_{x\sim p_{data}}[\log D(x)]+E_{z\sim p(z)}[\log(1-D(G(z)))].

Practical generator losses often use a non-saturating variant. A common diffusion noise-prediction loss is $E\|\epsilon-\epsilon_\theta(x_t,t)\|^2$ under a specified noising schedule.

Worked example

For a scalar diffusion example, take x0=2, alpha-bar=0.64 and epsilon=−1. The signal term is 0.8×2=1.6 and the noise term is 0.6×(−1)=−0.6, giving xt=1.0. The clean value cannot be inferred from xt alone without additional assumptions or a learned model.

For q=N(1,1) against a standard normal prior, the one-dimensional VAE KL is 0.5×(1²+1−1−ln 1)=0.5 nats. This is only the regularization term, not the complete VAE objective.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import sqrt, isclose
x0, alpha, noise = 2,.64,-1
xt=sqrt(alpha)*x0+sqrt(1-alpha)*noise
assert isclose(xt,1)
assert 2+.5*(-1) == 1.5
from math import log
assert isclose(.5*(1**2+1**2-1-log(1**2)),.5)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

A VAE's approximate posterior is not identical to its prior. GAN loss is not a universal sample-quality score. A diffusion noising equation is not sufficient to implement the reverse model.