Variational Autoencoders

A model that reconstructs known examples does not necessarily give us a good way to generate new ones. A variational autoencoder also shapes a distribution of compressed representations from which we can draw new codes.

14 sections · 10 questions · 5 coding challenges

1 / 14

The arrow keys move between sections too.

Could We Choose a New Code and Obtain a Plausible Example?

BOTIE

An ordinary autoencoder can give each training example a code, but that does not tell us where to choose a new code for generation. Unused regions between familiar codes may decode poorly. We need a model of how the hidden codes themselves are distributed.

Kingma and Welling's 2013 work developed efficient learning and approximate inference for models with continuous latent variables. A key contribution was reparameterising a random draw so that gradient-based training could adjust the distribution's parameters.

The encoder describes a distribution of possible codes for an input. The decoder learns to reconstruct from samples, while the objective encourages those code distributions to remain compatible with a chosen prior. We can later draw from that prior and decode. Reconstruction quality and the distribution constraint compete, so the method learns an organised generative representation rather than simply assigning an unrelated address to each example.

Sources for this history

Kingma and Welling (2013), Auto-Encoding Variational Bayes

Which Hidden Codes Should We Use to Create Something New?

An autoencoder gives each input a code. We might try generating a new example by choosing a new code and decoding it. But which codes should we choose? A decoder trained on isolated codes may behave poorly in the spaces between them.

A variational autoencoder gives us a more deliberate arrangement. For each input, the encoder describes a distribution of possible codes. Training balances reconstruction with a preference for code distributions that stay close to a simple shared prior.

Step 1 of 4

Replace one code with a distribution of codes

Instead of emitting a single latent value, the encoder describes a range of likely values. In the Gaussian case, each coordinate has a mean and a standard deviation. The mean sets the center; the deviation describes how spread out possible samples are.

1
Encoder reads
One input example
2
Encoder returns
- Mean
- Standard deviation
3
These describe
A distribution of possible codes

Two outputs describe one latent coordinate.

Step 2 of 4

Draw a code before decoding

A sample gives the decoder one concrete code to use. Different samples from the same distribution can produce different reconstructions. The reconstruction part of training asks the encoder and decoder to keep the input recoverable despite this variation.

1
Distribution
Possible codes
2
Random draw
One sampled code
3
Decoder
A reconstruction

A distribution is not itself the decoder's input sample.

Step 3 of 4

Give the code space a shared reference

The prior is a distribution we choose as the source of codes when generating new examples. A common choice is a standard normal distribution. A KL-divergence term penalizes each inferred code distribution for differing from that prior. It competes with reconstruction, rather than replacing it.

Reconstruction term: Preserve information about the input
KL term: Stay near the chosen prior

The two training pressures serve different purposes.

Step 4 of 4

Generate without an input to reconstruct

After training, we can draw a code from the prior and pass it to the decoder. The encoder is used to infer codes from existing inputs during training and reconstruction; it is not needed for this prior-sampling route.

Reconstruction: Input
Encoder
Sample
Decoder
Generation: Prior sample
Decoder
New output

Two routes through the trained system.

Make the random draw differentiable with respect to its parameters

The next calculation separates the random noise draw from the mean and deviation. That is the reparameterization step used to train the encoder.

The interactive example isolates that step and the KL term. A full VAE also needs an encoder, decoder, and reconstruction likelihood.

Part 1. Describe a Distribution Before Drawing From It

1. Distinguish the inferred distribution from the prior

The encoder's distribution depends on the input. It describes plausible latent codes for that particular observation and is often called an approximate posterior. The prior is a separate distribution chosen as the source of codes before an observation is given.

In the common diagonal Gaussian setup, the encoder supplies a mean and variance for each coordinate. Diagonal means the coordinates are modeled as independent within this conditional Gaussian. Implementations often predict log variance so the variance remains positive after exponentiation.

2. Give reconstruction and regularization different jobs

The decoder describes how an input can be generated from a code. The reconstruction term rewards codes that help account for the observed input. The KL term measures how far the inferred code distribution is from the prior.

These objectives compete. A code that ignores the input can match the prior well but reconstruct poorly. An overly powerful decoder can also learn to ignore its latent code, a failure called posterior collapse. Merely making KL small does not establish a useful representation.

Part 2. Sampling and the Variational Objective

3. Separate the random draw from the encoder's parameters

Let μ be the mean, σ the standard deviation, and ε a standard normal noise draw. Instead of asking an opaque sampler to draw directly from the encoder's Gaussian, express the sample as a transformation of independent noise.

ε ~ Normal(0, 1)
z = μ + σ ⊙ ε

Illustrative coordinate:
μ = 1,  σ = 0.5,  ε = −0.4
z = 1 + (0.5 × (−0.4)) = 0.8

The symbol ⊙ means coordinate-by-coordinate multiplication. For a fixed noise draw, the code changes smoothly with the mean and deviation, so gradients can pass from the decoder back into those parameters. This is the reparameterization trick.

4. Write the two terms of the loss

Use q for the encoder's approximate posterior and p for the prior and decoder distributions. The following negative evidence lower bound is minimized. The expectation averages reconstruction likelihood over sampled codes.

L(x) = E_{z ~ q_φ(z|x)}[−log p_θ(x|z)]
       + KL(q_φ(z|x) || p(z))

For a diagonal Gaussian q and standard normal p:
KL = ½ Σⱼ [μⱼ² + σⱼ² − 1 − log(σⱼ²)]

The first term depends on the decoder's observation model. A fixed-variance Gaussian likelihood leads to a scaled squared-error reconstruction term plus constants. Other observations require different likelihood choices.

For a mean of zero and deviation of one, the approximate posterior matches the standard normal prior and this KL is zero. Changing a particular noise draw does not change KL because KL compares distributions, not sampled points.

The KL term for the illustrative coordinate

Take the coordinate from step 3, with a mean of 1 and a deviation of 0.5. Its variance is 0.25, and the logarithm of a number below one is negative, so the last term inside the bracket adds to the total rather than taking from it.

μ² = 1,  σ² = 0.25,  log(σ²) ≈ −1.3863

KL = ½ (1 + 0.25 − 1 − (−1.3863))
   = ½ × 1.6363
   ≈ 0.8181

Every noise draw leaves this number where it is. A draw of −2 gives a sample of 0, a draw of 0 gives the mean itself, and a draw of 2 gives a sample of 2, while the KL term stays at 0.8181 for all three, because the distribution they were drawn from has not moved.

Holding the deviation at one and moving only the mean, the term grows with the square of the mean. A mean of 1 or −1 costs 0.5, and a mean of 2 or −2 costs 2.0. The term reaches zero only where the mean is zero and the deviation is one together, so a posterior pays for being off-center and for being the wrong width, and the reconstruction term is what has to make either worth paying for.

Part 3. From the Latent Operation to a VAE

5. Inspect one posterior directly

The SDK object exposes the reparameterized sample and exact Gaussian KL. The live example keeps the chosen draw visible so the distinction between sample and distribution is testable.

import numpy as np
from oop_ml.numpy.modern import DiagonalGaussian

posterior = DiagonalGaussian(
    mean=np.array([1.0]), deviation=np.array([0.5]),
)
print(posterior.sample(np.array([-0.4])))
print(posterior.kl_to_standard_normal)

This is the latent-distribution operation, not a trained VAE. A complete implementation must connect it to an encoder and decoder, define the reconstruction likelihood, sample during training, and propagate the resulting gradients.

6. Use the prior when generating

After fitting the model, draw a new code from the prior and decode it. There is no input image to encode on this route. Sampling more codes produces a distribution of outputs rather than a single reconstruction.

A latent coordinate does not automatically have an interpretable meaning such as rotation or color. Those meanings depend on the data, architecture, objective, and learned representation. Inspect them instead of assigning meanings in advance.

The original VAE paper derives the variational objective and the reparameterized gradient estimator. Auto-Encoding Variational Bayes.

Continue with Generative Adversarial Networks.

Part 4. Connect an Encoder to a Decoder

The distribution operation gives us a code, but it does not yet teach a model to represent observations. Let's connect the missing pieces using four measurements and one latent coordinate. Keeping the model linear makes every part small enough to inspect.

Training observations x = [-2, -1, 1, 2]
Encoder mean μ(x) = a × x + b
Encoder variance v = exp(c)
Code z = μ(x) + sqrt(v) × ε; ε ~ Normal(0,1)
Decoder mean = w × z + d
Observation model: x | z ~ Normal(w × z + d, 1)
Prior: z ~ Normal(0,1)

There are five learned numbers. The encoder slope and bias decide where each observation is represented. Its log variance decides how widely codes vary around that location. The decoder slope and bias turn a code back into a predicted observation. The observation variance stays fixed at one in this example.

The encoder variance is shared across inputs here. Larger VAEs usually predict a variance for each input as well as its mean. We are using a small restricted model so we can complete training and generation without hiding a second network inside a library call.

Part 5. Give All Five Parameters a Training Signal

The decoder should be able to reconstruct the observations from sampled codes. For this linear decoder with Gaussian codes, we can calculate the expected squared reconstruction error exactly. The code variance contributes to that expectation: getting the mean's reconstruction right does not make every sampled reconstruction right.

μ = a × x + b
v = exp(c)
e = x - (w × μ + d)

Expected reconstruction term = ½(e² + w²v)
KL term = ½(μ² + v - 1 - c)
Loss = average of both terms over the observations

This is the negative evidence lower bound with a constant omitted. Ordinary nonlinear VAEs estimate the reconstruction expectation using sampled codes. Here the exact expectation lets us investigate the same two competing terms without sampling noise in the training curve.

Each parameter needs the derivative of this complete objective. First calculate the signal arriving at the encoder mean. Then apply the chain rule to the slope and bias that produced it. Average per-observation derivatives before updating.

Signal at μ: r = -w × e + μ

dL/da = average(r × x)
dL/db = average(r)
dL/dw = average(-e × μ + w × v)
dL/dd = average(-e)
dL/dc = ½(w²v + v - 1)

parameter_new = parameter - learning_rate × derivative

The final derivative uses the fact that the derivative of exp(c) is exp(c). A numerical gradient check can perturb each of the five parameters independently and compare the resulting loss change with these expressions.

BOTIE

These four measurements are our training data. At first the encoder and decoder know very little about them. Take some training steps, then compare each reconstruction with its input. After training, a prior code can enter the decoder without an input measurement.

Training observations: -2, -1, 1, 2. One latent coordinate; standard-normal prior; observation variance one. This linear Gaussian teaching model calculates its expected reconstruction loss exactly.

Updates: 0. Average objective: 1.1995.

Input x	Encoder mean	Decoder mean at that code
-2	-0.400	-0.120
-1	-0.200	-0.060
1	0.200	0.060
2	0.400	0.120

Posterior variance: 1.000. Reconstructions need not copy every input: the objective also charges for departing from the prior.

Choose a prior code z: 0.0

Decoder mean for this code: 0.000. A random generated observation would also include a noise draw with variance one. The slider chooses a code manually; it is not a random sampler.

Part 6. Generate Without Reusing a Training Observation

Training uses an observation to obtain an encoder distribution. Generation takes another route. Draw a code from the standard-normal prior and pass it directly to the trained decoder. For this probabilistic observation model, add an independent standard-normal observation-noise draw to obtain a sampled measurement.

z ~ Normal(0,1)
η ~ Normal(0,1), independently
generated measurement = w × z + d + η

A decoder mean is useful to inspect, but it is different from a sampled observation. Likewise, a reconstruction of an encoded training measurement is different from a new draw through the prior. Compare all three before judging what the model learned.

Our four values cannot establish the shape of a real population. The model can only describe a Gaussian marginal with this linear decoder, and its fixed observation variance limits how tightly it can reconstruct. Try data with two separated groups: a single broad distribution can place generated values in the gap. That failure motivates more expressive decoders rather than declaring a lower training loss sufficient.