For the complete documentation index, see llms.txt. This page is also available as Markdown.

Autoencoders

Autoencoder and Variational Autoencoder layers in Deeplearning4j — architecture, configuration, and training

Autoencoders are neural networks trained to reconstruct their inputs through a compressed latent representation. Eclipse Deeplearning4j supports a denoising AutoEncoder layer and a full VariationalAutoencoder (VAE) layer with configurable reconstruction distributions.

Note: Restricted Boltzmann Machines (RBMs) were removed in version 0.9.x and are no longer supported.

AutoEncoder Layer

The AutoEncoder layer (org.deeplearning4j.nn.conf.layers.AutoEncoder) is a denoising autoencoder. It adds random noise (corruption) to the input during training, then learns to reconstruct the clean original. This forces the network to learn a robust representation.

Key Parameters

Parameter
Method
Description

Corruption level

corruptionLevel(double)

Fraction of input values zeroed out during training. Range 0.0 (none) to 1.0 (all). Typical: 0.3

Sparsity

sparsity(double)

Sparsity regularization penalty. Encourages few active hidden units.

Standard layer parameters (nIn, nOut, activation, weightInit, updater, etc.) all apply.

Configuration Example

import org.deeplearning4j.nn.conf.MultiLayerConfiguration;
import org.deeplearning4j.nn.conf.NeuralNetConfiguration;
import org.deeplearning4j.nn.conf.layers.AutoEncoder;
import org.deeplearning4j.nn.conf.layers.OutputLayer;
import org.deeplearning4j.nn.multilayer.MultiLayerNetwork;
import org.nd4j.linalg.activations.Activation;
import org.nd4j.linalg.lossfunctions.LossFunctions;

int inputSize  = 784;  // e.g. MNIST 28x28
int hiddenSize = 256;

MultiLayerConfiguration conf = new NeuralNetConfiguration.Builder()
    .updater(new org.nd4j.linalg.learning.config.Adam(1e-3))
    .list()
    .layer(new AutoEncoder.Builder()
        .nIn(inputSize).nOut(hiddenSize)
        .activation(Activation.RELU)
        .corruptionLevel(0.3)
        .sparsity(0.0)
        .build())
    // Tie weights back to reconstruction by adding a second AutoEncoder layer reversed,
    // or simply use a DenseLayer + OutputLayer for the decoder portion:
    .layer(new OutputLayer.Builder(LossFunctions.LossFunction.MSE)
        .nIn(hiddenSize).nOut(inputSize)
        .activation(Activation.SIGMOID)
        .build())
    .build();

MultiLayerNetwork model = new MultiLayerNetwork(conf);
model.init();

During training the model minimises reconstruction loss on the corrupted-then-decoded output.


VariationalAutoencoder Layer

The VariationalAutoencoder layer (org.deeplearning4j.nn.conf.layers.variational.VariationalAutoencoder) implements the VAE described in Kingma & Welling (2013), "Auto-Encoding Variational Bayes". It supports multiple encoder and decoder hidden layers, a configurable latent space size, and several reconstruction distributions.

Key ideas:

  • The encoder maps input x to a distribution q(z|x) over latent variable z.

  • A latent code z is sampled from q(z|x).

  • The decoder maps z back to a reconstruction p(x|z).

  • The training objective maximises the variational lower bound (ELBO).

Score sign convention: DL4J minimises the negative of the variational lower bound, so reported scores during pretraining are negative values of the ELBO described in the paper.

Builder Parameters

Method
Description

encoderLayerSizes(int...)

Sizes of hidden layers in the encoder. Each acts like a DenseLayer.

decoderLayerSizes(int...)

Sizes of hidden layers in the decoder. Typically mirrors the encoder.

nOut(int)

Size of the latent space Z.

reconstructionDistribution(ReconstructionDistribution)

Distribution used to model p(x|z). See distributions below.

pzxActivationFunction(Activation)

Activation for the mean/log-variance output feeding into p(z|x). Avoid bounded activations like RELU. Use TANH or IDENTITY.

numSamples(int)

Number of latent samples per data point during pretraining (default 1).

lossFunction(IActivation, ILossFunction)

Alternative: use a deterministic loss function instead of a reconstruction distribution.

Configuration Example


Reconstruction Distributions

The reconstruction distribution defines how the decoder output is interpreted when computing the reconstruction loss. Choose based on the nature of your data.

GaussianReconstructionDistribution

Models each output dimension as an independent Gaussian with learned mean and log-variance. Appropriate for continuous real-valued data.

The network learns both mean and log(variance) for each output. Avoid asymmetric activations like RELU or SIGMOID as the distribution parameter space is (-inf, inf).

BernoulliReconstructionDistribution

Models each output dimension as a Bernoulli random variable. Appropriate for binary data (pixel values 0 or 1).

The decoder output is passed through a sigmoid to produce probabilities. Do not use RELU, TANH, or other non-sigmoid activations — the output must be in [0, 1].

ExponentialReconstructionDistribution

Models outputs using an exponential distribution. Appropriate for data in range [0, infinity), such as waiting times or count data.

The network models gamma = log(lambda), so the parameterisation is unconstrained and IDENTITY or TANH are appropriate activations.

CompositeReconstructionDistribution

Combines multiple distributions for datasets with mixed data types (e.g., some continuous columns, some binary columns).

Distributions are applied to contiguous slices of the output in the order they are added.

LossFunctionWrapper

Allows using a standard loss function (e.g., MSE) in place of a probabilistic reconstruction distribution. This is not standard VAE design but is valid when a probabilistic interpretation is not required.

Note: reconstruction log-probability cannot be computed when using LossFunctionWrapper.


Training Patterns

Pretraining (Unsupervised)

Call pretrain(iterator) to train using the VAE's generative objective (ELBO maximisation):

During pretraining only the VAE layer parameters are updated. Add additional layers after the VAE for downstream classification or regression tasks.

Reconstruction and Generation

After training, use the underlying org.deeplearning4j.nn.layers.variational.VariationalAutoencoderParamInitializer API via the layer itself:

Fine-tuning After Pretraining

Stack a classification head on top and fine-tune with supervised training:

Choosing numSamples

The numSamples parameter controls how many latent samples are drawn per data point during pretraining. Kingma & Welling note that numSamples = 1 is sufficient when the minibatch size is large (e.g., >= 100). Increasing numSamples reduces variance in the gradient estimate but increases computation cost proportionally.


API Reference

Class
Package

AutoEncoder

org.deeplearning4j.nn.conf.layers

VariationalAutoencoder (config)

org.deeplearning4j.nn.conf.layers.variational

VariationalAutoencoder (layer)

org.deeplearning4j.nn.layers.variational

GaussianReconstructionDistribution

org.deeplearning4j.nn.conf.layers.variational

BernoulliReconstructionDistribution

org.deeplearning4j.nn.conf.layers.variational

ExponentialReconstructionDistribution

org.deeplearning4j.nn.conf.layers.variational

CompositeReconstructionDistribution

org.deeplearning4j.nn.conf.layers.variational

LossFunctionWrapper

org.deeplearning4j.nn.conf.layers.variational

ReconstructionDistribution (interface)

org.deeplearning4j.nn.conf.layers.variational

Last updated

Was this helpful?