For the complete documentation index, see llms.txt. This page is also available as Markdown.

Transfer Learning

Transfer learning in Deeplearning4j — TransferLearning.Builder, FineTuneConfiguration, freezing layers, and modifying pretrained networks

Overview

Transfer learning reuses a pretrained model as a starting point for a new task, rather than training from scratch. DL4J's transfer learning API lets you:

  • Freeze layers (hold parameters constant during training)

  • Modify the number of outputs of an existing layer (nOutReplace)

  • Remove layers from an existing model

  • Add new layers

  • Override the learning configuration (learning rate, updater, regularization) for unfrozen layers

The API supports both MultiLayerNetwork and ComputationGraph via TransferLearning.Builder and TransferLearning.GraphBuilder respectively.


Core Classes

Class
Description

TransferLearning.Builder

Modifies a MultiLayerNetwork

TransferLearning.GraphBuilder

Modifies a ComputationGraph

FineTuneConfiguration

Learning hyperparameters applied to all unfrozen layers

TransferLearningHelper

Pre-computes and caches activations at the freeze boundary to speed up training


FineTuneConfiguration

FineTuneConfiguration specifies the training hyperparameters that will be applied to all unfrozen (trainable) layers. Values set here override what was in the original model's configuration.

Key options:

Method
Description

.updater(IUpdater)

Optimizer for unfrozen layers (e.g., new Adam(lr), new Nesterovs(lr, momentum))

.seed(long)

Random seed

.l1(double) / .l2(double)

Regularization for unfrozen layers

.activation(Activation)

Override activation for all unfrozen layers

.dropOut(double)

Dropout retain probability

Note: Newly added layers can specify their own learning rate or updater in their layer builder, which takes priority over FineTuneConfiguration.


TransferLearning.Builder (for MultiLayerNetwork)

Basic Usage

TransferLearning.Builder returns a new MultiLayerNetwork. The original pretrainedNet is not modified.

Key Methods

Method
Description

fineTuneConfiguration(FineTuneConfiguration)

Set learning hyperparameters for unfrozen layers

setFeatureExtractor(int layerNum)

Freeze all layers from 0 up to and including layerNum

nOutReplace(int layerNum, int nOut, WeightInit scheme)

Change nOut of a layer and reinitialize affected weights

nInReplace(int layerNum, int nIn, WeightInit scheme)

Change nIn of a layer

removeOutputLayer()

Remove the last layer (convenience for replacing the output head)

removeLayersFromOutput(int n)

Remove the last n layers

addLayer(Layer layer)

Append a layer (call multiple times to add a stack)

setInputPreProcessor(int layer, InputPreProcessor)

Manually add a pre-processor


TransferLearning.GraphBuilder (for ComputationGraph)

Key Methods

Method
Description

fineTuneConfiguration(FineTuneConfiguration)

Learning config for unfrozen vertices

setFeatureExtractor(String vertexName)

Freeze all vertices up to and including vertexName

nOutReplace(String vertexName, int nOut, WeightInit scheme)

Change nOut of a named vertex

removeVertexKeepConnections(String vertexName)

Remove vertex, preserve its connections in the graph

removeVertexAndConnections(String vertexName)

Remove vertex and all its connections

addLayer(String name, Layer layer, String... inputs)

Add a new layer vertex

addVertex(String name, GraphVertex vertex, String... inputs)

Add a non-layer vertex

setOutputs(String...)

Declare new output vertices


Common Transfer Learning Patterns

Pattern 1: Replace the Classification Head Only

The most common pattern: keep the feature extractor frozen, replace only the final output layer.

After a small number of training iterations (e.g., 30-100), this pattern typically achieves high accuracy because the frozen feature extractor already captures rich visual representations.

Pattern 2: Modify an Intermediate Layer Width and Add New Layers

Important: nOutReplace automatically adjusts the nIn of all downstream layers that receive input from the modified layer. You do not need to update them manually.

Pattern 3: Progressive Unfreezing

Start with more layers frozen, then unfreeze progressively:

Pattern 4: MultiLayerNetwork Transfer Learning


TransferLearningHelper

TransferLearningHelper speeds up transfer learning when you have a large frozen section. Instead of running the full forward pass through frozen layers at every training step, it pre-computes and caches ("featurizes") the activations at the freeze boundary.

This can dramatically reduce training time when the frozen section is computationally expensive (e.g., deep VGG convolutional blocks).

Featurizing a Dataset

Training on Featurized Data

The helper modifies the model in place. The parameters of the unfrozen portion of pretrainedNet are updated with each call to fitFeaturized.

Helper Methods

Method
Description

featurize(DataSet)

Returns a DataSet where inputs are frozen-layer activations

featurize(MultiDataSet)

Multi-input/output version

fitFeaturized(DataSetIterator)

Train the unfrozen head on featurized data

fitFeaturized(MultiDataSetIterator)

Multi-dataset version

outputFromFeaturized(INDArray)

Inference from featurized (post-freeze-boundary) input

unfrozenMLN()

Returns the unfrozen portion as a standalone MultiLayerNetwork

unfrozenGraph()

Returns the unfrozen portion as a standalone ComputationGraph


Using Zoo Models as Starting Points

DL4J's model zoo provides pretrained weights for common architectures:


Important Notes

Frozen Layers Are Not Saved

When you serialize a model with frozen layers using ModelSerializer, the frozen state is not preserved. When you reload the model, no layers will be frozen. To continue training with frozen layers after reloading, you must re-apply the TransferLearning API or TransferLearningHelper.

TransferLearning Returns a New Model

TransferLearning.Builder.build() always returns a new model instance. The original pretrained model is not modified. Keep memory constraints in mind when working with large models.

Changing nOut Cascades to nIn of Downstream Layers

When you call nOutReplace("fc2", 1024, WeightInit.XAVIER), DL4J automatically updates the nIn of every layer that directly receives input from fc2. You do not need to manually specify nIn on those layers. You can optionally specify separate weight init schemes for the modified layer and its downstream consumers:

FineTuneConfiguration Selectively Updates

FineTuneConfiguration only updates the learning parameters of unfrozen layers. Frozen layers retain their original configuration. Newly added layers inherit FineTuneConfiguration unless they specify their own updater/LR in their layer builder.


Saving the Transfer-Learned Model

Last updated

Was this helpful?