Deeplearning4j
Community ForumND4J JavadocDL4J Javadoc
EN 1.0.0-beta7
EN 1.0.0-beta7
  • Eclipse DeepLearning4J
  • Getting Started
    • Quickstart
      • Untitled
    • Tutorials
      • Quickstart with MNIST
      • MultiLayerNetwork And ComputationGraph
      • Logistic Regression
      • Built-in Data Iterators
      • Feed Forward Networks
      • Basic Autoencoder
      • Advanced Autoencoder
      • Convolutional Networks
      • Recurrent Networks
      • Early Stopping
      • Layers and Preprocessors
      • Hyperparameter Optimization
      • Using Multiple GPUs
      • Clinical Time Series LSTM
      • Sea Temperature Convolutional LSTM
      • Sea Temperature Convolutional LSTM 2
      • Instacart Multitask Example
      • Instacart Single Task Example
      • Cloud Detection Example
    • Core Concepts
    • Cheat Sheet
    • Examples Tour
    • Deep Learning Beginners
    • Build from Source
    • Contribute
      • Eclipse Contributors
    • Benchmark Guide
    • About
    • Release Notes
  • Configuration
    • Backends
      • CPU and AVX
      • cuDNN
      • Performance Issues
    • Memory Management
      • Memory Workspaces
    • Snapshots
    • Maven
    • SBT, Gradle, & Others
  • Models
    • Autoencoders
    • Multilayer Network
    • Computation Graph
    • Convolutional Neural Network
    • Recurrent Neural Network
    • Layers
    • Vertices
    • Iterators
    • Listeners
    • Custom Layers
    • Model Persistence
    • Activations
    • Updaters
  • Model Zoo
    • Overview
    • Zoo Models
  • ND4J
    • Overview
    • Quickstart
    • Basics
    • Elementwise Operations
    • Matrix Manipulation
    • Syntax
    • Tensors
  • SAMEDIFF
    • Importing TensorFlow models
    • Variables
    • Ops
    • Adding Ops
  • ND4J & SameDiff Ops
    • Overview
    • Bitwise
    • Linalg
    • Math
    • Random
    • BaseOps
    • CNN
    • Image
    • Loss
    • NN
    • RNN
  • Tuning & Training
    • Evaluation
    • Visualization
    • Trouble Shooting
    • Early Stopping
    • t-SNE Visualization
    • Transfer Learning
  • Keras Import
    • Overview
    • Get Started
    • Supported Features
      • Activations
      • Losses
      • Regularizers
      • Initializers
      • Constraints
      • Optimizers
    • Functional Model
    • Sequential Model
    • Custom Layers
    • API Reference
      • Core Layers
      • Convolutional Layers
      • Embedding Layers
      • Local Layers
      • Noise Layers
      • Normalization Layers
      • Pooling Layers
      • Recurrent Layers
      • Wrapper Layers
      • Advanced Activations
  • DISTRIBUTED DEEP LEARNING
    • Introduction/Getting Started
    • Technical Explanation
    • Spark Guide
    • Spark Data Pipelines Guide
    • API Reference
    • Parameter Server
  • Arbiter
    • Overview
    • Layer Spaces
    • Parameter Spaces
  • Datavec
    • Overview
    • Records
    • Reductions
    • Schema
    • Serialization
    • Transforms
    • Analysis
    • Readers
    • Conditions
    • Executors
    • Filters
    • Operations
    • Normalization
    • Visualization
  • Language Processing
    • Overview
    • Word2Vec
    • Doc2Vec
    • Sentence Iteration
    • Tokenization
    • Vocabulary Cache
  • Mobile (Android)
    • Setup
    • Tutorial: First Steps
    • Tutorial: Classifier
    • Tutorial: Image Classifier
    • FAQ
    • Press
    • Support
    • Why Deep Learning?
Powered by GitBook
On this page
  • Network Configurations
  • Required imports
  • Building a MultiLayerConfiguration
  • What we did here?
  • Sanity checking for our MultiLayerConfiguration
  • Creating a MultiLayerNetwork
  • Building a ComputationGraphConfiguration
  • What we did here?
  • Sanity checking for our ComputationGraphConfiguration
  • Creating a ComputationGraph
  • More MultiLayerConfiguration Examples
  • Regularization
  • Dropout connects
  • Bias initialization
  • More ComputationGraphConfiguration Examples
  • Recurrent Network
  • Multiple Inputs and Merge Vertex
  • Multi-Task Learning

Was this helpful?

Edit on Git
Export as PDF
  1. Getting Started
  2. Tutorials

MultiLayerNetwork And ComputationGraph

DL4J provides the following classes to configure networks:

  1. MultiLayerNetwork

  2. ComputationGraph

MultiLayerNetwork consists of a single input layer and a single output layer with a stack of layers in between them.

ComputationGraph is used for constructing networks with a more complex architecture than MultiLayerNetwork. It can have multiple input layers, multiple output layers and the layers in between can be connected through a direct acyclic graph.

Network Configurations

Whether you create MultiLayerNetwork or ComputationGraph, you have to provide a network configuration to it through NeuralNetConfiguration.Builder. As the name implies, it provides a Builder pattern to configure a network. To create a MultiLayerNetwork, we build a MultiLayerConfiguraionand for ComputationGraph, it’s ComputationGraphConfiguration.

The pattern goes like this: [High Level Configuration] -> [Configure Layers] -> [Build Configuration]

Required imports

import org.deeplearning4j.nn.conf.graph.MergeVertex
import org.deeplearning4j.nn.conf.layers.{DenseLayer, LSTM, OutputLayer, RnnOutputLayer}
import org.deeplearning4j.nn.conf.{ComputationGraphConfiguration, MultiLayerConfiguration, NeuralNetConfiguration}
import org.deeplearning4j.nn.graph.ComputationGraph
import org.deeplearning4j.nn.multilayer.MultiLayerNetwork
import org.deeplearning4j.nn.weights.WeightInit
import org.nd4j.linalg.activations.Activation
import org.nd4j.linalg.learning.config.Nesterovs
import org.nd4j.linalg.lossfunctions.LossFunctions

Building a MultiLayerConfiguration

val multiLayerConf: MultiLayerConfiguration = new NeuralNetConfiguration.Builder()
  .seed(123)
  .updater(new Nesterovs(0.1, 0.9)) //High Level Configuration
  .list() //For configuring MultiLayerNetwork we call the list method
  .layer(0, new DenseLayer.Builder().nIn(784).nOut(100).weightInit(WeightInit.XAVIER).activation(Activation.RELU).build()) //Configuring Layers
  .layer(1, new OutputLayer.Builder(LossFunctions.LossFunction.XENT).nIn(100).nOut(10).weightInit(WeightInit.XAVIER).activation(Activation.SIGMOID).build())
  .build() //Building Configuration

What we did here?

High Level Configuration

Function

Details

seed

For keeping the network outputs reproducible during runs by initializing weights and other network randomizations through a seed

updater

Algorithm to be used for updating the parameters. The first value is the learning rate, the second one is the Nesterov's momentum.

Configuration of Layers

Here we are calling list() to get the ListBuilder. It provides us the necessary api to add layers to the network through the layer(arg1, arg2) function.

  • The first parameter is the index of the position where the layer needs to be added.

  • The second parameter is the type of layer we need to add to the network.

To build and add a layer we use a similar builder pattern as:

Function

Details

nIn

The number of inputs coming from the previous layer. (In the first layer, it represents the input it is going to take from the input layer)

nOut

he number of outputs it’s going to send to the next layer. (For output layer it represents the labels here)

weightInit

The type of weights initialization to use for the layer parameters.

activation

The activation function between layers

Building a Graph

Finally, the last build() call builds the configuration for us.

Sanity checking for our MultiLayerConfiguration

You can get your network configuration as String, JSON or YAML for sanity checking. For JSON we can use the toJson() function.

println(multiLayerConf.toJson)

Creating a MultiLayerNetwork

Finally, to create a MultiLayerNetwork, we pass the configuration to it as shown below

val multiLayerNetwork : MultiLayerNetwork = new MultiLayerNetwork(multiLayerConf)

Building a ComputationGraphConfiguration

val computationGraphConf : ComputationGraphConfiguration = new NeuralNetConfiguration.Builder()
      .seed(123)
      .updater(new Nesterovs(0.1, 0.9)) //High Level Configuration
      .graphBuilder()  //For configuring ComputationGraph we call the graphBuilder method
      .addInputs("input") //Configuring Layers
      .addLayer("L1", new DenseLayer.Builder().nIn(3).nOut(4).build(), "input")
      .addLayer("out1", new OutputLayer.Builder().lossFunction(LossFunctions.LossFunction.NEGATIVELOGLIKELIHOOD).nIn(4).nOut(3).build(), "L1")
      .addLayer("out2", new OutputLayer.Builder().lossFunction(LossFunctions.LossFunction.MSE).nIn(4).nOut(2).build(), "L1")
      .setOutputs("out1","out2")
      .build() //Building configuration

What we did here?

The only difference here is the way we are building layers. Instead of calling the list() function, we call the graphBuilder() to get a GraphBuilder for building our ComputationGraphConfiguration. Following table explains what each function of a GraphBuilder does.

Function

Details

addInputs

A list of strings telling the network what layers to use as input layers

addLayer

First parameter is the layer name, then the layer object and finally a list of strings defined previously to feed this layer as inputs

setOutputs

A list of strings telling the network what layers to use as output layers

The output layers defined here use another function lossFunction to define what loss function to use.

Sanity checking for our ComputationGraphConfiguration

You can get your network configuration as String, JSON or YAML for sanity checking. For JSON we can use the toJson() function

println(computationGraphConf.toJson)

Creating a ComputationGraph

Finally, to create a ComputationGraph, we pass the configuration to it as shown below

val computationGraph : ComputationGraph = new ComputationGraph(computationGraphConf)

More MultiLayerConfiguration Examples

Regularization

//You can add regularization in the higher level configuration in the network 
// configuring a regularization algorithm -> 'l1()', l2()' etc as shown 
// below:
new NeuralNetConfiguration.Builder()
    .l2(1e-4)

Dropout connects

//When creating layers, you can add a dropout connection by using 
// 'dropout(<dropOut_factor>)'
new NeuralNetConfiguration.Builder()
    .list() 
    .layer(0, new DenseLayer.Builder().dropOut(0.8).build())

Bias initialization

//You can initialize the bias of a particular layer by using 
// 'biasInit(<init_value>)'
new NeuralNetConfiguration.Builder()
    .list() 
    .layer(0, new DenseLayer.Builder().biasInit(0).build())

More ComputationGraphConfiguration Examples

Recurrent Network

with Skip Connections

val cgConf1 : ComputationGraphConfiguration = new NeuralNetConfiguration.Builder()
        .graphBuilder()
        .addInputs("input") //can use any label for this
        .addLayer("L1", new LSTM.Builder().nIn(5).nOut(5).build(), "input")
        .addLayer("L2",new RnnOutputLayer.Builder().nIn(5+5).nOut(5).build(), "input", "L1")
        .setOutputs("L2")
        .build();

Multiple Inputs and Merge Vertex

//Here MergeVertex concatenates the layer outputs
val cgConf2 : ComputationGraphConfiguration = new NeuralNetConfiguration.Builder()
        .graphBuilder()
        .addInputs("input1", "input2")
        .addLayer("L1", new DenseLayer.Builder().nIn(3).nOut(4).build(), "input1")
        .addLayer("L2", new DenseLayer.Builder().nIn(3).nOut(4).build(), "input2")
        .addVertex("merge", new MergeVertex(), "L1", "L2")
        .addLayer("out", new OutputLayer.Builder().nIn(4+4).nOut(3).build(), "merge")
        .setOutputs("out")
        .build();

Multi-Task Learning

val cgConf3 : ComputationGraphConfiguration = new NeuralNetConfiguration.Builder()
        .graphBuilder()
        .addInputs("input")
        .addLayer("L1", new DenseLayer.Builder().nIn(3).nOut(4).build(), "input")
        .addLayer("out1", new OutputLayer.Builder()
                .lossFunction(LossFunctions.LossFunction.NEGATIVELOGLIKELIHOOD)
                .nIn(4).nOut(3).build(), "L1")
        .addLayer("out2", new OutputLayer.Builder()
                .lossFunction(LossFunctions.LossFunction.MSE)
                .nIn(4).nOut(2).build(), "L1")
        .setOutputs("out1","out2")
        .build();
PreviousQuickstart with MNISTNextLogistic Regression

Last updated 5 years ago

Was this helpful?