1 of 100

EN 1.0.0-M2.1

Deeplearning4j Suite Overview

Introduction to core Deeplearning4j concepts.

Eclipse DeepLearning4J

Eclipse Deeplearning4j is a suite of tools for running deep learning on the JVM. It's the only framework that allows you to train models from java while interoperating with the python ecosystem through a mix of python execution via our cpython bindings, model import support, and interop of other runtimes such as tensorflow-java and onnxruntime.

Consider going to our Quickstart for an overview of where to get started. If you have dependency issues please use our Required Dependencies guide.

The use cases include importing and retraining models (Pytorch, Tensorflow, Keras) models and deploying in JVM Micro service environments, mobile devices, IoT, and Apache Spark. It is a great compliment to your python environment for running models built in python, deployed to or packaged for other environments.

Deeplearning4j has several submodules including:

Samediff: a tensorflow/pytorch like framework for execution of complex graphs. This framework is lower level, but very flexible. It's also the base api for running onnx and tensorflow graphs.
Nd4j: numpy ++ for java. Contains a mix of numpy operations and tensorflow/pytorch operations.
Libnd4j: A lightweight, standalone c++ library enable math code to run on different devices. Optimizable for running on a wide variety of devices.
Python4j: A python script execution framework easing deployment of python scripts in to production.
Apache Spark Integration: An integration with the Apache Spark framework enabling execution of deep learning pipelines on spark
Datavec: A data transformation library converting raw input data to tensors suitable for running neural networks on.

How to use this website

Multi project contains all cross project documentation such as end to end training and other whole project related documentation. This should be the default entry point for those getting started.
Deeplearning4j contains all of the documentation related to the core deeplearning4j apis such as the multi layer network and the computation graph. Consider this the high level framework for building neural networks. If you would like something lower level like tensorflow or pytorch, consider using samediff
Samediff contains all the documentation related to the samediff submodule of ND4j. Samediff is a lower level api for building neural networks similar to pytorch or tensorflow with built in automatic differentiation.
Datavec contains all the documentation related to our data transformation library datavec.
Python4j contains all the documentation related to our cpython execution framework python4j.
Libnd4j contains all the documentation related to our underlying C++ framework libnd4j.
Apache Spark contains all of the documentation related to our Apache Spark integration.
Concepts/Theory contains all of the documentation related to general mathematical or computer science theory needed to understand various aspects of the framework.

Open Source

JVM/Python/C++

Deeplearning4j can either be a compliment to your existing workflows in python and c++ or a standalone library for you to build and deploy models. Use what components you find useful.

Release Notes

1.0.0-M2

Highlights

As part of the same work flatbuffers has been upgraded to 1.12.1. This affects the samediff file format and the user interfaces. Flatbuffers as a file format is forwards and backwards compatible but if you have any issues please do let us know. The relevant files have been updated using the flatc compiler.

Removed rl4j: in continuing to cut unmaintained modules, the 1.0 will focus the framework on a few key use cases. This invites other folks to build external modules for a tightly maintained core that focuses on deployment, framework interop and training models in java.

Consolidated tests to platform-tests to allow for easy testing of behavior against different backends.

Adds proper support for jetson nano with curated binaries and an updated cuda 10.2

Nd4j/Samdiff/Libnd4j

Features and Enhancements

Bug Fixes

Update samediff api to allow dimensions as variables

Deeplearning4j

Features and Enhancements

Bug Fixes

Datavec

Features and Enhancements

Bug Fixes

Omnihub

Launches new Omnihub module. Allows access to models from: https://github.com/KonduitAI/omnihub-zoo

A pretrained omnihub module will provide access to pretrained samediff and dl4j modules. This will also supplant the old dl4j zoo.

Python4j

Clean up tests/consolidate tests to platform-tests

1.0.0-M2.1

Highlights

Significant changes for views being used within nd4j and samediff via the create_view op allow for users to directly leverage views in combination with graph operations reducing allocations and allowing for increased performance.

Cuda 11.4 and 11.6 added. Add nd4j-cuda-11.4 and nd4j-cuda-11.6 to your dependencies.

AliasWithName
CumSum
GenerateProposals
Loop
ResizeNearest
SequenceAt
SequenceConstruct
SequenceEmpty
SequenceErase
SequenceInsert
SequenceLength

3D Convolution Pooling serialization fixes for keras

Significant changes for migration of NEC Aurora backend to VEDA:

Upgrade protobuf to 3.21.2

Removes spark 2 support.

Nd4j/Samdiff/Libnd4j

Features and Enhancements

Add label serialization for multi

Bug Fixes

Deeplearning4j

Features and Enhancements

Bug Fixes

Datavec

Omnihub

Python4j

1.0.0-M1.1

Highlights

A number of bug fixes following the M1 release, thanks to the feedback from the community, allowed us to quickly sort out a few issues. This is a minor bug fix release to address short comings found with M1. Most fixes were related to keras import, the cnn/rnn helpers, and python4j.

Added backwards compatibility for centos 6 via a new linux-x86_64-compat classifier enabling use of older glibcs on centos 7:

Known issues

Deeplearning4j

Features and Enhancements

Bug fixes

Nd4j

Features and Enhancements

Bug fixes

Datavec

Features and Enhancements

Bug fixes

Python4j

Features and Enhancements

Bug fixes

Samediff

Features and Enhancements

Bug fixes

1.0.0-M1

Highlights

In light of the coming 1.0, the project has decided to cut a number of modules before the final release. These modules have not had many users in the past and have created confusion for many users just trying to use a few simple apis. Many of these modules have not been maintained.

There will likely be 1 or 2 more milestone releases before the final 1.0. These should be considered checkpoints.

These modules include:

Arbiter
Jumpy
Datavec modules for video, audio, audio, sound. The computer vision datavec module
will continue to be available.
Tokenizers: The tokenizers for chinese, japanese, korean were imported from other frameworks
and not really updated.
Scalnet, Nd4s: We removed the scala modules due to the small user base. We welcome 3rd party enhancements
to the framework for syntatic sugar such as kotlin and scala. The framework's focus will be on providing

TVM: We now support running TVM modules. Docs coming soon.

We've updated our shaded modules to newer versions to mitigate security risks. These modules include: 1. jackson 2. guava

Cuda 11: We've upgraded dl4j and associated modules to support cuda 11 and 11.2.

A more modular model import framework supporting tensorflow and onnx: 1. Model mapping procedures loadable as protobuf 2. Defining custom rules for import to work around unsupported or custom layers/operations 3. Op descriptor for all operations in nd4j

This will enable users to override model import behavior to run their own custom models. This means, in most circumstances, there will be no need to modify model import core code anymore. Instead, users will be able to provide definitions and custom rules for their graphs.

Users will be expected to convert their models in an external process. This means running standalone conversions for their models. This extends to keras import as well. Sometimes users convert their models in production directly from keras.

The workflow going forward is to ensure that your model is converted ahead of time to avoid performance issues with converting large models.

Removed ppc from nd4j-native-platform and nd4j-cuda-platform. If you need this architecture, please contact us or build from source.

We've upgraded arrow to 4.0.0 enabling the associated nd4j-arrow and datavec-arrow modules to be used without netty clashes.

Deeplearning4j

Bug fixes

Improved keras model import support for NWHC as well as NCHW input formats for both rnn and cnn

Nd4j

Features and Enhancements

Bug fixes

Python4j

Features and Enhancements

Rewritten and more stable python execution. This allows better support for multi threaded environments.

Bug fixes

1.0.0-beta7

Version 1.0.0-beta7

Deeplearning4j

Features and Enhancements

- Full inference and training support is available for ops/layers in the tf.keras namespace; inference only for general Tensorflow operations outside of the tf.keras namespace
- Note also improvements to Keras import for reshape, permute, etc operations due to NHWC and NWC support in DL4J

Bug Fixes and Optimizations

ND4J/SameDiff:

Features and Enhancements

Added new Image operations namespace operations:
Added new Random operations namespace operations:
Added new Math namespace operations:
Added new NN namespace operations:
Added new CNN namespace operations:
Added new linalg operations namespace
Added new RNN operation namespace operations:
Mapped operations for Tensorflow import:

Bug Fixes and Optimizations

Improved performance for bias_add operation

DataVec

Features and Enhancements

Bug Fixes and Optimizations

RL4J

Features and Enhancements

Arbiter

Bug Fixes and Optimizations

1.0.0-beta6

Highlights - 1.0.0-beta6 Release

Added support for CUDA 10.2. 1.0.0-beta6 released with CUDA 9.2, 10.0, 10.1 and 10.2 support
SameDiff optimizations - memory use for inference and training significantly reduced, with some performance improvements also
- Note: No API changes, only artifact ID change: replace deeplearning4j-ui_2.1x with deeplearning4j-ui
- Note that additional ND4J namespaces API will have additions (new namespaces and methods), and may have some API changes, in the next release
OpenMP replaced with thread pool c++ parallelism framework; enabled c++ parallelism for platforms without C++-level threading for operations

Deeplearning4J

Deeplearning4J: Features and Enhancements

DNNL (MKL-DNN) upgraded to version 1.1

Deeplearning4J: Bug Fixes and Optimizations

Deeplearning4j: Transition Guide, 1.0.0-beta5 to 1.0.0-beta6

Deeplearning4j UI artifact ID has changed: deeplearning4j-ui_2.1x (beta5 and earlier) with deeplearning4j-ui

ND4J and SameDiff

ND4J/SameDiff: Features and Enhancements

ND4J/SameDiff: Bug Fixes and Optimizations

ND4J: Transition Guide, 1.0.0-beta5 to 1.0.0-beta6

DataVec

DataVec: Bug Fixes and Optimizations

RL4J

RL4J: Features and Enhancements

RL4J: Bug Fixes and Optimizations

PyDataVec

PyDataVec Features and Enhancements

PyDataVec Bug Fixes and Optimizations

1.0.0-beta5

Highlights - 1.0.0-beta5 Release

Added model server - remote inference of SameDiff and DL4J models using JSON or (optionally) binary serialization
Added Scala 2.12 support, dropped Scala 2.10 support. Modules with Scala dependencies are now released with Scala 2.11 and 2.12 versions
Apache Spark 1.x support dropped (now only Spark 2.x is supported). Note: Spark version suffix dropped: For upgrading: 1.0.0-beta4_spark2 -> 1.0.0-beta5
Added FastText support to deeplearning4j-nlp
CUDA support for all ND4J/SameDiff Operations
- In 1.0.0-beta4, some operations were CPU only. Now, all operations have full CUDA support
Added support for new data types in ND4J (and DL4J/SameDiff): BFLOAT16, UINT16, UINT32, UINT64
ND4J: Implicit broadcasting support added to INDArray (already present in SameDiff - for example shape [3,1]+[3,2]=[3,2])
CUDA 9.2, 10.0 and 10.1-Update2 still supported
- NOTE: For CUDA 10.1, CUDA 10.1 update 2 is recommended. CUDA 10.1 and 10.1 Update 1 will still run, but rare internal cuBLAS issues may be encountered in heavily multi-threaded code on some systems
Dependency upgrades: Jackson (2.5.1 to 2.9.9/2.9.9.3), Commons Compress (1.16.1 to 1.18), Play Framework (2.4.8 to 2.7.3), Guava: (20.0 to 28.0-jre, and shaded to avoid dependency clashes)
CUDA: now host (RAM) buffers are only allocated when required (previously: host buffers were always allocated), in addition to device (GPU) buffer

Deeplearning4J

Deeplearning4J: Features and Enhancements

Deeplearning4J: Bug Fixes and Optimizations

Deeplearning4j: Transition Guide, 1.0.0-beta4 to 1.0.0-beta5

DL4J AsyncDataSetIterator and AsyncMultiDataSetIterator moved to ND4J, use org.nd4j.linalg.dataset.Async(Multi)DataSetIterator instead
Apache Spark 1.x support dropped (now only Spark 2.x is supported). Note: Spark version suffix dropped: For upgrading, change versions as follows: 1.0.0-beta4_spark2 -> 1.0.0-beta5
Scala 2.10 dropped, Scala 2.12 added (for modules with Scala dependencies)

Deeplearning4j: 1.0.0-beta5 Known Issues

Some layers (such as LSTM) may run slower on 1.0.0-beta5 than 1.0.0-beta4 on CUDA when not using cuDNN, due to added synchronization. This synchronization will be removed in the next release after 1.0.0-beta5
CUDA 10.1: Rare internal cuBLAS issues may be encountered in heavily multi-threaded code on some systems, when running CUDA 10.1 Update 1 (and maybe 10.1). CUDA 10.1 update 2 is recommended.

ND4J and SameDiff

ND4J/SameDiff: Features and Enhancements

CUDA: now host (RAM) buffers are only allocated when required (previously: host buffers were always allocated), in addition to device (GPU) buffer

ND4J/SameDiff: Bug Fixes and Optimizations

ND4J: Transition Guide, 1.0.0-beta4 to 1.0.0-beta5

OldAddOp, OldSubOp, etc removed: Replace with AddOp, SubOp, etc
Nd4j.trueScalar and trueVector removed; use Nd4j.scalar and Nd4j.createFromArray methods
INDArray.javaTensorAlongDimension removed; use INDArray.tensorAlongDimension instead
INDArray.lengthLong() removed; use INDArray.length() instead

ND4J: 1.0.0-beta5 Known Issues

DataVec

DataVec: Features and Enhancements

DataVec: Bug Fixes and Optimizations

RL4J

RL4J: Features and Enhancements

RL4J: Bug Fixes and Optimizations

Arbiter

Bug Fixes and Optimizations

Arbiter: Known Issues

ND4S

ND4S Features and Enhancements

1.0.0-beta4

Highlights - 1.0.0-beta4 Release

DOUBLE: double precision floating point, 64-bit (8 byte)
FLOAT: single precision floating point, 32-bit (4 byte)
HALF: half precision floating point, 16-bit (2 byte), "FP16"
LONG: long signed integer, 64 bit (8 byte)
INT: signed integer, 32 bit (4 byte)
SHORT: signed short integer, 16 bit (2 byte)
UBYTE: unsigned byte, 8 bit (1 byte), 0 to 255
BYTE: signed byte, 8 bit (1 byte), -128 to 127
BOOL: boolean type, (0/1, true/false). Uses ubyte storage for easier op parallelization
UTF8: String array type, UTF8 format

ND4J Behaviour changes of note:

When creating an INDArray from a Java primitive array, the INDArray datatype will be determined by the primitive array type (unless a datatype is specified)
- For example: Nd4j.createFromArray(double[]) -> DOUBLE datatype INDArray
- Similarly, Nd4j.scalar(1), Nd4j.scalar(1L), Nd4j.scalar(1.0) and Nd4j.scalar(1.0f) will produce INT, LONG, DOUBLE and FLOAT type scalar INDArrays respectively
Some operations require matched datatypes for operands
- For example, if x and y are different datatypes, a cast may be required: x.add(y.castTo(x.dataType()))
Some operations have datatype restrictions: for example, sum on a UTF8 array is not supported, nor is variance on a BOOL array. For some operations on boolean arrays (such as sum), casting to an integer or floating point type first may make sense.

DL4J Behaviour changes of note:

MultiLayerNetwork/ComputationGraph no longer depend in any way on ND4J global datatype.
- The datatype of a network (DataType for it's parameters and activations) can be set during construction using NeuralNetConfigutation.Builder().dataType(DataType)
- Networks can be converted from one type to another (double to float, float to half etc) using MultiLayerNetwork/ComputationGraph.convertDataType(DataType) method

Main new methods:

Nd4j.create(), zeros(), ones(), linspace(), etc methods with DataType argument
INDArray.castTo(DataType) method - to convert INDArrays from one datatype to another
New Nd4j.createFromArray(...) methods for

ND4J/DL4J: CUDA - 10.1 support added, CUDA 9.0 support dropped

CUDA versions supported in 1.0.0-beta4: CUDA 9.2, 10.0, 10.1.

ND4J: Mac/OSX CUDA support dropped

Mac (OSX) CUDA binaries are no longer provided. Linux (x86_64, ppc64le) and Windows (x86_64) CUDA support remains. OSX CPU support (x86_64) is still available.

DL4J/ND4J: MKL-DNN Support Added DL4J (and ND4J conv2d etc ops) now support MKL-DNN by default when running on CPU/native backend. MKL-DNN support is implemented for the following layer types:

ConvolutionLayer and Convolution1DLayer (and Conv2D/Conv2DDerivative ND4J ops)
SubsamplingLayer and Subsampling1DLayer (and MaxPooling2D/AvgPooling2D/Pooling2DDerivative ND4J ops)
BatchNormalization layer (and BatchNorm ND4J op)
LocalResponseNormalization layer (and LocalResponseNormalization ND4J op)
Convolution3D layer (and Conv3D/Conv3DDerivative ND4J ops)

MKL-DNN support for other layer types (such as LSTM) will be added in a future release.

MKL-DNN can be disabled globally (ND4J and DL4J) using Nd4jCpu.Environment.getInstance().setUseMKLDNN(false);

MKL-DNN can be disabled globally for specific ops by setting ND4J_MKL_FALLBACK environment variable to the name of the operations to have MKL-DNN support disabled for. For example: ND4J_MKL_FALLBACK=conv2d,conv2d_bp

ND4J: Improved Performance due to Memory Management Changes

Prior releases of ND4J used periodic garbage collection (GC) to release memory that was not allocated in a memory workspace. (Note that DL4J uses workspaces for almost all operations by default hence periodic GC could frequently be disabled when training DL4J networks). However, the reliance on garbage collection resulted in a performance overhead that scaled with the number of objects in the JVM heap.

In 1.0.0-beta4, the periodic garbage collection is disabled by default; instead, GC will be called only when it is required to reclaim memory from arrays that are allocated outside of workspaces.

To re-enable periodic GC (as per the default in beta3) and set the GC frequency to every 5 seconds (5000ms) you can use:

ND4J: Improved Rank 0/1 Array Support

In prior versions of ND4J, scalars and vectors would sometimes be rank 2 instead of rank 0/1 when getting rows/columns, getting sub-arrays using INDArray.get(NDArrayIndex...) or when creating arrays from Java arrays/scalars. Now, behaviour should be more consistent for these rank 0/1 cases. Note to maintain old behaviour for getRow and getColumn (i.e., return rank 2 array with shape [1,x] and [x,1] respectively), the getRow(long,boolean) and getColumn(long,boolean) methods can be used.

DL4J: Attention layers added

Deeplearning4J

Deeplearning4J: Features and Enhancements

Deeplearning4J: Bug Fixes and Optimizations

ND4J and SameDiff

ND4J/SameDiff: Features and Enhancements

Added basic ("technology preview") of SameDiff UI. Should be considered early WIP with breaking API changes expected in future releases. Supports plotting of SameDiff graphs as well as various metrics (line charts, histograms, etc)
- Currenty embedding in the DL4J UI - call UIServer.getInstance() then go to localhost:9000/samediff to access.
ND4J/SameDiff - new operations added:
SameDiff TensorFlow Import

ND4J/SameDiff: API Changes (Transition Guide): 1.0.0-beta3 to 1.0.0-beta4

ND4J datatypes - significant changes, see highlights at top of this section

ND4J/SameDiff: Bug Fixes and Optimizations

SameDiff: Numerous fixes and enhancements

ND4J: Known Issues

Most CustomOperation operations (such as those used in SameDiff) are CPU only until next release. GPU support was not completed in time for 1.0.0-beta4 release.

DataVec

DataVec: Features and Enhancements

DataVec: Optimizations and Bug Fixes

Arbiter

Arbiter: Enhancements

Arbiter: Fixes

1.0.0-beta2

Highlights - 1.0.0-beta2 Release

ND4J/Deeplearning4j: Added support for CUDA 9.2. Dropped support for CUDA 9.1. (1.0.0-beta2 release has CUDA 8.0, 9.0 and 9.2 support)
Deeplearning4j resource (datasets, pretrained models) storage directory can now be configured via DL4JResources.setBaseDirectory method or org.deeplearning4j.resources.directory system property
ND4J: all indexing is now done with longs instead of ints to allow for arrays with dimensions and lengths greater than Integer.MAX_VALUE (approx. 2.1 billion)
ND4J: nd4j-native-platform will now use Intel MKL-DNN as the default/bundled BLAS implementation (replacing OpenBLAS as the previous default)
Deeplearning4j: Added Out-of-memory (OOM) crash dump reporting functionality. Provides a dump with memory use and configuration if training/inference OOMs (to assist with debugging and tuning memory configuration).

Deeplearning4J

Deeplearning4J: New Features

Deeplearning4J: Bug Fixes and Optimizations

Deeplearning4J: API Changes (Transition Guide): 1.0.0-beta to 1.0.0-beta2

GravesLSTM has been deprecated in favor of LSTM due to lack of CuDNN support but otherwise similar accuracy to in practice. Use LSTM class instead.

Deelpearning4J: 1.0.0-beta2 Known Issues

Deeplearing4J: Keras Import

Keras model import now imports every Keras application
Supports GlobalPooling3D layer import
Supports RepeatVector layer import
Supports LocallyConnected1D and LocallyConnected2D layers
Keras Lambda layers can now be imported by registering custom SameDiff layers
All Keras optimizers are now supported
All advanced activation functions can now be imported.
Many minor bugs have been fixed, including proper weight setting for all configurations of BatchNormalization, improvements to Reshape SeparableConvolution2D, and full support of Bidirectional layers.

ND4J

ND4J: New Features

ND4J: all indexing is now done with longs instead of ints to allow for arrays with dimensions and lengths greater than Integer.MAX_VALUE (approx. 2.1 billion)
SameDiff: A significant number of new ops, and backprop implementations for existing ops

ND4J: Bug Fixes and Optimizations

SameDiff: a significant number of bug fixes for execution and individual ops

ND4J: Known Issues

ND4J: API Changes (Transition Guide): 1.0.0-beta to 1.0.0-beta2

CUDA 9.1 support has been removed. CUDA 8.0, 9.0 and 9.2 support is available
Due to long indexing changes, long/long[] should be used in place of int/int[] in some places (such as INDArray.size(int), INDArray.shape())
Unused and not properly tested/maintained utility class BigDecimalMath has been removed. Users should find an aternative library for this functionality, if required.

DataVec

DataVec: New Features

DataVec: Optimizations and Bug Fixes

DataVec: API Changes (Transition Guide): 1.0.0-beta to 1.0.0-beta2

Arbiter

Arbiter: New Features

Arbiter: Fixes

DataProvider has been deprecated. Use DataSource instead.

RL4J

1.0.0-beta

Highlights - 1.0.0-beta Release

Performance and memory optimizations for DL4J

Deeplearning4J

Deeplearning4J: New Features

New or enhanced layers:

Deeplearning4J: Bug Fixes and Optimizations

- Fixes issues with custom and some Keras import layers on Android
Added new model zoo models:
- (to do)

Deeplearning4J: API Changes (Transition Guide): 1.0.0-alpha to 1.0.0-beta

WorkspaceMode.SINGLE and SEPARATE have been deprecated; use WorkspaceMode.ENABLED instead
Internal layer API changes: custom layers will need to be updated to the new Layer API - see built-in layers or custom layer example
Custom layers etc in pre-1.0.0-beta JSON (ModelSerializer) format need to be registered before they can be deserialized due to JSON format change. Built-in layers and models saved in 1.0.0-beta or later do not require this. Use NeuralNetConfiguration.registerLegacyCustomClassesForJSON(Class) for this purpose
ExistingDataSetIterator has been deprecated; use fit(DataSetIterator, int numEpochs) method instead

Deelpearning4J: 1.0.0-beta Known Issues

ComputationGraph TrainingListener onEpochStart and onEpochEnd methods are not being called correctly
DL4J Zoo Model FaceNetNN4Small2 model configuration is incorrect, causing issues during forward pass
Early stopping score calculators with values thar should be maximized (accuracy, f1 etc) are not working properly (values are minimized not maximized). Workaround: override ScoreCalculator.calculateScore(...) and return 1.0 - super.calculateScore(...).

Deeplearing4J: Keras Import

Deeplearning4J: Keras Import - API Changes (Transition Guide): 1.0.0-alpha to 1.0.0-beta

ND4J

ND4J: New Features

ND4J: Known Issues

Not all op gradients implemented for automatic differentiation
Vast majority of new operations added in 1.0.0-beta do NOT use GPU yet.

ND4J: API Changes (Transition Guide): 1.0.0-alpha to 1.0.0-beta

DataVec

DataVec: New Features

DataVec: Optimizations and Bug Fixes

DataVec: API Changes (Transition Guide): 1.0.0-alpha to 1.0.0-beta

Arbiter

Arbiter: New Features

Added LayerSpace for OCNN (one-class neural network)

Arbiter: Fixes

1.0.0-alpha

Highlights - 1.0.0-alpha Release

ND4J: Added SameDiff - Java automatic differentiation library (alpha release) with Tensorflow import (technology preview) and hundreds of new operations
ND4J: Added CUDA 9.0 and 9.1 support (with cuDNN), dropped support for CUDA 7.5, continued support for CUDA 8.0
ND4J: Native binaries (nd4j-native on Maven Central) now ship with AVX/AVX2/AVX-512 support (Windows/Linux)
DL4J: Large number of new layers and API improvements
DL4J: Keras 2.0 import support

Deeplearning4J

Deeplearning4J: New Features

Layers (new and enhanced)
- Added support for both iteration-based and epoch-based schedules via ISchedule. Also added support for custom (user defined) schedules
- Learning rate schedules are configured on the updaters, via the .updater(IUpdater) method
Adds ComputationGraphConfiguration GraphBuilder .layer(String, Layer, String...) alias for .addLayer(String, Layer, String...)
Added deeplearning4j-ui-standalone module: uber-jar for easy launching of UI server (usage: java -jar deeplearning4j-ui-standalone-1.0.0-alpha.jar -p 9124 -r true -f c:/UIStorage.bin)
Weight initializations:
Added new model zoo models:
New iterators, and iterator improvements:

Deeplearning4J: Bug Fixes and Optimizations

Evaluation no-arg constructor could cause NaN evaluation metrics when used on Spark
ParallelInference fixes:

Deeplearning4J: API Changes (Transition Guide): 0.9.1 to 1.0.0-alpha

Previously deprecated updater configuration methods (.learningRate(double), .momentum(double) etc) all removed
- To configure learning rate: use .updater(new Adam(lr)) instead of .updater(Updater.ADAM).learningRate(lr)
- To configure bias learning rate: use .biasUpdater(IUpdater) method
- To configure learning rate schedules: use .updater(new Adam(ISchedule)) and similar
Updater configuration via enumeration (i.e., .updater(Updater)) has been deprecated; use .updater(IUpdater)
.regularization(boolean) config removed; functionality is now always equivalent to .regularization(true)
.useDropConnect(boolean) removed; use .weightNoise(new DropConnect(double)) instead
.iterations(int) method has been removed (was rarely used and confusing to users)
Multiple utility classes (in org.deeplearning4j.util) have been deprecated and/or moved to nd4j-common. Use same class names in nd4j-common org.nd4j.util instead.
Previously deprecated .activation(String) has been removed; use .activation(Activation) or .activation(IActivation) instead
Layer API change: Custom layers may need to implement applyConstraints(int iteration, int epoch) method
Parameter initializer API change: Custom parameter initializers may need to implement isWeightParam(String) and isBiasParam(String) methods
GravesBidirectionalLSTM has been deprecated; use new Bidirectional(Bidirectional.Mode.ADD, new GravesLSTM.Builder()....build())) instead

Deeplearning4J: 1.0.0-alpha Known Issues

Performance on some networks types may be reduced on CUDA compared to 0.9.1 (with workspaces configured). This will be addressed in the next release

Deeplearing4J: Keras Import

Keras 2 support, keeping backward compatibility for keras 1
Keras 2 and 1 import use exact same API and are inferred by DL4J
Keras unit test coverage increased by 10x, many more real-world integration tests
Unit tests for importing and checking layer weights
Leaky ReLU, ELU, SELU support for model import
All Keras layers can be imported with optional bias terms
Old deeplearning4j-keras module removed, old "Model" API removed
All Keras initializations (Lecun normal, Lecun uniform, ones, zeros, Orthogonal, VarianceScaling, Constant) supported
1D convolution and pooling supported in DL4J and Keras model import
Atrous Convolution 1D and 2D layers supported in Keras model import
1D Zero padding layers supported
Keras constraints module fully supported in DL4J and model import
Upsampling 1D and 2D layers in DL4J and Keras model import (including GAN examples in tests)
Most merge modes supported in Keras model import, Keras 2 Merge layer API supported
Separable Convolution 2D layer supported in DL4J and Keras model import
Deconvolution 2D layer supported in DL4J and Keras model import
Full support of Keras noise layers on import (Alpha dropout, Gaussian dropout and noise)
Support for SimpleRNN layer in Keras model import
Support for Bidirectional layer wrapper Keras model import
Addition of LastTimestepVertex in DL4J to support return_sequences=False for Keras RNN layers.
DL4J support for recurrent weight initializations and Keras import integration.
SpaceToBatch and BatchToSpace layers in DL4J for better YOLO support, plus end-to-end YOLO Keras import test.
Cropping2D support in DL4J and Keras model import

Deeplearning4J: Keras Import - API Changes (Transition Guide): 0.9.1 to 1.0.0-alpha

Deeplearning4J: Keras Import - Known Issues

Embedding layer: In DL4J the output of an embedding layer is 2D by default, unless preprocessors are specified. In Keras the output is always 3D, but depending on specified parameters can be interpreted as 2D. This often leads to difficulties when importing Embedding layers. Many cases have been covered and issues fixed, but inconsistencies remain.
Batchnormalization layer: DL4J's batch normalization layer is much more restrictive (in a good way) than Keras' version of it. For instance, DL4J only allows to normalize spatial dimensions for 4D convolutional inputs, while in Keras any axis can be used for normalization. Depending on the dimension ordering (NCHW vs. NHWC) and the specific configuration used by a Keras user, this can lead to expected (!) and unexpected import errors.
Support for importing a Keras model for training purposes in DL4J (enforceTrainingConfig == true) is still very limited and will be tackled properly for the next release.
Keras Merge layers: seem to work fine with the Keras functional API, but have issues when used in a Sequential model.
Reshape layers: can be somewhat unreliable on import. DL4J rarely has a need to explicitly reshape input beyond (inferred) standard input preprocessors. In Keras, Reshape layers are used quite often. Mapping the two paradigms can be difficult in edge cases.

ND4J

ND4J: New Features

Hundreds of new operations added
Technology preview of tensorflow import added (supports 1.4.0 and up)
nVidia CUDA 8/9.0/9.1 now supported
Worskpaces improvements were introduced to ensure safety: SCOPE_PANIC profiling mode is enabled by default
FlatBuffers support for INDArray serde
Support for auto-broadcastable operations was added
libnd4j, underlying c++ library, got functionality boost and now offers: NDArray class, Graph class, and can be used as standalone library or executable.
Convolution-related ops now support NHWC in addition to NCHW data format.
Accumulation ops now have option to keep reduced dimensions.

ND4J: Known Issues

Not all op gradients implemented for automatic differentiation
Vast majority of new operations added in 1.0.0-alpha do NOT use GPU yet.

ND4J: API Changes (Transition Guide): 0.9.1 to 1.0.0-alpha

ND4J - SameDiff

Control flow is supported with IF and WHILE primitives.

Features

Two execution modes available: Java-driven execution, and Native execution for serialized graphs.
SameDiff graphs can be serialized using FlatBuffers
Building and running computation graphs build from SameDiff operations.
Graphs can run forward pass on input data and compute gradients for the backward pass.
Already supports many high-level layers, like dense layers, convolutions (1D-3D) deconvolutions, separable convolutions, pooling and upsampling, batch normalization, local response normalization, LSTMs and GRUs.
In total there are about 350 SameDiff operations available, including many basic operations used in building complex graphs.

Known Issues and Limitations

Vast majority of new operations added in 1.0.0-alpha do NOT use GPU yet.
While many of the widely used base operations and high-level layers used in practice are supported, op coverage is still limited. Goal is to achieve feature parity with TensorFlow and fully support import for TF graphs.
Some of the existing ops do not have a backward pass implemented (called doDiff in SameDiff).

DataVec

DataVec: New Features

Add new RecordReader / SequenceRecordReader implementations:
Add new transforms:

DataVec: Fixes

DataVec: API Changes (Transition Guide): 0.9.1 to 1.0.0-alpha

RecordWriter and SequenceRecordWriter APIs have been updated with multiple new methods

Arbiter

Arbiter: New Features

Arbiter: Fixes

Arbiter: API Changes (Transition Guide): 0.9.1 to 1.0.0-alpha

As per DL4J updater API changes: old updater configuration (learningRate, momentum, etc) methods have been removed. Use .updater(IUpdater) or .updater(ParameterSpace<IUpdater>) methods instead

RL4J

Add support for LSTM layer to A3C
Fix A3C to make it actually work using new ActorCriticLoss and correct use of randomness
Fix cases when QLearning would fail (non-flat input, incomplete serialization, incorrect normalization)
Fix logic of HistoryProcessor with async algorithms and failures when preprocessing images
Tidy up and correct the output of statistics, also allowing the use of IterationListener
Fix issues preventing efficient execution with CUDA
Provide access to more of the internal structures with NeuralNet.getNeuralNetworks(), Policy.getNeuralNet(), and convenience constructors for Policy
Add MDPs for ALE (Arcade Learning Environment) and MALMO to support Atari games and Minecraft
Update MDP for Doom to allow using the latest version of VizDoom

ScalNet

Can be built with sbt and maven.
Project structure is closely aligned to both DL4J model-import module and Keras.
Supports the following layers: Convolution2D, Dense, EmbeddingLayer, AvgPooling2D, MaxPooling2D, GravesLSTM, LSTM, Bidirectional layer wrapper, Flatten, Reshape. Additionally, DL4J OutputLayers are supported.

ND4S

Scala 2.12 support

0.9.1

Deeplearning4J

Fixed issue with incorrect version dependencies in 0.9.0
Numerical stability improvements to LossMCXENT / LossNegativeLogLikelihood with softmax (should reduce NaNs with very large activations)

ND4J

Known Issues

Deeplearning4j: Use of Evaluation class no-arg constructor (i.e., new Evaluation()) can result in accuracy/stats being reported as 0.0. Other Evaluation class constructors, and ComputationGraph/MultiLayerNetwork.evaluate(DataSetIterator) methods work as expected.
- This also impacts Spark (distributed) evaluation: workaround is to replace sparkNet.evaluate(testData); with sparkNet.doEvaluation(testData, 64, new Evaluation(10))[0];, where 10 is the number of classes and 64 in the evaluation minibatch size to use.
SequenceRecordReaderDataSetIterator applies preprocessors (such as normalization) twice to each DataSet (possible workaround: use RecordReaderMultiDataSetIterator + MultiDataSetWrapperIterator)
TransferLearning: ComputationGraph may incorrectly apply l1/l2 regularization (defined in FinetuneConfiguration) to frozen layers. Workaround: set 0.0 l1/l2 on FineTuneConfiguration, and required l1/l2 on new/non-frozen layers directly. Note that MultiLayerNetwork with TransferLearning appears to be unaffected.

0.9.0

Deeplearning4J

VPTree performance significantly improved
Convolution performance improvements, including activation caching
Evaluation improvements
- ComputationGraph and SparkComputationGraph evaluation convenience methods added (evaluateROC, etc)
- RegressionEvaluation, ROCBinary etc now support per-output masking (in addition to per-example/per-time-step masking)
Optimizations: updaters, bias calculation
New loss functions:

ND4J

Native parallel sort was added
New ops added: SELU/SELUDerivative, TAD-based comparisons, percentile/median, Reverse, Tan/TanDerivative, SinH, CosH, Entropy, ShannonEntropy, LogEntropy, AbsoluteMin/AbsoluteMax/AbsoluteSum, Atan2
New distance functions added: CosineDistance, HammingDistance, JaccardDistance

DataVec

TransformProcess and Transforms now support NDArrayWritables and NDArrayWritable columns
Multiple new Transform classes

Arbiter

- UI now uses Play framework, integrates with DL4J UI (replaces Dropwizard backend). Dependency issues/clashing versions fixed.
- Supports DL4J StatsStorage and StatsStorageRouter mechanisms (FileStatsStorage, Remote UI via RemoveUIStatsStorageRouter)
- General UI improvements (additional information, formatting fixes)

0.8.0

0.8.0 -> 0.9.0 Transition Notes

Deeplearning4j

Updater configuration methods such as .momentum(double) and .epsilon(double) have been deprecated. Instead: use .updater(new Nesterovs(0.9)) and .updater(Adam.builder().beta1(0.9).beta2(0.999).build()) etc to configure

DataVec

CsvRecordReader constructors: now uses characters for delimiters, instead of Strings (i.e., ',' instead of ",")

Arbiter

Arbiter UI is now a separate module, with Scala version suffixes: arbiter-ui_2.10 and arbiter-ui_2.11

Version 0.8.0

Spark 2.0 support (DL4J and DataVec; see transition notes below)
New layers
New ComputationGraph vertices
- L2 distance vertex
- L2 normalization vertex
Per-output masking is now supported for most loss functions (for per output masking, use a mask array equal in size/shape to the labels array; previous masking functionality was per-example for RNNs)
L1 and L2 regularization can now be configured for biases (via l1Bias and l2Bias configuration options)
Evaluation improvements:
- For both MultiLayerNetwork and SparkDl4jMultiLayer: added evaluateRegression, evaluateROC, evaluateROCMultiClass convenience methods
- TSNE re-added to new UI
- Training UI: now usable without an internet connection (no longer relies on externally hosted fonts)
- UI: improvements to error handling for ‘no data’ condition
Epsilon configuration now used for Adam and RMSProp updaters
Fix for bidirectional LSTMs + variable-length time series (using masking)
Spark + Kryo: now test serialization + throw exception if misconfigured (instead of logging an error that can be missed)
MultiLayerNetwork now adds default layer names if no name is specified
DataVec:
- JSON/YAML support for DataAnalysis, custom Transforms etc
- ImageRecordReader refactored to reduce garbage collection load (hence improve performance with large training sets)
- Faster quality analysis.
Arbiter: added new layer types to match DL4J
- Performance improvement for Word2Vec/ParagraphVectors tokenization & training.
Batched inference introduced for ParagraphVectors
Nd4j improvements
- New native operations available for ND4j: firstIndex, lastIndex, remainder, fmod, or, and, xor.
- OpProfiler NAN_PANIC & INF_PANIC now also checks result of BLAS calls.
- Nd4.getMemoryManager() now provides methods to tweak GC behavior.
Alpha version of parameter server for Word2Vec/ParagraphVectors were introduced for Spark. Please note: It’s not recommended for production use yet.
Performance improvements for CNN inference

0.7.2 -> 0.8.0 Transition Notes

Spark versioning schemes: with the addition of Spark 2 support, the versions for Deeplearning4j and DataVec Spark modules has changed
- For Spark 1: use <version>0.8.0_spark_1</version>
- For Spark 2: use <version>0.8.0_spark_2</version>
- Also note: Modules with Spark 2 support are released with Scala 2.11 support only. Spark 1 modules are released with both Scala 2.10 and 2.11 support

0.8.0 Known Issues (At Launch)

Keras 1D convolutional and pooling layers cannot be imported yet. Will be supported in forthcoming release.
Keras v2 model configurations cannot be imported yet. Will be supported in forthcoming release.

0.7.2

Activation function refactor
- New activation functions added: hard sigmoid, randomized leaky rectified linear units (RReLU)
Multiple fixes/improvements for Keras model import
Added P-norm pooling for CNNs (option as part of SubsamplingLayer configuration)
Iteration count persistence: stored/persisted properly in model configuration + fixes to learning rate schedules for Spark network training
LSTM: gate activation function can now be configured (previously: hard-coded to sigmoid)
UI:
- Added Chinese translation
- Fixes for UI + pretrain layers
- Improvements in front-end for handling NaNs
- Added UIServer.stop() method
- Fixed score vs. iteration moving average line (with subsampling)
Solved Jaxb/Jackson issue with Spring Boot based applications
RecordReaderDataSetIterator now supports NDArrayWritable for the labels (set regression == true; used for multi-label classification + images, etc)

0.7.1 -> 0.7.2 Transition Notes

Activation functions (built-in): now specified using Activation enumeration, not String (String-based configuration has been deprecated)

1.00-M2.2

Multi-Project

Tutorials

How To Guides

Developer Docs

Explanation

Configuration

Deeplearning4j

Tutorials

How To Guides

Tuning and Training

Reference

1.0.0-beta2

Highlights - 1.0.0-beta2 Release

ND4J/Deeplearning4j: Added support for CUDA 9.2. Dropped support for CUDA 9.1. (1.0.0-beta2 release has CUDA 8.0, 9.0 and 9.2 support)
Deeplearning4j: New SameDiff layers with training support -
Deeplearning4j resource (datasets, pretrained models) storage directory can now be configured via DL4JResources.setBaseDirectory method or org.deeplearning4j.resources.directory system property
ND4J: all indexing is now done with longs instead of ints to allow for arrays with dimensions and lengths greater than Integer.MAX_VALUE (approx. 2.1 billion)
ND4J: nd4j-native-platform will now use Intel MKL-DNN as the default/bundled BLAS implementation (replacing OpenBLAS as the previous default)
Deeplearning4j: Added Out-of-memory (OOM) crash dump reporting functionality. Provides a dump with memory use and configuration if training/inference OOMs (to assist with debugging and tuning memory configuration).
Deeplearning4j - new layers: Locally connected 1d , Locally connected 2d

Deeplearning4J

Deeplearning4J: New Features

Added new SameDiff layers (automatic differentiation - only single class, forward pass definition required) to DL4J with full training support - SameDiffLayer, SameDiffVertex, SameDiffOutputLayer, SameDiffLambdaLayer, SameDiffLambdaVertex - note that these are CPU-only execution for now
Resource (datasets, pretrained models) storage directory can now be configured via DL4JResources.setBaseDirectory method or org.deeplearning4j.resources.directory system property. Note that it is also possible to set a different base location for downloads (for local mirrors of DL4J resources)
Added Out-of-memory (OOM) crash dump reporting functionality. Provides a dump with memory use and configuration if training/inference OOMs. Same information is available (without a crash) for MultiLayerNetwork/ComputationGraph.memoryInfo methods. Can be disabled (or output directory set) using -
Added Composite[Multi]DataSetPreProcessor to enable multiple [Multi]DataSetPreProcessors to be applied in a single iterator
Added ComputationGraph evaluate methods for multi-output networks: evaluate(DataSetIterator, Map<Integer,IEvaluation[]>) and evaluate(MultiDataSetIterator, Map<Integer,IEvaluation[]>)
Added JointMultiDataSetIterator - utility iterator used to create MultiDataSetIterator from multiple DataSetIterators
GraphVertices may now have trainable parameters directly (not just enclose layers with trainable parameters)
Added MultiLayerNetwork/ComputationGraph getLearningRate methods
Added RandomDataSetIterator and RandomMultiDataSetIterator (mainly for testing/debugging)
Added cyclical "1cycle" schedule for learning rate schedules etc -
RDD repartitioning for Spark training is more configurable (adds Repartitioner interface)
Added ComputationGraph.getIterationCount() and .getEpochCount() for consistency with MultiLayerNetwork
Added locally connected 1d layer
Spark "data loader" API (mainly for Spark)
Spark evaluation: added evaluation method overloads that allow specifying the number of evaluation workers (less than number of Spark threads)
CnnSentenceDataSetIterator now has a Format argument, and supports outputting data for RNNs and 1D CNNs
Added ComputationGraph/MultiLayerNetwork.pretrain((Multi)DataSetIterator, int epochs) method overloads
MultiLayerNetwork and ComputationGraph now have output method overloads where the network output can be placed in the user-specified workspace, instead of being detached . This can be used to avoid creating INDArrays that need to be garbage collected before native memory can be freed.
EmbeddingSequenceLayer now supports [minibatch,1,seqLength] format sequence data in addition to [minibatch,seqLength] format data
CuDNN batch norm implementation will now be used for rank 2 input, not just rank 4 input
Environment variables and system properties for DL4J have been centralized into DL4JResources and DL4JEnvironmentVars classes, with proper descriptions
MultiLayerNetwork and ComputationGraph output/feedForward/fit methods are now thread-safe via synchronization. Note that concurrent use is not recommended due to performance (instead: use ParallelInference); however the now-synchronized methods should avoid obscure errors due to concurrent modifications
BarnesHutTSNE now throws a useful exception in the case where the distance metric is undefined (for example, all zeros plus cosine similarity)

Deeplearning4J: Bug Fixes and Optimizations

ComputationGraph.addListeners was not working correctly if listeners were already present ,
TinyImageNetDataSetIterator did not validate/correctly use input shape configuration ,
BatchNormalization layer now correctly asserts that nOut is set if required (instead of unfriendly shape errors later)
Fixed issue where OutputLayer may not initialize parameter constraints correctly
Fixed performance issue with Nesterov updater using CPU-only op for CUDA execution
Removed TerminationCondition for DL4J optimizers - was not used in practice, and had minor overhead
Fixed issue where EvaluativeListener could hit a workspace validation exception when workspaces are enabled
Fixed issue where TrainingListener.onEpochStart/onEpochEnd were not being called correctly for ComputationGraph
Fixed workspace issue with TensorFlowCnnToFeedForwardPreProcessor
Performance optimization for BatchNormalization when using CuDNN
Performance optimization: Dropout will be applied in-place when safe to do so, avoiding a copy
Added CuDNN implementation of Dropout
Reduced memory use for CuDNN: CuDNN working memory is now shared and reused between layers within a network
CuDNN batch normalization implementation would fail with FP16 datatype
Fixed issue Bidirectional LSTM may incorrectly use workspaces causing an exception
Fixed issue with early stopping where scores to be maximized (accuracy, f1, etc) were not properly triggering termination conditions
Fixed issue where label mask counter could be incorrectly incremented in ComputationGraph.computeGradientAndScore()
ComputationGraph was not setting lastEtlTime field during training
Fixed issue with AutoEncoder layer when workspaces are enabled
Fixed issue with EmbeddingSequenceLayer use of mask arrays
Lombok is now provided scope everywhere, isn't on user classpath when using DL4J
Fixed issue where WordVectorSerializer.readParagraphVectors(File) initialization of label source
Spark training (gradient sharing) now properly handles empty partition edge case when encountered during training
Errors are propagated better/more consistently for Spark gradient sharing training
Fixed issue with 1D CNN layers with mask arrays and stride > 1 (masks not being correctly downsized)
DL4J Batch norm implementation was not correctly adding epsilon value during inference, only during training (CuDNN unaffected)
CuDNN subsampling layers with max pooling and ConvolutionMode.SAME may have taken padding value (0) as the maximum for border values when all non-padding values are less than 0
Spark training with gradient sharing now passes listeners to workers correctly
Fixed rare (and non-terminal) concurrent modification issue with UI and FileStatsStorage
CuDNN convolution layer now supports dilation > 2 (previously: used DL4J conv layer implementation as a fallback)
Yolo2OutputLayer now implements computeScoreForExamples()
SequenceRecordReeaderDataSetIterator now handles the "no labels" case correctly
Fixed issue where BarnesHutTSNE could hit a workspace validation exception
EMNIST iterator could produce incorrect data in some cases after a reset

Deeplearning4J: API Changes (Transition Guide): 1.0.0-beta to 1.0.0-beta2

GravesLSTM has been deprecated in favor of LSTM due to lack of CuDNN support but otherwise similar accuracy to in practice. Use LSTM class instead.
deeplearning4j-modelexport-solr: now uses Lucene/Solr version 7.4.0 (was 7.3.0)
Mask arrays for CNN2d layers must be in broadcastable 4d format: [minibatch,depth or 1, height or 1, width or 1] - previously they were 2d with shape [minibatch,height] or [minibatch,width]. This provents ambiguity in later cases (pooling layers), and allows for more complex masking scenarios (such as masking for different image sizes in same minibatch).
Some older/deprecated Model and Layer methods have been removed. (validateInput(), initParams()). Some custom layers may need to be updated as a result

Deelpearning4J: 1.0.0-beta2 Known Issues

Windows users are unable to load the HDF5 files used in SvhnLabelProvider (used in HouseNumberDetection example). Linux/Mac users are unaffected. A workaround for windows users is to add the sonatype snapshot dependency org.bytedeco.javacpp-presets:hdf5-platform:jar:1.10.2-1.4.3-SNAPSHOT

Deeplearing4J: Keras Import

Keras model import now imports every Keras application
Supports GlobalPooling3D layer import
Supports RepeatVector layer import
Supports LocallyConnected1D and LocallyConnected2D layers
Keras Lambda layers can now be imported by registering custom SameDiff layers
All Keras optimizers are now supported
All advanced activation functions can now be imported.
Many minor bugs have been fixed, including proper weight setting for all configurations of BatchNormalization, improvements to Reshape SeparableConvolution2D, and full support of Bidirectional layers.

ND4J

ND4J: New Features

ND4J: all indexing is now done with longs instead of ints to allow for arrays with dimensions and lengths greater than Integer.MAX_VALUE (approx. 2.1 billion)
Added the ability to write Numpy .npy format using Nd4j.writeAsNumpy(INDArray,File) and convert an INDArray to a numpy strict in-memory using Nd4j.convertToNumpy(INDArray)
ND4j-common ClassPathResource: added ClassPathResource.copyDirectory(File)
SameDiff: A significant number of new ops, and backprop implementations for existing ops
Added Nd4j.randomBernoulli/Binomial/Exponential convenience methods
Added way to disable/suppress ND4J initialization logging via org.nd4j.log.initialization system property
SameDiff class - most op/constructor methods now have complete/useful javadoc
Workspaces can now be disabled globally, ignoring workspace configuration. This is mainly used for debugging; use Nd4j.getWorkspaceManager().setDebugMode(DebugMode.DISABLED) or Nd4j.getWorkspaceManager().setDebugMode(DebugMode.SPILL_EVERYTHING); to enable this. [Link]
Added EnvironmentalAction API for environment variable processing
ND4J environment variables and system properties have been centralized in ND4jEnvironmentVars and ND4jSystemProperties classes and

ND4J: Bug Fixes and Optimizations

SameDiff: a significant number of bug fixes for execution and individual ops
Fixed issue where INDArray.toDoubleArray() with true scalars (rank 0 arrays)
Fixed issue with DataSet.sample() not working for rank 3+ features
IActivation implementations now validate/enforce same shape for activations and gradients
Fixed issue with muliColumnVector where vector is 1d
ImagePreProcessingScaler now supports serialization via NormalizerSerializerStrategy and ModelSerializer
Performance optimization for threshold encoding used in DL4J's Spark gradient sharing distributed training implementation
SameDiff: Fixed issue where memory wasn't always released after execution
DataSet.save() and MultiDataSet.save() methods now save example metadata when present
Fixed issue with KFoldIterator when dataset does not divide equally into folds with no remainder
Fixed issue where version check functionality could fail to load resources if resources are on a path with spaces

ND4J: Known Issues

ND4J: API Changes (Transition Guide): 1.0.0-beta to 1.0.0-beta2

CUDA 9.1 support has been removed. CUDA 8.0, 9.0 and 9.2 support is available
Due to long indexing changes, long/long[] should be used in place of int/int[] in some places (such as INDArray.size(int), INDArray.shape())
Simplified DataSetIterator API: totalExamples(), cursor() and numExamples() - these were unsupported on most DataSetIterator implementations, and not used in practice for training. Custom iterators should remove these methods also
Long-deprecated DataSet.getFeatureMatrix() has been removed. Use DataSet.getFeatures() instead.
Unused and not properly tested/maintained utility class BigDecimalMath has been removed. Users should find an aternative library for this functionality, if required.
Not properly maintained complex number support classes (IComplexNumber, IComplexNDArray) have been removed entirely

DataVec

DataVec: New Features

Added AnalyzeLocal class to mirror functionality of AnalyzeSpark (but without Spark dependency)
Added JacksonLineSequenceRecordReader: RecordReader used for multi-example JSON/XML where each line in a file is an independent example
Added RecordConvert.toRecord(Schema, List<Object>)
Added missing FloatColumnCondition
Added CSVLineSequenceRecordReader for "each line in CSV is a sequence, and sequence is single-valued/univariate"
Added CSVMultiSequenceRecordReader for "multiple multi-valued sequences in a single CSV" data

DataVec: Optimizations and Bug Fixes

Fixed issue with NativeImageLoader on Android
Fixed issue with ExcelRecordReader

DataVec: API Changes (Transition Guide): 1.0.0-beta to 1.0.0-beta2

Arbiter

Arbiter: New Features

Arbiter: Fixes

DataProvider has been deprecated. Use DataSource instead.

RL4J

1.0.0-alpha

Highlights - 1.0.0-alpha Release

ND4J: Added SameDiff - Java automatic differentiation library (alpha release) with Tensorflow import (technology preview) and hundreds of new operations
ND4J: Added CUDA 9.0 and 9.1 support (with cuDNN), dropped support for CUDA 7.5, continued support for CUDA 8.0
ND4J: Native binaries (nd4j-native on Maven Central) now ship with AVX/AVX2/AVX-512 support (Windows/Linux)
DL4J: Large number of new layers and API improvements
DL4J: Keras 2.0 import support

Deeplearning4J

Deeplearning4J: New Features

Layers (new and enhanced)
- Added support for both iteration-based and epoch-based schedules via ISchedule. Also added support for custom (user defined) schedules
- Learning rate schedules are configured on the updaters, via the .updater(IUpdater) method
Adds ComputationGraphConfiguration GraphBuilder .layer(String, Layer, String...) alias for .addLayer(String, Layer, String...)
Added deeplearning4j-ui-standalone module: uber-jar for easy launching of UI server (usage: java -jar deeplearning4j-ui-standalone-1.0.0-alpha.jar -p 9124 -r true -f c:/UIStorage.bin)
Weight initializations:
Added new model zoo models:
New iterators, and iterator improvements:

Deeplearning4J: Bug Fixes and Optimizations

Evaluation no-arg constructor could cause NaN evaluation metrics when used on Spark
ParallelInference fixes:

Deeplearning4J: API Changes (Transition Guide): 0.9.1 to 1.0.0-alpha

Previously deprecated updater configuration methods (.learningRate(double), .momentum(double) etc) all removed
- To configure learning rate: use .updater(new Adam(lr)) instead of .updater(Updater.ADAM).learningRate(lr)
- To configure bias learning rate: use .biasUpdater(IUpdater) method
- To configure learning rate schedules: use .updater(new Adam(ISchedule)) and similar
Updater configuration via enumeration (i.e., .updater(Updater)) has been deprecated; use .updater(IUpdater)
.regularization(boolean) config removed; functionality is now always equivalent to .regularization(true)
.useDropConnect(boolean) removed; use .weightNoise(new DropConnect(double)) instead
.iterations(int) method has been removed (was rarely used and confusing to users)
Multiple utility classes (in org.deeplearning4j.util) have been deprecated and/or moved to nd4j-common. Use same class names in nd4j-common org.nd4j.util instead.
Previously deprecated .activation(String) has been removed; use .activation(Activation) or .activation(IActivation) instead
Layer API change: Custom layers may need to implement applyConstraints(int iteration, int epoch) method
Parameter initializer API change: Custom parameter initializers may need to implement isWeightParam(String) and isBiasParam(String) methods
GravesBidirectionalLSTM has been deprecated; use new Bidirectional(Bidirectional.Mode.ADD, new GravesLSTM.Builder()....build())) instead

Deeplearning4J: 1.0.0-alpha Known Issues

Performance on some networks types may be reduced on CUDA compared to 0.9.1 (with workspaces configured). This will be addressed in the next release

Deeplearing4J: Keras Import

Keras 2 support, keeping backward compatibility for keras 1
Keras 2 and 1 import use exact same API and are inferred by DL4J
Keras unit test coverage increased by 10x, many more real-world integration tests
Unit tests for importing and checking layer weights
Leaky ReLU, ELU, SELU support for model import
All Keras layers can be imported with optional bias terms
Old deeplearning4j-keras module removed, old "Model" API removed
All Keras initializations (Lecun normal, Lecun uniform, ones, zeros, Orthogonal, VarianceScaling, Constant) supported
1D convolution and pooling supported in DL4J and Keras model import
Atrous Convolution 1D and 2D layers supported in Keras model import
1D Zero padding layers supported
Keras constraints module fully supported in DL4J and model import
Upsampling 1D and 2D layers in DL4J and Keras model import (including GAN examples in tests)
Most merge modes supported in Keras model import, Keras 2 Merge layer API supported
Separable Convolution 2D layer supported in DL4J and Keras model import
Deconvolution 2D layer supported in DL4J and Keras model import
Full support of Keras noise layers on import (Alpha dropout, Gaussian dropout and noise)
Support for SimpleRNN layer in Keras model import
Support for Bidirectional layer wrapper Keras model import
Addition of LastTimestepVertex in DL4J to support return_sequences=False for Keras RNN layers.
DL4J support for recurrent weight initializations and Keras import integration.
SpaceToBatch and BatchToSpace layers in DL4J for better YOLO support, plus end-to-end YOLO Keras import test.
Cropping2D support in DL4J and Keras model import

Deeplearning4J: Keras Import - API Changes (Transition Guide): 0.9.1 to 1.0.0-alpha

Deeplearning4J: Keras Import - Known Issues

Embedding layer: In DL4J the output of an embedding layer is 2D by default, unless preprocessors are specified. In Keras the output is always 3D, but depending on specified parameters can be interpreted as 2D. This often leads to difficulties when importing Embedding layers. Many cases have been covered and issues fixed, but inconsistencies remain.
Batchnormalization layer: DL4J's batch normalization layer is much more restrictive (in a good way) than Keras' version of it. For instance, DL4J only allows to normalize spatial dimensions for 4D convolutional inputs, while in Keras any axis can be used for normalization. Depending on the dimension ordering (NCHW vs. NHWC) and the specific configuration used by a Keras user, this can lead to expected (!) and unexpected import errors.
Support for importing a Keras model for training purposes in DL4J (enforceTrainingConfig == true) is still very limited and will be tackled properly for the next release.
Keras Merge layers: seem to work fine with the Keras functional API, but have issues when used in a Sequential model.
Reshape layers: can be somewhat unreliable on import. DL4J rarely has a need to explicitly reshape input beyond (inferred) standard input preprocessors. In Keras, Reshape layers are used quite often. Mapping the two paradigms can be difficult in edge cases.

ND4J

ND4J: New Features

Hundreds of new operations added
Technology preview of tensorflow import added (supports 1.4.0 and up)
nVidia CUDA 8/9.0/9.1 now supported
Worskpaces improvements were introduced to ensure safety: SCOPE_PANIC profiling mode is enabled by default
FlatBuffers support for INDArray serde
Support for auto-broadcastable operations was added
libnd4j, underlying c++ library, got functionality boost and now offers: NDArray class, Graph class, and can be used as standalone library or executable.
Convolution-related ops now support NHWC in addition to NCHW data format.
Accumulation ops now have option to keep reduced dimensions.

ND4J: Known Issues

Not all op gradients implemented for automatic differentiation
Vast majority of new operations added in 1.0.0-alpha do NOT use GPU yet.

ND4J: API Changes (Transition Guide): 0.9.1 to 1.0.0-alpha

ND4J - SameDiff

Control flow is supported with IF and WHILE primitives.

Features

Two execution modes available: Java-driven execution, and Native execution for serialized graphs.
SameDiff graphs can be serialized using FlatBuffers
Building and running computation graphs build from SameDiff operations.
Graphs can run forward pass on input data and compute gradients for the backward pass.
Already supports many high-level layers, like dense layers, convolutions (1D-3D) deconvolutions, separable convolutions, pooling and upsampling, batch normalization, local response normalization, LSTMs and GRUs.
In total there are about 350 SameDiff operations available, including many basic operations used in building complex graphs.

Known Issues and Limitations

Vast majority of new operations added in 1.0.0-alpha do NOT use GPU yet.
While many of the widely used base operations and high-level layers used in practice are supported, op coverage is still limited. Goal is to achieve feature parity with TensorFlow and fully support import for TF graphs.
Some of the existing ops do not have a backward pass implemented (called doDiff in SameDiff).

DataVec

DataVec: New Features

Add new RecordReader / SequenceRecordReader implementations:
Add new transforms:

DataVec: Fixes

DataVec: API Changes (Transition Guide): 0.9.1 to 1.0.0-alpha

RecordWriter and SequenceRecordWriter APIs have been updated with multiple new methods

Arbiter

Arbiter: New Features

Arbiter: Fixes

Arbiter: API Changes (Transition Guide): 0.9.1 to 1.0.0-alpha

As per DL4J updater API changes: old updater configuration (learningRate, momentum, etc) methods have been removed. Use .updater(IUpdater) or .updater(ParameterSpace<IUpdater>) methods instead

RL4J

Add support for LSTM layer to A3C
Fix A3C to make it actually work using new ActorCriticLoss and correct use of randomness
Fix cases when QLearning would fail (non-flat input, incomplete serialization, incorrect normalization)
Fix logic of HistoryProcessor with async algorithms and failures when preprocessing images
Tidy up and correct the output of statistics, also allowing the use of IterationListener
Fix issues preventing efficient execution with CUDA
Provide access to more of the internal structures with NeuralNet.getNeuralNetworks(), Policy.getNeuralNet(), and convenience constructors for Policy
Add MDPs for ALE (Arcade Learning Environment) and MALMO to support Atari games and Minecraft
Update MDP for Doom to allow using the latest version of VizDoom

ScalNet

Can be built with sbt and maven.
Project structure is closely aligned to both DL4J model-import module and Keras.
Supports the following layers: Convolution2D, Dense, EmbeddingLayer, AvgPooling2D, MaxPooling2D, GravesLSTM, LSTM, Bidirectional layer wrapper, Flatten, Reshape. Additionally, DL4J OutputLayers are supported.

ND4S

Scala 2.12 support

Benchmark

General guidelines for benchmarking in DL4J and ND4J.

General Benchmarking Guidelines

Guideline 1: Run Warm-Up Iterations Before Benchmarking

A warm-up period is where you run a number of iterations (for example, a few hundred) of your benchmark without timing, before commencing timing for further iterations.

Why is a warm-up required? The first few iterations of any ND4J/DL4J execution may be slower than those that come later, for a number of reasons:

In the initial benchmark iterations, the JVM has not yet had time to perform just-in-time compilation of code. Once JIT has completed, code is likely to execute faster for all subsequent operations
ND4J and DL4J (and, some other libraries) have some degree of lazy initialization: the first operation may trigger some one-off execution code.
DL4J or ND4J (when using workspaces) can take some iterations to learn memory requirements for execution. During this learning phase, performance will be lower than after its completion.

Guideline 2: Run Multiple Iterations of All Benchmarks

Your benchmark isn't the only thing running on your computer (not to mention if you are using cloud hardware, that might have shared resources). And operation runtime is not perfectly deterministic.

For benchmark results to be reliable, it is important to run multiple iterations - and ideally report both mean and standard deviation for the runtime. Without this, it's impossible to compare the performance of operations, as performance differences may simply be due to random variation.

Guideline 3: Pay Careful Attention to What You Are Benchmarking

This is especially important when comparing frameworks. Before you declare that "performance on operation X is Y" or "A is faster than B", make sure that:

You are bench-marking only the operations of interest.

If your goal is to check the performance of an operation, make sure that only this operation is being timed.

You should carefully check whether you unintentionally including other things - for example, does it include: JVM initialization time? Library initialization time? Result array allocation time? Garbage collection time? Data loading time?

Ideally, these should be excluded from any timing/performance results you report. If they cannot be excluded, make sure you note this whenever making performance claims.

What native libraries are you using?
For example: what BLAS implementation (MKL, OpenBLAS, etc)? If you are using CUDA, are you using CuDNN? ND4J and DL4J can use these libraries (MKL, CuDNN) when they are available - but are not always available by default. If they are not made available, performance can be lower - sometimes considerably.
This is especially important when comparing results between libraries: for example, if you compared two libraries (one using OpenBLAS, another using MKL) your results may simply reflect the performance differences it the BLAS library being used - and not the performance of the libraries being tested. Similarly, one library with CuDNN and another without CuDNN may simply reflect the performance benefit of using CuDNN.
How are things configured?
For better or worse, DL4J and ND4J allow a lot of configuration. The default values for a lot of this configuration is adequate for most users - but sometimes manual configuration is required for optimal performance. This can be especially true in some benchmarks! Some of these configuration options allow users to trade off higher memory use for better performance, for example. Some configuration options of note: (a) Memory configuration (b) Workspaces and garbage collection (c) CuDNN (d) DL4J Cache Mode (enable using .cacheMode(CacheMode.DEVICE))

If you aren't sure if you are only measuring what you intend to measure when running DL4J or ND4J code, you can use a profiler such as VisualVM or YourKit Profilers.

What versions are you using? When benchmarking, you should use the latest version of whatever libraries you are benchmarking. There's no point identifying and reporting a bottleneck that was fixed 6 months ago. An exception to this would be when you are comparing performance over time between versions. Note also that snapshot versions of DL4J and ND4J are also available - these may contain performance improvements (feel free to ask)

Guideline 4: Focus on Real-World Use Cases - And Run a Range of Sizes

Consider for example a benchmark a benchmark that adds two numbers:

And something equivalent in ND4J:

Of course, the ND4J benchmark above is going to be much slower - method calls are required, input validation is performed, native code has to be called (with context switching overhead), and so on. One must ask the question, however: is this what users will actually be doing with ND4J or an equivalent linear algebra library? It's an extreme example - but the general point is a valid one.

Note also that performance on mathematical operations can be size - and shape - specific. For example, if you are benchmarking the performance on matrix multiplication - the matrix dimensions can matter a lot. In some internal benchmarks, we found that different BLAS implementations (MKL vs OpenBLAS) - and different backends (CPU vs GPU) - can perform very differently with different matrix dimensions. None of the BLAS implementations (OpenBLAS, MKL, CUDA) we have tested internally were uniformly faster than others for all input shapes and sizes.

Therefore - whenever you are running benchmarks, it's important to run those benchmarks with multiple different input shapes/sizes, to get the full performance picture.

Guideline 5: Understand Your Hardware

When comparing different hardware, it's important to be aware of what it excels at. For example, you might find that neural network training performs faster on a CPU with minibatch size 1 than on a GPU - yet larger minibatch sizes show exactly the opposite. Similarly, small layer sizes may not be able to adequately utilize the power of a GPU.

Furthermore, some deep learning distributions may need to be specifically compiled to provide support for hardware features such as AVX2 (note that recent version of ND4J are packaged with binaries for CPUs that support these features). When running benchmarks, the utilization (or lack there-of) of these features can make a considerable difference to performance.

Guideline 6: Make It Reproducible

When running benchmarks, it's important to make your benchmarks reproducible. Why? Good or bad performance may only occur under certain limited circumstances.

And finally - remember that (a) ND4J and DL4J are in constant development, and (b) benchmarks do sometimes identify performance bottlenecks (after all we - ND4J includes literally hundreds of distinct operations). If you identify a performance bottleneck, great - we want to know about it - so we can fix it. Any time a potential bottleneck is identified, we first need to reproduce it - so that we can study it, understand it and ultimately fix it.

Guideline 7: Understand the Limitations of Your Benchmarks

Linear algebra libraries contain hundreds of distinct operations. Neural network libraries contain dozens of layer types. When benchmarking, it's important to understand the limitations of those benchmarks. Benchmarking one type of operation or layer cannot tell you anything about the performance on other types of layers or operations - unless they share code that has been identified to be a performance bottleneck.

Guideline 8: If You Aren't Sure - Ask

And if you do happen to find a performance issue - let us know!

ND4J Specific Benchmarking

A Note on BLAS and Array Orders

BLAS - or Basic Linear Algebra Subprograms - refers to an interface and set of methods used for linear algebra operations. Some examples include 'gemm' - General Matrix Multiplication - and 'axpy', which implements Y = a*X+b.

Note that ND4J will log the BLAS backend used when it initializes. For example:

Performance can depend on the available BLAS library - in internal tests, we have found that OpenBLAS has been between 30% faster and 8x slower than MKL - depending on the array sizes and array orders.

For matrix multiplication, this means there are 8 possible combinations of array orders (c/f for each of input 1, input 2 and result arrays). Performance won't be the same for all cases.

Similarly, an operation such as element-wise addition (i.e., z=x+y) will be much faster for some combinations of input orders than others - notably, when x, y and z are all the same order. In short, this is due to memory striding: it's cheaper to read a sequence of memory addresses when those memory addresses are adjacent to each other in memory, as compared to being spread far apart.

Note that, by default, ND4J expects result arrays (for matrix multiplication) to be defined in column major ('f') order, to be consistent across backends, given that CuBLAS (i.e., NVIDIA's BLAS library for CUDA) requires results to be in f order. As a consequence, some ways of performing matrix multiplication with the result array being in c order will have lower performance than if the same operation was executed with an 'f' order array.

Finally, when it comes to CUDA: array orders/striding can matter even more than when running on CPU. For example, certain combinations of orders can be much faster than others - and input/output dimensions that are even multiples of 32 or 64 typically perform faster (sometimes considerably) than when input/output dimensions are not multiples of 32.

DL4J Specific Benchmarking

Most of what has been said for ND4J also applies to DL4J.

In addition:

If you are using the nd4j-native (CPU) backend, ensure you are using Intel MKL. This is faster than the default of OpenBLAS in most cases.
Watch out for ETL bottlenecks. You can add PerformanceListener to your network training to see if ETL is a bottleneck.
Don't forget that performance is dependent on minibatch sizes. Don't benchmark with minibatch size 1 - use something more realistic.
If you need multi-GPU training or inference support, use ParallelWrapper or ParallelInference.
Don't forget that CuDNN is configurable: you can specify DL4J/CuDNN to prefer performance - at the expense of memory - using .cudnnAlgoMode(ConvolutionLayer.AlgoMode.PREFER_FASTEST) configuration on convolution layers
When using GPUs, multiples of 8 (or 32) for input sizes and layer sizes may perform better.
When using RNNs (and manually creating INDArrays), use 'f' ordered arrays for both features and (RnnOutputLayer) labels. Otherwise, use 'c' ordered arrays. This is for faster memory access.

Common Benchmark Mistakes

Finally, here's a summary list of common benchmark mistakes:

Not using the latest version of ND4J/DL4J (there's no point identifying a bottleneck that was fixed many releases back). Consider trying snapshots to get the latest performance improvements.
Not paying attention to what native libraries (MKL, OpenBLAS, CuDNN etc) are being used
Providing no warm-up period before benchmarking begins
Running only a single (or too few) iterations, or not reporting mean, standard deviation and number of iterations
Not configuring workspaces, garbage collection, etc
Running only one possible case - for example, benchmarking a single set of array dimensions/orders when benchmarking BLAS operations
Running unusually small inputs - for example, minibatch size 1 on a GPU (which might be slower - but isn't realistic!)
Not measuring exactly - and only - what you claim to be measuring (for example, not accounting for array allocation, initialization or garbage collection time)
Not making your benchmarks reproducible (does the benchmark conclusion generalize? are there problems with the benchmark? what can we do to fix it?)
Comparing results across different hardware, not accounting for differences (for example, testing on one machine with AVX2 support, and on another without)

How to Run Deeplearning4j Benchmarks - A Guide

Total training time is always ETL plus computation. That is, both the data pipeline and the matrix manipulations determine how long a neural network takes to train on a dataset.

When programmers familiar with Python try to run benchmarks comparing Deeplearning4j to well-known Python frameworks, they usually end up comparing ETL + computation on DL4J to just computation on the Python framework. That is, they're comparing apples to oranges. We'll explain how to optimize several parameters below.

The JVM has knobs to tune, and if you know how to tune them, you can make it a very fast environment for deep learning. There are several things to keep in mind on the JVM. You need to:

Get garbage collection right
Make ETL asynchronous
Presave datasets (aka pickling)

Setting Heap Space

Users have to reconfigure their JVMs themselves, including setting the heap space. We can't give it to you preconfigured, but we can show you how to do it. Here are the two most important knobs for heap space.

Xms sets the minimum heap space
Xmx sets the maximum heap space

You can set these in IDEs like IntelliJ and Eclipse, as well as via the CLI like so:

What’s the ideal amount to set Xmx to? That depends on how much RAM is on your computer. In general, allocate as much heap space as you think the JVM will need to get work done. Let’s say you’re on a 16G RAM laptop — allocate 8G of RAM to the JVM. A sound minimum on laptops with less RAM would be 3g, so

It may seem counterintuitive, but you want the min and max to be the same; i.e. Xms should equal Xmx. If they are unequal, the JVM will progressively allocate more memory as needed until it reaches the max, and that process of gradual allocation slows things down. You want to pre-allocate it at the beginning. So

Another way to do this is by setting your environmental variables. Here, you would alter your hidden .bash_profile file, which adds environmental variables to bash. To see those variables, enter env in the command line. To add more heap space, enter this command in your console:

We need to increase heap space because Deeplearning4j loads data in the background, which means we're taking more RAM in memory. By allowing more heap space for the JVM, we can cache more data in memory.

Garbage Collection

A garbage collector is a program which runs on the JVM and gets rid of objects no longer used by a Java application. It is automatic memory management. Creating a new object in Java takes on-heap memory: A new Java object takes up 8 bytes of memory by default. So every new DatasetIterator you create takes another 8 bytes.

You may need to alter the garbage collection algorithm that Java is using. This can be done via the command line like so:

JavaCPP, created by a Skymind engineer, relies on the garbage collector to tell it what has been done. We rely on the Java GC to tell us what to collect; the Java GC points at things, and we know how to de-allocate them with JavaCPP. This applies equally to how we work with GPUs.

The larger the batch size you use, the more RAM you’re taking in memory.

ETL & Asynchronous ETL

In our dl4j-examples repo, we don't make the ETL asynchronous, because the point of examples is to keep them simple. But for real-world problems, you need asynchronous ETL, and we'll show you how to do it with examples.

Data is stored on disk and disk is slow. That’s the default. So you run into bottlenecks when loading data onto your hard drive. When optimizing throughput, the slowest component is always the bottleneck. For example, a distributed Spark job using three GPU workers and one CPU worker will have a bottleneck with the CPU. The GPUs have to wait for that CPU to finish.

The Deeplearning4j class DatasetIterator hides the complexity of loading data on disk. The code for using any Datasetiterator will always be the same, invoking looks the same, but they work differently.

one loads from disk
one loads asynchronously
one loads pre-saved from RAM

Here's how the DatasetIterator is uniformly invoked for MNIST:

You can optimize by using an asynchronous loader in the background. Java can do real multi-threading. It can load data in the background while other threads take care of compute. So you load data into the GPU at the same time that compute is being run. The neural net trains even as you grab new data from memory.

Notice in the code above that prefetchSize is another parameter to set. Normal batch size might be 1000 examples, but if you set prefetchSize to 3, it would pre-fetch 3,000 instances.

ETL: Comparing Python frameworks With Deeplearning4j

But Java has robust tools for moving big data, and if compared correctly, is much faster than Python. The Deeplearning4j community has reported up to 3700% increases in speed over Python frameworks, when ETL and computation are optimized.

We try to be more flexible. That means you can point DL4J at raw photos, and it will load the image, run the transforms and put it into an NDArray to generate a dataset on the fly.

But if your training pipeline is doing that every time, Deeplearning4j will seem about 10x slower than other frameworks, because you’re spending your time creating datasets. Every time you call fit, you're recreating a dataset, over and over again. We allow it to happen for ease of use, but we can show you how to speed things up. There are ways to make it just as fast.

One way is to pre-save the datasets, in a manner similar to the Python frameworks. (Pickles are pre-formatted data.) When you pre-save the dataset, you create a separate class.

A Recordreaderdatasetiterator talks to Datavec and outputs datasets for DL4J.

Line 90 is where you see the asynchronous ETL. In this case, it's wrapping the pre-saved iterator, so you're taking advantage of both methods, with the asynch loading the pre-saved data in the background as the net trains.

MKL and Inference on CPUs

If you are running inference benchmarks on CPUs, make sure you are using Deeplearning4j with Intel's MKL library, which is available via a clickwrap; i.e. Deeplearning4j does not bundle MKL like Anaconda, which is used by libraries like PyTorch.

Benchmark memory usage

Memory usage can vary depending on a wide variety of configurations. Memory in the dl4j suite comes in 2 buckets:

On heap memory: This is java based memory you are used to in java. The main concern here is heap space. Java profiling tools such as yourkit and jvisualvm can monitor this.

Memory usage maybe rampant for a wide variety of reasons. The worse case is lots of small arrays created in multiple threads. When many small arrays are created, this can create a situation called memory pressure. Memory pressure is when the GC is racing to dealocate memory and can't keep up. When this happens, the java runtime can fall behind and eventually freeze. This means that the GC can hang for a while and can cause severe performance degradation.

Note this deallocator relies on the java virtual machine's garbage collector to hand us references we know are out of scope and can be deallocated. This is crucial to managing off heap memory.

You can also close references manually. Note that arrays attached to a workspace (you can check this with isAttached()) will not be deallocated. If you want to manage memory manualy, either create a workspace using: Nd4j.getMemoryManager.scopeOutofWorkspaces() or use Nd4j.createUnitializedDetached* methods for creating ndarrays that are not associated with a workspace.

If you don't do this and use the standard Nd4j.create methods, memory will always be associated with a default workspace.

Something else to be aware of in terms of memory are constant shape buffers. Constant shape buffers are always allocated and reused instead of deallocated. This is due to ndarrays always reallocating the same shape buffers.

Potential Memory Leaks

Note that when running jemalloc and jeprof to consider the following: 1. jemalloc must be compiled using --with-prof

jemalloc can generate lots of small files. You may run in to memory limits with your operating system. On linux this can be handled with ulimit -n and ulimit -s increasing the limits for being able to generate a visualization.
We recommend using the interactive mode to learn the tool. Just call jeprof with the list of .heap files generated by jemalloc. You can capture these files using jeprof_* or a similar wildcard pattern.

Performance Issues

How to Debug Performance Issues

This page is a how-to guide for debugging performance issues encountered when training neural networks with Deeplearning4j. Much of the information also applies to debugging performance issues encountered when using ND4J.

Deeplearning4j and ND4J provide excellent performance in most cases (utilizing optimized c++ code for all numerical operations as well as high performance libraries such as NVIDIA cuDNN and Intel MKL). However, sometimes bottlenecks or misconfiguration issues may limit performance to well below the maximum. This page is intended to be a guide to help users identify the cause of poor performance, and provide steps to fix these issues.

Performance issues may include:

Poor CPU/GPU utilization
Slower than expected training or operation execution

To start, here’s a summary of some possible causes of performance issues:

Wrong ND4J backend is used (for example, CPU backend when GPU backend is expected)
Not using cuDNN when using CUDA GPUs
ETL (data loading) bottlenecks
Garbage collection overheads
Small batch sizes
Multi-threaded use of MultiLayerNetwork/ComputationGraph for inference (not thread safe)
Double precision floating point data type used when single precision should be used
Not using workspaces for memory management (enabled by default)
Poorly configured network
Layer or operation is CPU-only
CPU: Lack of hardware support for modern AVX etc extensions
Other processes using CPU or GPU resources
CPU: Lack of configuration of OMP_NUM_THREADS when using many models/threads simultaneously

Step 1: Check if correct backend is used

ND4J (and by extension, Deeplearning4j) can perform computation on either the CPU or GPU. The device used for computation is determined by your project dependencies - you include nd4j-native-platform to use CPUs for computation or nd4j-cuda-x.x-platform to use GPUs for computation (where x.x is your CUDA version - such as 9.2, 10.0 etc).

It is straightforward to check which backend is used. ND4J will log the backend upon initialization.

For CPU execution, you will expect output that looks something like:

For CUDA execution, you would expect the output to look something like:

Pay attention to the Loaded [X] backend and Backend used: [X] messages to confirm that the correct backend is used. If the incorrect backend is being used, check your program dependencies to ensure tho correct backend has been included.

Step 2: Check for cuDNN

If you are using CPUs only (nd4j-native backend) then you can skip to step 3 as cuDNN only applies when using NVIDIA GPUs (nd4j-cuda-x.x-platform dependency).

cuDNN is NVIDIA’s library for accelerating neural network training on NVIDIA GPUs. Deeplearning4j can make use of cuDNN to accelerate a number of layers - including ConvolutionLayer, SubsamplingLayer, BatchNormalization, Dropout, LocalResponseNormalization and LSTM. When training on GPUs, cuDNN should always be used if possible as it is usually much faster than the built-in layer implementations.

How to determine if CuDNN is used

CUDNN will be shown in the build info as follows (note for those coming from prior versions, deeplearning4j-cuda was removed long ago and now all op delegation has been moved to c++):

Step 3: Check for ETL (Data Loading) Bottlenecks

Neural network training requires data to be in memory before training can proceed. If the data is not loaded fast enough, the network will have to wait until data is available. DL4J uses asynchronous prefetch of data to improve performance by default. Under normal circumstances, this asynchronous prefetching means the network should never be waiting around for data (except on the very first iteration) - the next minibatch is loaded in another thread while training is proceeding in the main thread.

However, when data loading takes longer than the iteration time, data can be a bottleneck. For example, if a network takes 100ms to perform fitting on a single minibatch, but data loading takes 200ms, then we have a bottleneck: the network will have to wait 100ms per iteration (200ms loading - 100ms loading in parallel with training) before continuing the next iteration. Conversely, if network fit operation was 100ms and data loading was 50ms, then no data loading bottleck will occur, as the 50ms loading time can be completed asynchronously within one iteration.

How to check for ETL / data loading bottlenecks

The way to identify ETL bottlenecks is simple: add PerformanceListener to your network, and train as normal. For example:

When training, you will see output such as:

The above output shows that there is no ETL bottleneck (i.e., ETL: 0 ms). However, if ETL time is greater than 0 consistently (after the first iteration), an ETL bottleneck is present.

How to identify the cause of an ETL bottleneck

There are a number of possible causes of ETL bottlenecks. These include (but are not limited to):

Slow hard drives
Network latency or throughput issues (when reading from remote or network storage)
Computationally intensive or inefficient ETL (especially for custom ETL pipelines)

Step 4: Check for Garbage Collection Overhead

Even though DL4J/ND4J array memory is off-heap, garbage collection can still cause performance issues.

In summary:

Garbage collection will sometimes (temporarily and briefly) pause/stop application execution (“stop the world”)
These GC pauses slow down program execution
The overall performance impact of GC pauses depends on both the frequency of GC pauses, and the duration of GC pauses
The frequency is controllable (in part) by ND4J, using Nd4j.getMemoryManager().setAutoGcWindow(10000); and Nd4j.getMemoryManager().togglePeriodicGc(false);
Not every GC event is caused by or controlled by the above ND4J configuration.

In our experience, garbage collection time depends strongly on the number of objects in the JVM heap memory. As a rough guide:

Less than 100,000 objects in heap memory: short GC events (usually not a performance problem)
100,000-500,000 objects: GC overhead becomes noticeable, often in the 50-250ms range per full GC event
500,000 or more objects: GC can be a bottleneck if performed frequently. Performance may still be good if GC events are infrequent (for example, every 10 seconds or less).
10 million or more objects: GC is a major bottleneck even if infrequently called, with each full GC takes multiple seconds

How to configure ND4J garbage collection settings

In simple terms, there are two settings of note:

How to determine GC impact using PerformanceListener

NOTE: this feature was added after 1.0.0-beta3 and will be available in future releases To determine the impact of garbage collection using PerformanceListener, you can use the following:

This will report GC activity:

The garbage collection activity is reported for all available garbage collectors - the GC: [PS Scavenge: 2 (1ms)], [PS MarkSweep: 2 (24ms)] means that garbage collection was performed 2 times since the last PerformanceListener reporting, and took 1ms and 24ms total respectively for the two GC algorithms, respectively.

Keep in mind: PerformanceListener reports GC events every N iterations (as configured by the user). Thus, if PerformanceListener is configured to report statistics every 10 iterations, the garbage collection stats would be for the period of time corresponding to the last 10 iterations.

How to determine GC impact using -verbose:gc

When these options are enabled, you will have information reported on each GC event, such as:

This information can be used to determine the frequency, cause (System.gc() calls, allocation failure, etc) and duration of GC events.

How to determine GC impact using a profiler

An alternative approach is to use a profiler to collect garbage collection information.

How to determine number (and type) of JVM heap objects using memory dumps

If you determine that garbage collection is a problem, and suspect that this is due to the number of objects in memory, you can perform a heap dump.

To perform a heap dump:

Step 1: Run your program
Step 2: While running, determine the process ID
- One approach is to use jps:
  - For basic details, run jps on the command line. If jps is not on the system PATH, it can be found (on Windows) at C:\Program Files\Java\jdk<VERSION>\bin\jps.exe
  - For more details on each process, run jps -lv instead
- Alternatively, you can use the top command on Linux or Task Manager (Windows) to find the PID (on Windows, the PID column may not be enabled by default)
Step 3: Create a heap dump using jmap -dump:format=b,file=file_name.hprof 123 where 123 is the process id (PID) to create the heap dump for

After a memory dump has been collected, it can be opened in tools such as YourKit profiler and VisualVM to determine the number, type and size of objects. With this information, you should be able to pinpoint the cause of the large number of objects and make changes to your code to reduce or eliminate the objects that are causing the garbage collection overhead.

Step 5: Check Minibatch Size

Another common cause of performance issues is a poorly chosen minibatch size. A minibatch is a number of examples used together for one step of inference and training. Minibatch sizes of 32 to 128 are commonly used, though smaller or larger are sometimes used.

In summary:

If minibatch size is too small (for example, training or inference with 1 example at a time), poor hardware utilization and lower overall throughput is expected
If minibatch size is too large
- Hardware utilization will usually be good
- Iteration times will slow down
- Memory utilization may be too high (leading to out-of-memory errors)

For inference, avoid using minibatch size of 1, as throughput will suffer. Unless there are strict latency requirements, you should use larger minibatch sizes as this will give you the best hardware utilization and hence throughput, and is especially important for GPUs.

For training, you should never use a minibatch size of 1 as overall performance and hardware utilization will be reduced. Network convergence may also suffer. Start with a minibatch size of 32-128, if memory will allow this to be used.

Step 6: Ensure you are not using a single MultiLayerNetwork/ComputationGraph for inference from multiple threads

MultiLayerNetwork and ComputationGraph are not considered thread-safe, and should not be used from multiple threads. That said, most operations such as fit, output, etc use synchronized blocks. These synchronized methods should avoid hard to understand exceptions (race conditions due to concurrent use), they will limit throughput to a single thread (though, note that native operation parallelism will still be parallelized as normal). In summary, using the one network from multiple threads should be avoided as it is not thread safe and can be a performance bottleneck.

Step 7: Check Data Types

As of 1.0.0-beta3 and earlier, ND4J has a global datatype setting that determines the datatype of all arrays. The default value is 32-bit floating point. The data type can be set using Nd4j.setDataType(DataBuffer.Type.FLOAT); for example.

Performance on CPUs can also be reduced for double precision due to the additional memory batchwidth requirements vs. float precision.

You can check the data type setting using:

Step 8: Check workspace configuration for memory management (enabled by default)

In summary, workspaces are enabled by default for all Deeplearning4j networks, and enabling them improves performance and reduces memory requirements. There are very few reasons to disable workspaces.

You can check that workspaces are enabled for your MultiLayerNetwork using:

or for a ComputationGraph using:

You want to see the output as ENABLED output for both training and inference. To change the workspace configuration, use the setter methods, for example: net.getLayerWiseConfigurations().setTrainingWorkspaceMode(WorkspaceMode.ENABLED);

Step 9: Check for a badly configured network or network with layer bottlenecks

Another possible cause (especially for newer users) is a poorly designed network. A network may be poorly designed if:

It has too many layers. A rough guideline:
- More than about 100 layers for a CNN may be too many
- More than about 10 layers for a RNN/LSTM network may be too many
- More than about 20 feed-forward layers may be too many for a MLP
The input/activations are too large
- For CNNs, inputs in the range of 224x224 (for image classification) to 600x600 (for object detection and segmentation) are used. Large image sizes (such as 500x500) are computationally demanding, and much larger than this should be considered too large in most cases.
The output number of classes is too large
- Classification with more than about 10,000 classes can become a performance bottleneck with standard softmax output layers
The layers are too large
- For CNNs, most layers have kernel sizes in the range 2x2 to 7x7, with channels equal to 32 to 1024 (with larger number of channels appearing later in the network). Much larger than this may cause a performance bottleneck.
- For MLPs, most layers have at most 2048 units/neurons (often much smaller). Much larger than this may be too large.
- For RNNs such as LSTMs, layers are typically in the range of 128 to 512, though the largest RNNs may use around 1024 units per layer.
The network has too many parameters
- This is usually a consequence of the other issues already mentioned - too many layers, too large input, too many output classes
- For comparison, less than 1 million parameters would be considered small, and more than about 100 million parameters would be considered very large.
- You can check the number of parameters using MultiLayerNetwork/ComputationGraph.numParams() or MultiLayerNetwork/ComputationGraph.summary()

Note that these are guidelines only, and some reasonable network may exceed the numbers specified here. Some networks can become very large, such as those commonly used for imagenet classification or object detection. However, in these cases, the network is usually carefully designed to provide a good tradeoff between accuracy and computation time.

If your network architecture is significantly outside of the guidelines specified here, you may want to reconsider the design to improve performance.

Step 10: Check for CPU-only ops (when using GPUs)

If you are using CPUs only (nd4j-native backend), you can skip this step, as it only applies when using the GPU (nd4j-cuda) backend.

As of 1.0.0-beta3, a handful of recently added operations do not yet have GPU implementations. Thus, when these layer are used in a network, they will execute on CPU only, irrespective of the nd4j-backend used. GPU support for these layers will be added in an upcoming release.

The layers without GPU support as of 1.0.0-beta3 include:

Convolution3D
Upsampling1D/2D/3D
Deconvolution2D
LocallyConnected1D/2D
SpaceToBatch
SpaceToDepth

Unfortunately, there is no workaround or fix for now, until these operations have GPU implementations completed.

Step 11: Check CPU support for hardware extensions (AVX etc)

If you are running on a GPU, this section does not apply.

When running on older CPUs or those that lack modern AVX extensions such as AVX2 and AVX512, performance will be reduced compared to running on CPUs with these features. Though there is not much you can do about the lack of such features, it is worth knowing about if you are comparing performance between different CPU models.

In summary, CPU models with AVX2 support will perform better than those without it; similarly, AVX512 is an improvement over AVX2.

Step 12: Check other processes using CPU or GPU resources

Another obvious cause of performance issues is other processes using CPU or GPU resources.

For CPU, it is straightforward to see if other processes are using resources using tools such as top (for Linux) or task managed (for Windows).

For NVIDIA CUDA GPUs, nvidia-smi can be used. nvidia-smi is usually installed with the NVIDIA display drivers, and (when run) shows the overall GPU and memory utilization, as well as the GPU utilization of programs running on the system.

On Linux, this is usually on the system path by default. On Windows, it may be found at C:\Program Files\NVIDIA Corporation\NVSMI\nvidia-smi

Step 13: Check OMP_NUM_THREADS performing concurrent inference using CPU in multiple threads simultaneously

If you are using GPUs (nd4j-cuda backend), you can skip this section.

One issue to be aware of when running multiple DL4J networks (or ND4J operations generally) concurrently in multiple threads is the OpenMP number of threads setting. In summary, in ND4J we use OpenMP pallelism at the c++ level to increase operation performance. By default, ND4J will use a value equal to the number of physical CPU cores (not logical cores) as this will give optimal performance

This also applies if the CPU resources are shared with other computationally demanding processes.

Debugging Performance Issues with JVM Profiling

Profiling is a process whereby you can trace how long each method in your code takes to execute, to identify and debug performance bottlenecks.

A full guide to profiling is beyond the scope of this page, but the summary is that you can trace how long each method takes to execute (and where it is being called from) using a profiling tool. This information can then be used to identify bottlenecks (and their causes) in your program.

How to Perform Profiling

The YourKit profiling documentation is quite good. To perform profiling with YourKit:

Install and start YourKit Profiler
Collect a snapshot and analyze

Profiling on Spark

When debugging performance issues for Spark training or inference jobs, it can often be useful to perform profiling here also.

One approach that we have used internally is to combine manual profiling settings (-agentpath JVM argument) with spark-submit arguments for YourKit profiler.

To perform profiling in this manner, 5 steps are required:

Download YourKit profiler to a location on each worker (must be the same location on each worker) and (optionally) the driver
[Optional] Copy the profiling configuration onto each worker (must be the same location on each worker)
Create a local output directory for storing the profiling result files on each worker
Launch the Spark job with the appropriate configuration (see example below)
The snapshots will be saved when the Spark job completes (or is cancelled) to the specified directories.

For example, to perform tracing on both the driver and the workers,

The configuration (tracing_settings_path) is optional. A sample tracing settings file is provided below:

Keras Import API Overview

Keras model import API

KerasModelImport

Reads stored Keras configurations and weights from one of two archives: either as

a single HDF5 file storing model and training JSON configurations and weights
separate text file storing model JSON configuration and HDF5 file storing weights.

importKerasModelAndWeights

public static ComputationGraph importKerasModelAndWeights( InputStream modelHdf5Stream, boolean enforceTrainingConfig)
            throws IOException, UnsupportedKerasConfigurationException, InvalidKerasConfigurationException

Load Keras (Functional API) Model saved using model.save_model(…).

param modelHdf5Stream InputStream containing HDF5 archive storing Keras Model
param enforceTrainingConfig whether to enforce training configuration options
return ComputationGraph
see ComputationGraph

importKerasModelAndWeights

public static ComputationGraph importKerasModelAndWeights(InputStream modelHdf5Stream) throws IOException, UnsupportedKerasConfigurationException, InvalidKerasConfigurationException

Load Keras (Functional API) Model saved using model.save_model(…).

param modelHdf5Stream InputStream containing HDF5 archive storing Keras Model
return ComputationGraph
see ComputationGraph

importKerasSequentialModelAndWeights

public static MultiLayerNetwork importKerasSequentialModelAndWeights(InputStream modelHdf5Stream,
                                                                         boolean enforceTrainingConfig)
            throws IOException, InvalidKerasConfigurationException, UnsupportedKerasConfigurationException

Load Keras Sequential model saved using model.save_model(…).

param modelHdf5Stream InputStream containing HDF5 archive storing Keras Sequential model
param enforceTrainingConfig whether to enforce training configuration options
return ComputationGraph
see ComputationGraph

importKerasSequentialModelAndWeights

public static MultiLayerNetwork importKerasSequentialModelAndWeights(InputStream modelHdf5Stream)
            throws IOException, InvalidKerasConfigurationException, UnsupportedKerasConfigurationException

Load Keras Sequential model saved using model.save_model(…).

param modelHdf5Stream InputStream containing HDF5 archive storing Keras Sequential model
return ComputationGraph
see ComputationGraph

importKerasModelAndWeights

public static ComputationGraph importKerasModelAndWeights(String modelHdf5Filename, int[] inputShape,
                                                              boolean enforceTrainingConfig)
            throws IOException, UnsupportedKerasConfigurationException, InvalidKerasConfigurationException

Load Keras (Functional API) Model saved using model.save_model(…).

param modelHdf5Filename path to HDF5 archive storing Keras Model
param inputShape optional input shape for models that come without such (e.g. notop = false models)
param enforceTrainingConfig whether to enforce training configuration options
return ComputationGraph
throws IOException IO exception
throws InvalidKerasConfigurationException Invalid Keras config
throws UnsupportedKerasConfigurationException Unsupported Keras config
see ComputationGraph

importKerasModelAndWeights

public static ComputationGraph importKerasModelAndWeights(String modelHdf5Filename, boolean enforceTrainingConfig)
            throws IOException, UnsupportedKerasConfigurationException, InvalidKerasConfigurationException

Load Keras (Functional API) Model saved using model.save_model(…).

param modelHdf5Filename path to HDF5 archive storing Keras Model
param enforceTrainingConfig whether to enforce training configuration options
return ComputationGraph
throws IOException IO exception
throws InvalidKerasConfigurationException Invalid Keras config
throws UnsupportedKerasConfigurationException Unsupported Keras config
see ComputationGraph

importKerasModelAndWeights

public static ComputationGraph importKerasModelAndWeights(String modelHdf5Filename)
            throws IOException, UnsupportedKerasConfigurationException, InvalidKerasConfigurationException

Load Keras (Functional API) Model saved using model.save_model(…).

param modelHdf5Filename path to HDF5 archive storing Keras Model
return ComputationGraph
throws IOException IO exception
throws InvalidKerasConfigurationException Invalid Keras config
throws UnsupportedKerasConfigurationException Unsupported Keras config
see ComputationGraph

importKerasSequentialModelAndWeights

public static MultiLayerNetwork importKerasSequentialModelAndWeights(String modelHdf5Filename,
                                                                         int[] inputShape,
                                                                         boolean enforceTrainingConfig)
            throws IOException, InvalidKerasConfigurationException, UnsupportedKerasConfigurationException

Load Keras Sequential model saved using model.save_model(…).

param modelHdf5Filename path to HDF5 archive storing Keras Sequential model
param inputShape optional input shape for models that come without such (e.g. notop = false models)
param enforceTrainingConfig whether to enforce training configuration options
return MultiLayerNetwork
throws IOException IO exception
see MultiLayerNetwork

importKerasSequentialModelAndWeights

public static MultiLayerNetwork importKerasSequentialModelAndWeights(String modelHdf5Filename,
                                                                         boolean enforceTrainingConfig)
            throws IOException, InvalidKerasConfigurationException, UnsupportedKerasConfigurationException

Load Keras Sequential model saved using model.save_model(…).

param modelHdf5Filename path to HDF5 archive storing Keras Sequential model
param enforceTrainingConfig whether to enforce training configuration options
return MultiLayerNetwork
throws IOException IO exception
see MultiLayerNetwork

importKerasSequentialModelAndWeights

public static MultiLayerNetwork importKerasSequentialModelAndWeights(String modelHdf5Filename)
            throws IOException, InvalidKerasConfigurationException, UnsupportedKerasConfigurationException

Load Keras Sequential model saved using model.save_model(…).

param modelHdf5Filename path to HDF5 archive storing Keras Sequential model
return MultiLayerNetwork
throws IOException IO exception
see MultiLayerNetwork

importKerasModelAndWeights

public static ComputationGraph importKerasModelAndWeights(String modelJsonFilename, String weightsHdf5Filename,
                                                              boolean enforceTrainingConfig)
            throws IOException, InvalidKerasConfigurationException, UnsupportedKerasConfigurationException

Load Keras (Functional API) Model for which the configuration and weights were saved separately using calls to model.to_json() and model.save_weights(…).

param modelJsonFilename path to JSON file storing Keras Model configuration
param weightsHdf5Filename path to HDF5 archive storing Keras model weights
param enforceTrainingConfig whether to enforce training configuration options
return ComputationGraph
throws IOException IO exception
see ComputationGraph

importKerasModelAndWeights

public static ComputationGraph importKerasModelAndWeights(String modelJsonFilename, String weightsHdf5Filename)
            throws IOException, InvalidKerasConfigurationException, UnsupportedKerasConfigurationException

Load Keras (Functional API) Model for which the configuration and weights were saved separately using calls to model.to_json() and model.save_weights(…).

param modelJsonFilename path to JSON file storing Keras Model configuration
param weightsHdf5Filename path to HDF5 archive storing Keras model weights
return ComputationGraph
throws IOException IO exception
see ComputationGraph

importKerasSequentialModelAndWeights

public static MultiLayerNetwork importKerasSequentialModelAndWeights(String modelJsonFilename,
                                                                         String weightsHdf5Filename,
                                                                         boolean enforceTrainingConfig)
            throws IOException, InvalidKerasConfigurationException, UnsupportedKerasConfigurationException

Load Keras Sequential model for which the configuration and weights were saved separately using calls to model.to_json() and model.save_weights(…).

param modelJsonFilename path to JSON file storing Keras Sequential model configuration
param weightsHdf5Filename path to HDF5 archive storing Keras model weights
param enforceTrainingConfig whether to enforce training configuration options
return MultiLayerNetwork
throws IOException IO exception
see MultiLayerNetwork

importKerasSequentialModelAndWeights

public static MultiLayerNetwork importKerasSequentialModelAndWeights(String modelJsonFilename,
                                                                         String weightsHdf5Filename)
            throws IOException, InvalidKerasConfigurationException, UnsupportedKerasConfigurationException

Load Keras Sequential model for which the configuration and weights were saved separately using calls to model.to_json() and model.save_weights(…).

param modelJsonFilename path to JSON file storing Keras Sequential model configuration
param weightsHdf5Filename path to HDF5 archive storing Keras model weights
return MultiLayerNetwork
throws IOException IO exception
see MultiLayerNetwork

importKerasModelConfiguration

public static ComputationGraphConfiguration importKerasModelConfiguration(String modelJsonFilename,
                                                                              boolean enforceTrainingConfig)
            throws IOException, InvalidKerasConfigurationException, UnsupportedKerasConfigurationException

Load Keras (Functional API) Model for which the configuration was saved separately using calls to model.to_json() and model.save_weights(…).

param modelJsonFilename path to JSON file storing Keras Model configuration
param enforceTrainingConfig whether to enforce training configuration options
return ComputationGraph
throws IOException IO exception
see ComputationGraph

importKerasModelConfiguration

public static ComputationGraphConfiguration importKerasModelConfiguration(String modelJsonFilename)
            throws IOException, InvalidKerasConfigurationException, UnsupportedKerasConfigurationException

Load Keras (Functional API) Model for which the configuration was saved separately using calls to model.to_json() and model.save_weights(…).

param modelJsonFilename path to JSON file storing Keras Model configuration
return ComputationGraph
throws IOException IO exception
see ComputationGraph

importKerasSequentialConfiguration

public static MultiLayerConfiguration importKerasSequentialConfiguration(String modelJsonFilename,
                                                                             boolean enforceTrainingConfig)
            throws IOException, InvalidKerasConfigurationException, UnsupportedKerasConfigurationException

Load Keras Sequential model for which the configuration was saved separately using calls to model.to_json() and model.save_weights(…).

param modelJsonFilename path to JSON file storing Keras Sequential model configuration
param enforceTrainingConfig whether to enforce training configuration options
return MultiLayerNetwork
throws IOException IO exception
see MultiLayerNetwork

importKerasSequentialConfiguration

public static MultiLayerConfiguration importKerasSequentialConfiguration(String modelJsonFilename)
            throws IOException, InvalidKerasConfigurationException, UnsupportedKerasConfigurationException

Load Keras Sequential model for which the configuration was saved separately using calls to model.to_json() and model.save_weights(…).

param modelJsonFilename path to JSON file storing Keras Sequential model configuration
return MultiLayerNetwork
throws IOException IO exception
see MultiLayerNetwork

Core Layers

KerasPermute

Imports Permute layer from Keras

KerasPermute

Constructor from parsed Keras layer configuration dictionary.

param layerConfig dictionary containing Keras layer configuration
throws InvalidKerasConfigurationException Invalid Keras config
throws UnsupportedKerasConfigurationException Unsupported Keras config

isInputPreProcessor

Constructor from parsed Keras layer configuration dictionary.

param layerConfig dictionary containing Keras layer configuration
param enforceTrainingConfig whether to enforce training-related configuration options
throws InvalidKerasConfigurationException Invalid Keras config
throws UnsupportedKerasConfigurationException Unsupported Keras config

getInputPreprocessor

Gets appropriate DL4J InputPreProcessor for given InputTypes.

param inputType Array of InputTypes
return DL4J InputPreProcessor
throws InvalidKerasConfigurationException Invalid Keras config
see InputPreProcessor

getOutputType

Get layer output type.

param inputType Array of InputTypes
return output type as InputType
throws InvalidKerasConfigurationException Invalid Keras config

KerasFlatten

Imports a Keras Flatten layer as a DL4J {Cnn,Rnn}ToFeedForwardInputPreProcessor.

KerasFlatten

Constructor from parsed Keras layer configuration dictionary.

param layerConfig dictionary containing Keras layer configuration
throws InvalidKerasConfigurationException Invalid Keras config
throws UnsupportedKerasConfigurationException Unsupported Keras config

isInputPreProcessor

Constructor from parsed Keras layer configuration dictionary.

param layerConfig dictionary containing Keras layer configuration
param enforceTrainingConfig whether to enforce training-related configuration options
throws InvalidKerasConfigurationException Invalid Keras config
throws UnsupportedKerasConfigurationException Unsupported Keras config

getInputPreprocessor

Gets appropriate DL4J InputPreProcessor for given InputTypes.

param inputType Array of InputTypes
return DL4J InputPreProcessor
throws InvalidKerasConfigurationException Invalid Keras config
see org.deeplearning4j.nn.conf.InputPreProcessor

getOutputType

Get layer output type.

param inputType Array of InputTypes
return output type as InputType
throws InvalidKerasConfigurationException Invalid Keras config

KerasReshape

Imports Reshape layer from Keras

KerasReshape

Constructor from parsed Keras layer configuration dictionary.

param layerConfig dictionary containing Keras layer configuration
throws InvalidKerasConfigurationException Invalid Keras config
throws UnsupportedKerasConfigurationException Unsupported Keras config

isInputPreProcessor

Constructor from parsed Keras layer configuration dictionary.

param layerConfig dictionary containing Keras layer configuration
param enforceTrainingConfig whether to enforce training-related configuration options
throws InvalidKerasConfigurationException Invalid Keras config
throws UnsupportedKerasConfigurationException Unsupported Keras config

getInputPreprocessor

Gets appropriate DL4J InputPreProcessor for given InputTypes.

param inputType Array of InputTypes
return DL4J InputPreProcessor
throws InvalidKerasConfigurationException Invalid Keras config
see org.deeplearning4j.nn.conf.InputPreProcessor

getOutputType

Get layer output type.

param inputType Array of InputTypes
return output type as InputType
throws InvalidKerasConfigurationException Invalid Keras config

KerasMerge

Imports a Keras Merge layer as a DL4J Merge (graph) vertex.

TODO: handle axes arguments that alter merge behavior (requires changes to DL4J?)

KerasMerge

Pass-through constructor from KerasLayer

param kerasVersion major keras version
throws UnsupportedKerasConfigurationException Unsupported Keras config

getOutputType

Constructor from parsed Keras layer configuration dictionary.

param layerConfig dictionary containing Keras layer configuration.
throws InvalidKerasConfigurationException Invalid Keras config
throws UnsupportedKerasConfigurationException Unsupported Keras config

KerasDropout

Imports a Dropout layer from Keras.

KerasDropout

Constructor from parsed Keras layer configuration dictionary.

param layerConfig dictionary containing Keras layer configuration
throws InvalidKerasConfigurationException Invalid Keras config
throws UnsupportedKerasConfigurationException Unsupported Keras config

getOutputType

Constructor from parsed Keras layer configuration dictionary.

param layerConfig dictionary containing Keras layer configuration
param enforceTrainingConfig whether to enforce training-related configuration options
throws InvalidKerasConfigurationException Invalid Keras config
throws UnsupportedKerasConfigurationException Unsupported Keras config

getDropoutLayer

Get DL4J DropoutLayer.

return DropoutLayer

KerasMasking

Imports Keras masking layers.

KerasMasking

Constructor from parsed Keras layer configuration dictionary.

param layerConfig dictionary containing Keras layer configuration
throws InvalidKerasConfigurationException Invalid Keras config
throws UnsupportedKerasConfigurationException Unsupported Keras config

getOutputType

Constructor from parsed Keras layer configuration dictionary.

param layerConfig dictionary containing Keras layer configuration
param enforceTrainingConfig whether to enforce training-related configuration options
throws InvalidKerasConfigurationException Invalid Keras config
throws UnsupportedKerasConfigurationException Unsupported Keras config

getMaskingLayer

Get DL4J MaskZeroLayer.

return MaskZeroLayer

KerasSpatialDropout

Keras wrapper for DL4J dropout layer with SpatialDropout, works 1D-3D.

KerasSpatialDropout

Pass-through constructor from KerasLayer

param kerasVersion major keras version
throws UnsupportedKerasConfigurationException Unsupported Keras config

getOutputType

Constructor from parsed Keras layer configuration dictionary.

param layerConfig dictionary containing Keras layer configuration.
throws InvalidKerasConfigurationException Invalid Keras config
throws UnsupportedKerasConfigurationException Unsupported Keras config

getSpatialDropoutLayer

Get DL4J DropoutLayer with spatial dropout.

return DropoutLayer

KerasLambda

Wraps a DL4J SameDiffLambda into a KerasLayer

KerasLambda

Constructor from parsed Keras layer configuration dictionary.

param layerConfig dictionary containing Keras layer configuration
throws InvalidKerasConfigurationException Invalid Keras config
throws UnsupportedKerasConfigurationException Unsupported Keras config

getOutputType

Constructor from parsed Keras layer configuration dictionary.

param layerConfig dictionary containing Keras layer configuration
param enforceTrainingConfig whether to enforce training-related configuration options
throws InvalidKerasConfigurationException Invalid Keras config
throws UnsupportedKerasConfigurationException Unsupported Keras config

getSameDiffLayer

Get DL4J SameDiffLayer.

return SameDiffLayer

KerasActivation

Imports an Activation layer from Keras.

KerasActivation

Constructor from parsed Keras layer configuration dictionary.

param layerConfig dictionary containing Keras layer configuration
throws InvalidKerasConfigurationException Invalid Keras config
throws UnsupportedKerasConfigurationException Unsupported Keras config

getOutputType

Constructor from parsed Keras layer configuration dictionary.

param layerConfig dictionary containing Keras layer configuration
param enforceTrainingConfig whether to enforce training-related configuration options
throws InvalidKerasConfigurationException Invalid Keras config
throws UnsupportedKerasConfigurationException Unsupported Keras config

getActivationLayer

Get DL4J ActivationLayer.

return ActivationLayer

KerasDense

Imports a Dense layer from Keras.

KerasDense

Pass-through constructor from KerasLayer

param kerasVersion major keras version
throws UnsupportedKerasConfigurationException Unsupported Keras config

getDenseLayer

Constructor from parsed Keras layer configuration dictionary.

param layerConfig dictionary containing Keras layer configuration
throws InvalidKerasConfigurationException Invalid Keras config
throws UnsupportedKerasConfigurationException Unsupported Keras config

getOutputType

Get layer output type.

param inputType Array of InputTypes
return output type as InputType
throws InvalidKerasConfigurationException Invalid Keras config

getNumParams

Returns number of trainable parameters in layer.

return number of trainable parameters (2)

setWeights

Set weights for layer.

param weights Dense layer weights

KerasRepeatVector

Imports a Keras RepeatVector layer

KerasRepeatVector

Constructor from parsed Keras layer configuration dictionary.

param layerConfig dictionary containing Keras layer configuration
throws InvalidKerasConfigurationException Invalid Keras config
throws UnsupportedKerasConfigurationException Unsupported Keras config

getOutputType

Constructor from parsed Keras layer configuration dictionary.

param layerConfig dictionary containing Keras layer configuration
param enforceTrainingConfig whether to enforce training-related configuration options
throws InvalidKerasConfigurationException Invalid Keras config
throws UnsupportedKerasConfigurationException Unsupported Keras config

getRepeatVectorLayer

Get DL4J RepeatVector.

return RepeatVector

Convolutional Layers

KerasConvolution2D

Imports a 2D Convolution layer from Keras.

KerasConvolution2D

public KerasConvolution2D(Integer kerasVersion) throws UnsupportedKerasConfigurationException

Pass-through constructor from KerasLayer

param kerasVersion major keras version
throws UnsupportedKerasConfigurationException Unsupported Keras config

getConvolution2DLayer

public ConvolutionLayer getConvolution2DLayer()

Constructor from parsed Keras layer configuration dictionary.

param layerConfig dictionary containing Keras layer configuration
throws InvalidKerasConfigurationException Invalid Keras config
throws UnsupportedKerasConfigurationException Unsupported Keras config

getOutputType

public InputType getOutputType(InputType... inputType) throws InvalidKerasConfigurationException

Get layer output type.

param inputType Array of InputTypes
return output type as InputType
throws InvalidKerasConfigurationException Invalid Keras config

KerasCropping2D

Imports a Keras Cropping 2D layer.

KerasCropping2D

public KerasCropping2D(Map<String, Object> layerConfig)
            throws InvalidKerasConfigurationException, UnsupportedKerasConfigurationException

Constructor from parsed Keras layer configuration dictionary.

param layerConfig dictionary containing Keras layer configuration.
throws InvalidKerasConfigurationException Invalid Keras config
throws UnsupportedKerasConfigurationException Unsupported Keras config

getCropping2DLayer

public Cropping2D getCropping2DLayer()

Constructor from parsed Keras layer configuration dictionary.

param layerConfig dictionary containing Keras layer configuration
param enforceTrainingConfig whether to enforce training-related configuration options
throws InvalidKerasConfigurationException Invalid Keras config
throws UnsupportedKerasConfigurationException Unsupported Keras config

getOutputType

public InputType getOutputType(InputType... inputType) throws InvalidKerasConfigurationException

Get layer output type.

param inputType Array of InputTypes
return output type as InputType
throws InvalidKerasConfigurationException Invalid Keras config

KerasUpsampling3D

Keras Upsampling3D layer support

KerasUpsampling3D

public KerasUpsampling3D(Map<String, Object> layerConfig)
            throws InvalidKerasConfigurationException, UnsupportedKerasConfigurationException

Constructor from parsed Keras layer configuration dictionary.

param layerConfig dictionary containing Keras layer configuration.
throws InvalidKerasConfigurationException Invalid Keras configuration exception
throws UnsupportedKerasConfigurationException Unsupported Keras configuration exception

getUpsampling3DLayer

public Upsampling3D getUpsampling3DLayer()

Constructor from parsed Keras layer configuration dictionary.

param layerConfig dictionary containing Keras layer configuration
param enforceTrainingConfig whether to enforce training-related configuration options
throws InvalidKerasConfigurationException Invalid Keras configuration exception
throws UnsupportedKerasConfigurationException Invalid Keras configuration exception

getOutputType

public InputType getOutputType(InputType... inputType) throws InvalidKerasConfigurationException

Get layer output type.

param inputType Array of InputTypes
return output type as InputType
throws InvalidKerasConfigurationException Invalid Keras config

KerasConvolution1D

Imports a 1D Convolution layer from Keras.

KerasConvolution1D

public KerasConvolution1D(Integer kerasVersion) throws UnsupportedKerasConfigurationException

Pass-through constructor from KerasLayer

param kerasVersion major keras version
throws UnsupportedKerasConfigurationException

getConvolution1DLayer

public Convolution1DLayer getConvolution1DLayer()

Constructor from parsed Keras layer configuration dictionary.

param layerConfig dictionary containing Keras layer configuration
throws InvalidKerasConfigurationException
throws UnsupportedKerasConfigurationException

getOutputType

public InputType getOutputType(InputType... inputType) throws InvalidKerasConfigurationException

Get layer output type.

param inputType Array of InputTypes
return output type as InputType
throws InvalidKerasConfigurationException

getInputPreprocessor

public InputPreProcessor getInputPreprocessor(InputType... inputType) throws InvalidKerasConfigurationException

Gets appropriate DL4J InputPreProcessor for given InputTypes.

param inputType Array of InputTypes
return DL4J InputPreProcessor
throws InvalidKerasConfigurationException Invalid Keras configuration exception
see org.deeplearning4j.nn.conf.InputPreProcessor

setWeights

public void setWeights(Map<String, INDArray> weights) throws InvalidKerasConfigurationException

Set weights for layer.

param weights Map from parameter name to INDArray.

KerasUpsampling1D

Keras Upsampling1D layer support

KerasUpsampling1D

public KerasUpsampling1D(Map<String, Object> layerConfig)
            throws InvalidKerasConfigurationException, UnsupportedKerasConfigurationException

Constructor from parsed Keras layer configuration dictionary.

param layerConfig dictionary containing Keras layer configuration.
throws InvalidKerasConfigurationException Invalid Keras configuration exception
throws UnsupportedKerasConfigurationException Unsupported Keras configuration exception

getUpsampling1DLayer

public Upsampling1D getUpsampling1DLayer()

Constructor from parsed Keras layer configuration dictionary.

param layerConfig dictionary containing Keras layer configuration
param enforceTrainingConfig whether to enforce training-related configuration options
throws InvalidKerasConfigurationException Invalid Keras configuration exception
throws UnsupportedKerasConfigurationException Invalid Keras configuration exception

getOutputType

public InputType getOutputType(InputType... inputType) throws InvalidKerasConfigurationException

Get layer output type.

param inputType Array of InputTypes
return output type as InputType
throws InvalidKerasConfigurationException Invalid Keras config

KerasAtrousConvolution2D

Keras 1D atrous / dilated convolution layer. Note that in keras 2 this layer has been removed and dilations are now available through the “dilated” argument in regular Conv1D layers

author: Max Pumperla

KerasAtrousConvolution2D

public KerasAtrousConvolution2D(Integer kerasVersion) throws UnsupportedKerasConfigurationException

Pass-through constructor from KerasLayer

param kerasVersion major keras version
throws UnsupportedKerasConfigurationException Unsupported Keras config

getAtrousConvolution2D

public ConvolutionLayer getAtrousConvolution2D()

Constructor from parsed Keras layer configuration dictionary.

param layerConfig dictionary containing Keras layer configuration
throws InvalidKerasConfigurationException Invalid Keras config
throws UnsupportedKerasConfigurationException Unsupported Keras config

getOutputType

public InputType getOutputType(InputType... inputType) throws InvalidKerasConfigurationException

Get layer output type.

param inputType Array of InputTypes
return output type as InputType
throws InvalidKerasConfigurationException Invalid Keras config

KerasAtrousConvolution1D

Keras 1D atrous / dilated convolution layer. Note that in keras 2 this layer has been removed and dilations are now available through the “dilated” argument in regular Conv1D layers

author: Max Pumperla

KerasAtrousConvolution1D

public KerasAtrousConvolution1D(Integer kerasVersion) throws UnsupportedKerasConfigurationException

Pass-through constructor from KerasLayer

param kerasVersion major keras version
throws UnsupportedKerasConfigurationException Unsupported Keras config

getAtrousConvolution1D

public Convolution1DLayer getAtrousConvolution1D()

Constructor from parsed Keras layer configuration dictionary.

param layerConfig dictionary containing Keras layer configuration
throws InvalidKerasConfigurationException Invalid Keras config
throws UnsupportedKerasConfigurationException Unsupported Keras config

getOutputType

public InputType getOutputType(InputType... inputType) throws InvalidKerasConfigurationException

Get layer output type.

param inputType Array of InputTypes
return output type as InputType
throws InvalidKerasConfigurationException Invalid Keras config

KerasCropping3D

Imports a Keras Cropping 3D layer.

KerasCropping3D

public KerasCropping3D(Map<String, Object> layerConfig)
            throws InvalidKerasConfigurationException, UnsupportedKerasConfigurationException

Constructor from parsed Keras layer configuration dictionary.

param layerConfig dictionary containing Keras layer configuration.
throws InvalidKerasConfigurationException Invalid Keras config
throws UnsupportedKerasConfigurationException Unsupported Keras config

getCropping3DLayer

public Cropping3D getCropping3DLayer()

Constructor from parsed Keras layer configuration dictionary.

param layerConfig dictionary containing Keras layer configuration
param enforceTrainingConfig whether to enforce training-related configuration options
throws InvalidKerasConfigurationException Invalid Keras config
throws UnsupportedKerasConfigurationException Unsupported Keras config

getOutputType

public InputType getOutputType(InputType... inputType) throws InvalidKerasConfigurationException

Get layer output type.

param inputType Array of InputTypes
return output type as InputType
throws InvalidKerasConfigurationException Invalid Keras config

KerasZeroPadding2D

Imports a Keras ZeroPadding 2D layer.

KerasZeroPadding2D

public KerasZeroPadding2D(Map<String, Object> layerConfig)
                    throws InvalidKerasConfigurationException, UnsupportedKerasConfigurationException

Constructor from parsed Keras layer configuration dictionary.

param layerConfig dictionary containing Keras layer configuration.
throws InvalidKerasConfigurationException Invalid Keras config
throws UnsupportedKerasConfigurationException Unsupported Keras config

getZeroPadding2DLayer

public ZeroPaddingLayer getZeroPadding2DLayer()

Constructor from parsed Keras layer configuration dictionary.

param layerConfig dictionary containing Keras layer configuration
param enforceTrainingConfig whether to enforce training-related configuration options
throws InvalidKerasConfigurationException Invalid Keras config
throws UnsupportedKerasConfigurationException Unsupported Keras config

getOutputType

public InputType getOutputType(InputType... inputType) throws InvalidKerasConfigurationException

Get layer output type.

param inputType Array of InputTypes
return output type as InputType
throws InvalidKerasConfigurationException Invalid Keras config

KerasConvolution3D

Imports a 3D Convolution layer from Keras.

KerasConvolution3D

public KerasConvolution3D(Integer kerasVersion) throws UnsupportedKerasConfigurationException

Pass-through constructor from KerasLayer

param kerasVersion major keras version
throws UnsupportedKerasConfigurationException Unsupported Keras config

getConvolution3DLayer

public ConvolutionLayer getConvolution3DLayer()

Constructor from parsed Keras layer configuration dictionary.

param layerConfig dictionary containing Keras layer configuration
throws InvalidKerasConfigurationException Invalid Keras config
throws UnsupportedKerasConfigurationException Unsupported Keras config

getOutputType

public InputType getOutputType(InputType... inputType) throws InvalidKerasConfigurationException

Get layer output type.

param inputType Array of InputTypes
return output type as InputType
throws InvalidKerasConfigurationException Invalid Keras config

KerasDeconvolution2D

Imports a 2D Deconvolution layer from Keras.

KerasDeconvolution2D

public KerasDeconvolution2D(Integer kerasVersion) throws UnsupportedKerasConfigurationException

Pass-through constructor from KerasLayer

param kerasVersion major keras version
throws UnsupportedKerasConfigurationException Unsupported Keras config

getDeconvolution2DLayer

public Deconvolution2D getDeconvolution2DLayer()

Constructor from parsed Keras layer configuration dictionary.

param layerConfig dictionary containing Keras layer configuration
throws InvalidKerasConfigurationException Invalid Keras config
throws UnsupportedKerasConfigurationException Unsupported Keras config

getOutputType

public InputType getOutputType(InputType... inputType) throws InvalidKerasConfigurationException

Get layer output type.

param inputType Array of InputTypes
return output type as InputType
throws InvalidKerasConfigurationException Invalid Keras config

KerasZeroPadding3D

Imports a Keras ZeroPadding 3D layer.

KerasZeroPadding3D

public KerasZeroPadding3D(Map<String, Object> layerConfig)
                    throws InvalidKerasConfigurationException, UnsupportedKerasConfigurationException

Constructor from parsed Keras layer configuration dictionary.

param layerConfig dictionary containing Keras layer configuration.
throws InvalidKerasConfigurationException Invalid Keras config
throws UnsupportedKerasConfigurationException Unsupported Keras config

getZeroPadding3DLayer

public ZeroPadding3DLayer getZeroPadding3DLayer()

Constructor from parsed Keras layer configuration dictionary.

param layerConfig dictionary containing Keras layer configuration
param enforceTrainingConfig whether to enforce training-related configuration options
throws InvalidKerasConfigurationException Invalid Keras config
throws UnsupportedKerasConfigurationException Unsupported Keras config

getOutputType

public InputType getOutputType(InputType... inputType) throws InvalidKerasConfigurationException

Get layer output type.

param inputType Array of InputTypes
return output type as InputType
throws InvalidKerasConfigurationException Invalid Keras config

KerasConvolutionUtils

Utility functionality for Keras convolution layers.

getConvolutionModeFromConfig

public static ConvolutionMode getConvolutionModeFromConfig(Map<String, Object> layerConfig,
                                                               KerasLayerConfiguration conf)
            throws InvalidKerasConfigurationException, UnsupportedKerasConfigurationException

Get (convolution) stride from Keras layer configuration.

param layerConfig dictionary containing Keras layer configuration
return Strides array from Keras configuration
throws InvalidKerasConfigurationException Invalid Keras config

KerasZeroPadding1D

Imports a Keras ZeroPadding 1D layer.

KerasZeroPadding1D

public KerasZeroPadding1D(Map<String, Object> layerConfig)
            throws InvalidKerasConfigurationException, UnsupportedKerasConfigurationException

Constructor from parsed Keras layer configuration dictionary.

param layerConfig dictionary containing Keras layer configuration.
throws InvalidKerasConfigurationException Invalid Keras config
throws UnsupportedKerasConfigurationException Unsupported Keras config

getZeroPadding1DLayer

public ZeroPadding1DLayer getZeroPadding1DLayer()

Constructor from parsed Keras layer configuration dictionary.

param layerConfig dictionary containing Keras layer configuration
param enforceTrainingConfig whether to enforce training-related configuration options
throws InvalidKerasConfigurationException Invalid Keras config
throws UnsupportedKerasConfigurationException Unsupported Keras config

getOutputType

public InputType getOutputType(InputType... inputType) throws InvalidKerasConfigurationException

Get layer output type.

param inputType Array of InputTypes
return output type as InputType
throws InvalidKerasConfigurationException Invalid Keras config

KerasCropping1D

Imports a Keras Cropping 1D layer.

KerasCropping1D

public KerasCropping1D(Map<String, Object> layerConfig)
            throws InvalidKerasConfigurationException, UnsupportedKerasConfigurationException

Constructor from parsed Keras layer configuration dictionary.

param layerConfig dictionary containing Keras layer configuration.
throws InvalidKerasConfigurationException Invalid Keras config
throws UnsupportedKerasConfigurationException Unsupported Keras config

getCropping1DLayer

public Cropping1D getCropping1DLayer()

Constructor from parsed Keras layer configuration dictionary.

param layerConfig dictionary containing Keras layer configuration
param enforceTrainingConfig whether to enforce training-related configuration options
throws InvalidKerasConfigurationException Invalid Keras config
throws UnsupportedKerasConfigurationException Unsupported Keras config

getOutputType

public InputType getOutputType(InputType... inputType) throws InvalidKerasConfigurationException

Get layer output type.

param inputType Array of InputTypes
return output type as InputType
throws InvalidKerasConfigurationException Invalid Keras config

KerasSpaceToDepth

Constructor from parsed Keras layer configuration dictionary.

KerasSpaceToDepth

public KerasSpaceToDepth(Map<String, Object> layerConfig, boolean enforceTrainingConfig)
            throws InvalidKerasConfigurationException, UnsupportedKerasConfigurationException

Constructor from parsed Keras layer configuration dictionary.

param layerConfig dictionary containing Keras layer configuration
param enforceTrainingConfig whether to enforce training-related configuration options
throws InvalidKerasConfigurationException Invalid Keras configuration exception
throws UnsupportedKerasConfigurationException Unsupported Keras configuration exception

getSpaceToDepthLayer

public SpaceToDepthLayer getSpaceToDepthLayer()

Get DL4J SpaceToDepth layer.

return SpaceToDepth layer

getOutputType

public InputType getOutputType(InputType... inputType) throws InvalidKerasConfigurationException

Get layer output type.

param inputType Array of InputTypes
return output type as InputType
throws InvalidKerasConfigurationException Invalid Keras config

KerasUpsampling2D

Keras Upsampling2D layer support

KerasUpsampling2D

public KerasUpsampling2D(Map<String, Object> layerConfig)
            throws InvalidKerasConfigurationException, UnsupportedKerasConfigurationException

Constructor from parsed Keras layer configuration dictionary.

param layerConfig dictionary containing Keras layer configuration.
throws InvalidKerasConfigurationException Invalid Keras configuration exception
throws UnsupportedKerasConfigurationException Unsupported Keras configuration exception

getUpsampling2DLayer

public Upsampling2D getUpsampling2DLayer()

Constructor from parsed Keras layer configuration dictionary.

param layerConfig dictionary containing Keras layer configuration
param enforceTrainingConfig whether to enforce training-related configuration options
throws InvalidKerasConfigurationException Invalid Keras configuration exception
throws UnsupportedKerasConfigurationException Invalid Keras configuration exception

getOutputType

public InputType getOutputType(InputType... inputType) throws InvalidKerasConfigurationException

Get layer output type.

param inputType Array of InputTypes
return output type as InputType
throws InvalidKerasConfigurationException Invalid Keras config

KerasSeparableConvolution2D

Keras separable convolution 2D layer support

KerasSeparableConvolution2D

public KerasSeparableConvolution2D(Integer kerasVersion) throws UnsupportedKerasConfigurationException

Pass-through constructor from KerasLayer

param kerasVersion major keras version
throws UnsupportedKerasConfigurationException Unsupported Keras configuration

setWeights

public void setWeights(Map<String, INDArray> weights) throws InvalidKerasConfigurationException

Constructor from parsed Keras layer configuration dictionary.

param layerConfig dictionary containing Keras layer configuration
throws InvalidKerasConfigurationException Invalid Keras configuration
throws UnsupportedKerasConfigurationException Unsupported Keras configuration

getSeparableConvolution2DLayer

public SeparableConvolution2D getSeparableConvolution2DLayer()

Get DL4J SeparableConvolution2D.

return SeparableConvolution2D

getOutputType

public InputType getOutputType(InputType... inputType) throws InvalidKerasConfigurationException

Get layer output type.

param inputType Array of InputTypes
return output type as InputType
throws InvalidKerasConfigurationException Invalid Keras config

KerasDepthwiseConvolution2D

Keras depth-wise convolution 2D layer support

KerasDepthwiseConvolution2D

public KerasDepthwiseConvolution2D(Integer kerasVersion) throws UnsupportedKerasConfigurationException

Pass-through constructor from KerasLayer

param kerasVersion major keras version
throws UnsupportedKerasConfigurationException Unsupported Keras configuration

setWeights

public void setWeights(Map<String, INDArray> weights) throws InvalidKerasConfigurationException

Constructor from parsed Keras layer configuration dictionary.

param layerConfig dictionary containing Keras layer configuration
throws InvalidKerasConfigurationException Invalid Keras configuration
throws UnsupportedKerasConfigurationException Unsupported Keras configuration

getDepthwiseConvolution2DLayer

public DepthwiseConvolution2D getDepthwiseConvolution2DLayer()

Get DL4J DepthwiseConvolution2D.

return DepthwiseConvolution2D

getOutputType

public InputType getOutputType(InputType... inputType) throws InvalidKerasConfigurationException

Get layer output type.

param inputType Array of InputTypes
return output type as InputType
throws InvalidKerasConfigurationException Invalid Keras config

Visualization

How to visualize, monitor and debug neural network learning.

Note: This information here pertains to DL4J versions 1.0.0-beta6 and later.

DL4J Provides a user interface to visualize in your browser (in real time) the current network status and progress of training. The UI is typically used to help with tuning neural networks - i.e., the selection of hyperparameters (such as learning rate) to obtain good performance for a network.

Step 1: Add the Deeplearning4j UI dependency to your project.

Step 2: Enable the UI in your project

This is relatively straightforward:

To access the UI, open your browser and go to http://localhost:9000/train/overview. You can set the port by using the org.deeplearning4j.ui.port system property: i.e., to use port 9001, pass the following to the JVM on launch: -Dorg.deeplearning4j.ui.port=9001

Information will then be collected and routed to the UI when you call the fit method on your network.

The overview page (one of 3 available pages) contains the following information:

Top left: score vs iteration chart - this is the value of the loss function on the current minibatch
Top right: model and training information
Bottom left: Ratio of parameters to updates (by layer) for all network weights vs. iteration
Bottom right: Standard deviations (vs. time) of: activations, gradients and updates

Note that for the bottom two charts, these are displayed as the logarithm (base 10) of the values. Thus a value of -3 on the update: parameter ratio chart corresponds to a ratio of 10-3 = 0.001.

The ratio of updates to parameters is specifically the ratio of mean magnitudes of these values (i.e., log10(mean(abs(updates))/mean(abs(parameters))).

See the later section of this page on how to use these values in practice.

The model page contains a graph of the neural network layers, which operates as a selection mechanism. Click on a layer to display information for it.

On the right, the following charts are available, after selecting a layer:

Table of layer information
Update to parameter ratio for this layer, as per the overview page. The components of this ratio (the parameter and update mean magnitudes) are also available via tabs.
Layer activations (mean and mean +/- 2 standard deviations) over time
Histograms of parameters and updates, for each parameter type
Learning rate vs. time (note this will be flat, unless learning rate schedules are used)

Note: parameters are labeled as follows: weights (W) and biases (b). For recurrent neural networks, W refers to the weights connecting the layer to the layer below, and RW refers to the recurrent weights (i.e., those between time steps).

The DL4J UI can be used with Spark. However, as of 0.7.0, conflicting dependencies mean that running the UI and Spark is the same JVM can be difficult.

Two alternatives are available:

Collect and save the relevant stats, to be visualized (offline) at a later point
Run the UI in a separate server, and Use the remote UI functionality to upload the data from the Spark master to your UI instance

Collecting Stats for Later Offline Use

Then, later you can load and display the saved information using:

Using the Remote UI Functionality

First, in the JVM running the UI (note this is the server):

This will require the deeplearning4j-ui dependency. (NOTE THIS IS NOT THE CLIENT THIS IS YOUR SERVER - SEE BELOW FOR THE CLIENT WHICH USES: deeplearning4j-ui-model)

To avoid dependency conflicts with Spark, you should use the deeplearning4j-ui-model dependency to get the StatsListener, not the full deeplearning4j-ui UI dependency.

Note: you should replace UI_MACHINE_IP with the IP address of the machine running the user interface instance.

Tuning neural networks is often more an art than a science. However, here's some ideas that may be useful:

Overview Page - Model Score vs. Iteration Chart

The score vs. iteration should (overall) go down over time.

If the score increases consistently, your learning rate is likely set too high. Try reducing it until scores become more stable.
Increasing scores can also be indicative of other network issues, such as incorrect data normalization
If the score is flat or decreases very slowly (over a few hundred iterations) (a) your learning rate may be too low, or (b) you might be having difficulties with optimization. In the latter case, if you are using the SGD updater, try a different updater such as Nesterovs (momentum), RMSProp or Adagrad.
Note that data that isn't shuffled (i.e., each minibatch contains only one class, for classification) can result in very rough or abnormal-looking score vs. iteration graphs
Some noise in this line chart is expected (i.e., the line will go up and down within a small range). However, if the scores vary quite significantly between runs variation is very large, this can be a problem
- The issues mentioned above (learning rate, normalization, data shuffling) may contribute to this.
- Setting the minibatch size to a very small number of examples can also contribute to noisy score vs. iteration graphs, and might lead to optimization difficulties

Overview Page and Model Page - Using the Update: Parameter Ratio Chart

The ratio of mean magnitude of updates to parameters is provided on both the overview and model pages
- "Mean magnitude" = the average of the absolute value of the parameters or updates at the current time step
The most important use of this ratio is in selecting a learning rate. As a rule of thumb: this ratio should be around 1:1000 = 0.001. On the (log10) chart, this corresponds to a value of -3 (i.e., 10-3 = 0.001)
- Note that is a rough guide only, and may not be appropriate for all networks. It's often a good starting point, however.
- If the ratio diverges significantly from this (for example, > -2 (i.e., 10-2=0.01) or < -4 (i.e., 10-4=0.0001), your parameters may be too unstable to learn useful features, or may change too slowly to learn useful features
- To change this ratio, adjust your learning rate (or sometimes, parameter initialization). In some networks, you may need to set the learning rate differently for different layers.
Keep an eye out for unusually large spikes in the ratio: this may indicate exploding gradients

Model Page: Layer Activations (vs. Time) Chart

This chart can be used to detect vanishing or exploding activations (due to poor weight initialization, too much regularization, lack of data normalization, or too high a learning rate).

This chart should ideally stabilize over time (usually a few hundred iterations)
A good standard deviation for the activations is on the order of 0.5 to 2.0. Significantly outside of this range may indicate one of the problems mentioned above.

Model Page: Layer Parameters Histogram

The layer parameters histogram is displayed for the most recent iteration only.

For weights, these histograms should have an approximately Gaussian (normal) distribution, after some time
For biases, these histograms will generally start at 0, and will usually end up being approximately Gaussian
- One exception to this is for LSTM recurrent neural network layers: by default, the biases for one gate (the forget gate) are set to 1.0 (by default, though this is configurable), to help in learning dependencies across long time periods. This results in the bias graphs initially having many biases around 0.0, with another set of biases around 1.0
Keep an eye out for parameters that are diverging to +/- infinity: this may be due to too high a learning rate, or insufficient regularization (try adding some L2 regularization to your network).
Keep an eye out for biases that become very large. This can sometimes occur in the output layer for classification, if the distribution of classes is very imbalanced

Model Page: Layer Updates Histogram

The layer update histogram is displayed for the most recent iteration only.

Note that these are the updates - i.e., the gradients after applying learning rate, momentum, regularization etc
As with the parameter graphs, these should have an approximately Gaussian (normal) distribution
Keep an eye out for very large values: this can indicate exploding gradients in your network
- Exploding gradients are problematic as they can 'mess up' the parameters of your network
- In this case, it may indicate a weight initialization, learning rate or input/labels data normalization issue

Model Page: Parameter Learning Rates Chart

This chart simply shows the learning rates of the parameters of selected layer, over time.

If you are not using learning rate schedules, the chart will be flat. If you are using learning rate schedules, you can use this chart to track the current value of the learning rate (for each parameter), over time.

The recommended solution (for Maven) is to use the Maven Shade plugin to produce an uber-jar, configured as follows:

Then, create your uber-jar with mvn package and run via cd target && java -cp dl4j-examples-0.9.1-bin.jar org.deeplearning4j.examples.userInterface.UIExample. Note the "-bin" suffix for the generated JAR file: this includes all dependencies.

Note also that this Maven Shade approach is configured for DL4J's examples repository.