Eclipse DeepLearning4J

Eclipse Deeplearning4j is a suite of tools for running deep learning on the JVM. It's the only framework that allows you to train models from java while interoperating with the python ecosystem through a mix of python execution via our cpython bindings, model import support, and interop of other runtimes such as tensorflow-java and onnxruntime.

The use cases include importing and retraining models (Pytorch, Tensorflow, Keras) models and deploying in JVM Micro service environments, mobile devices, IoT, and Apache Spark. It is a great compliment to your python environment for running models built in python, deployed to or packaged for other environments.

Deeplearning4j has several submodules including:

1. Samediff: a tensorflow/pytorch like framework for execution of complex graphs. This framework is lower level, but very flexible. It's also the base api for running onnx and tensorflow graphs.

2. Nd4j: numpy ++ for java. Contains a mix of numpy operations and tensorflow/pytorch operations.

3. Libnd4j: A lightweight, standalone c++ library enable math code to run on different devices. Optimizable for running on a wide variety of devices.

4. Python4j: A python script execution framework easing deployment of python scripts in to production.

5. Apache Spark Integration: An integration with the Apache Spark framework enabling execution of deep learning pipelines on spark

6. Datavec: A data transformation library converting raw input data to tensors suitable for running neural networks on.

How to use this website

This website follows the divio framework layout. This website has several sections of documentation following this layout. Below is an overview of the sections of the site:

1. Multi project contains all cross project documentation such as end to end training and other whole project related documentation. This should be the default entry point for those getting started.

2. Deeplearning4j contains all of the documentation related to the core deeplearning4j apis such as the multi layer network and the computation graph. Consider this the high level framework for building neural networks. If you would like something lower level like tensorflow or pytorch, consider using samediff

3. Samediff contains all the documentation related to the samediff submodule of ND4j. Samediff is a lower level api for building neural networks similar to pytorch or tensorflow with built in automatic differentiation.

4. Datavec contains all the documentation related to our data transformation library datavec.

5. Python4j contains all the documentation related to our cpython execution framework python4j.

6. Libnd4j contains all the documentation related to our underlying C++ framework libnd4j.

7. Apache Spark contains all of the documentation related to our Apache Spark integration.

8. Concepts/Theory contains all of the documentation related to general mathematical or computer science theory needed to understand various aspects of the framework.

Open Source

The libraries are completely open-source, Apache 2.0 under open governance at the Eclipse foundation. The Eclipse Deeplearning4j project welcomes all contributions. See our community and our Contribution guide to get involved.

JVM/Python/C++

Deeplearning4j can either be a compliment to your existing workflows in python and c++ or a standalone library for you to build and deploy models. Use what components you find useful.

Version 1.0.0-beta7

Read the announcement at https://blog.konduit.ai/2020/05/14/deeplearning4j-1-0-0-beta7-released/ for the highlights of this release.

Deeplearning4j

Features and Enhancements

• Added Keras model import support for tf.keras models Link, Link

  • Full inference and training support is available for ops/layers in the tf.keras namespace; inference only for general Tensorflow operations outside of the tf.keras namespace

  • Note also improvements to Keras import for reshape, permute, etc operations due to NHWC and NWC support in DL4J

• DL4J now supports NHWC (channels last) data format for all CNN 2D layers, in addition to NCHW Link

• DL4J now supports NWC (channels last - [minibatch, sequence_length, size]) for all RNN and CNN 1D layers, in addition to NCW Link

• Added Deconvolution3D layer Link

• Keras import: added ReLU, ELU and Softmax advanced activation layers Link and Swish activation function Link

• Added DL4J SameDiffLoss class (for easily-defined DL4J ILossFunction's via SameDiff) Link

• Useful exceptions are now thrown when attempting to perform unsupported operations on FastText Link

• Added MultiLayerNetwork.evaluate(MultiDataSetIterator) and .evaluateRegression(MultiDataSetIterator) methods Link, Link

Bug Fixes and Optimizations

• Updaters (Adam, AdaGrad, etc) optimized via C++ operations (significant training performance boost) for DL4J and SameDiff Link, Link

• Some packages relocated to avoid split packages (that can be a problem for OSGi and Java 9 modules) Link

  • Note: this is a breaking change for some class packages/imports. See this link for details on exact package changes

• Deeplearning4j UI: Webjars versions locked down using dependency management to avoid check on each build Link

• Added MKLDNN (DNNL/OneDNN) support for depthwise_conv2d operation for DL4J and SameDiff Link

• Refactored/merged modules dl4j-perf and dl4j-util into deeplearning4j-core Link

• Fixed an issue with BertWordPieceTokenizer - potential StackOverflowError with certain inputs Link

• Fixed an issue with GlobalPooling layer with masks of different datatype to the activations datatype Link

• Fixed an issue with DL4JModelValidator for ComputationGraph Link

• Fixed an issue where SameDiff layers in DL4J could throw an exception when used with transfer learning Link

• Weight initialization for EmbeddingLayer and EmbeddingSequenceLayer now no longer depend on the vocabulary size (only the vector size) Link

• Fixed an issue with Keras import with bidirectional layers + preprocessors Link

• DL4J UI: added redirect from /train to /train/overview Link

• Fixed an issue where RecordReaderDataSetIterator builder collectMetaData configuration was not being applied Link

• Fixed an issue where MultiLayerNetwork evaluation was not passing metadata to the IEvaluation instances during evaluation Link, Link

• Fixed an issue with Spark training SharedTrainingMaster when training with a ComputationGraph and MultiDataSets Link

• Assorted fixes for edge cases for DL4J Keras import Link

• deelpearning4j-nlp-korean will no longer be released for Scala 2.12 due to required dependency only having Scala 2.11 version avairable Link

• Fix for ConvolutionalIterationListener for ComputationGraph Link

• Fixed an issue where dataset and model zoo downloads could get stuck if the server fails to send any data (now: timeout + retry) Link

• DL4J ModelSerializer no longer writes temporary files when restoring models from InputStream Link

• Fixes issues with UIServer multi session mode, and potential shutdown race condition Link

• Fixed an issue where TfidfVectorizer.vectorize() could throw a NPE when fit from LabelAwareIterator Link

ND4J/SameDiff:

Features and Enhancements

• SameDiff multi-threaded inference enhanced (and fixed) - a single SameDiff instance can now be used for inference safely and efficiently from multiple threads Link Link

• cuDNN support added to SameDiff (automatically enabled for nd4j-cuda-10.x backend) Link

• Added ND4J namespaces: Nd4j.cnn, Nd4j.rnn, Nd4j.image Link

• Added new Image operations namespace operations:

  • rgbToHsv, hsvToRgb Link

  • rgbToYiq, yiqToRgb, rgbToYuv, yuvToRgb Link

  • imageResize Link

• Added new Random operations namespace operations:

  • gamma, poisson, shuffle Link

• Added new Math namespace operations:

  • clipByAvgNorm, embeddingLookup Link

  • mergeMaxIndex Link

• Added new NN namespace operations:

  • cReLU Link

• Added new CNN namespace operations:

  • upsampling3d Link

• Added new linalg operations namespace

  • triangular_solve Link

  • tri operation Link

  • triu operation Link

• Added new RNN operation namespace operations:

  • lstmLayer (note old lstmLayer method renamed to lstmBlock) Link

  • gru Link

• Added new Loss operations namespace - Nd4j.loss Link

• Mapped operations for Tensorflow import:

  • HSVToRGB, RGBToHSV, Igamma, Igammac, RandomGamma, RandomPoisson, RandomPoissonV2, RandomShuffle Link

• Added SameDiff ProfilingListener - writes op performance profiles in Chrome profiler format (load in chrome://tracing/) Link Link

• Added SameDiff ProfileAnalyzer tool to compare profiles output from ProfilingListener (or Tensorflow) Link Link

• SameDiff listener API: added frame and iteration information for listener methods Link Link

• Added (non-backend-specific) method of accessing Nd4j environment: Nd4j.getEnvironment() method (environment info and low-level configuration options) Link Link

• Improved memory limits/configuration support for libnd4j (c++) Link

• Added pairwise (broadcastable) power backprop operation Link

• Updated JavaCPP presets MKL version to 2020.0 from 2019.5 Link

• Added DynamicCustomOp dargs - datatype arguments Link Link

  • Output datatype configuration for Range op Link, SequenceOp Link, ConfusionMatrix Link

• Added tensormmul_bp op Link

• OpenBLAS version upgraded to 0.3.8 Link

• libnd4j (c++ codebase underlying DL4J, ND4J and SameDiff) refactored to be more easily embeddable in other C++ projects Link

• ImagePreProcessingScaler now supports preprocessing of labels (for segmentation) Link

• Additional datatypes now supported for nd4j-tensorflow TensorflowConversion Link

• SameDiff operation namespaces (sd.math, sd.image, etc) are now code generated to ensure SameDiff and ND4J namespaces are identical (all operations included, same API) Link

• Added ND4J ArchiveUtils.unzipFileTo(String, String, boolean logFiles) overload to enable/disable extracted file path logging Link

• Added weight format configuration for following operations: conv1D, conv2D, conv3D, deconv2d, deconv3d, depthwiseConv2d, pointwiseConv2d, sconv2d Link

• Added backprop operation implementations for mergemax, mergeadd, mergeavg operations Link

• MKL version upgraded to 2020.0 2020.1; OpenCV upgraded from 4.2.0 to 4.3.0 Link

• SameDiff: DifferentialFunctionFactory class removed in favor of namespace methods (sd.math, sd.linalg, etc) Link

• Added lstmLayer_bp operation Link

• Added gru_bp operation Link

• linspace operation can now use both targs and arrays for start/end/size arguments Link

• Assorted dependency updates - OpenBLAS (0.3.9), OpenCV (4.3.0), Leptonica (1.79.0) Link

• Upgraded assorted dependency versions: javax.activation:activation (1.1 -> 1.1.1), stream analytics (2.7.0->2.9.8), Apache Spark (2.4.3->2.4.5), Jackson databind (2.10.1 -> 2.10.3), Vertx (3.8.3 -> 3.9.0) Link

• Added nd4j-common-tests ResourceUtils.listClassPathfiles method Link

Bug Fixes and Optimizations

• Updaters (Adam, AdaGrad, etc) optimized via C++ operations (significant training performance boost) for DL4J and SameDiff Link, Link

• SameDiff - added CuDNN support Link

• Some packages relocated to avoid split packages (that can be a problem for OSGi and Java 9 modules) Link

  • Note: this is a breaking change for some class packages/imports. See this link for details on exact package changes

• Fixed some issues with Tensorflow import of FusedBatchNorm operation Link

• Fixed an issue where the Roll operation did not match Tensorflow operation Link Link

• Fixed an issue where ArchiveUtils could fail to create the top level destination directory when it does not exist Link

• Fixed an issue where resize_bicubic operation did not match Tensorflow for some configuration values Link Link

• Pad operation now supports long/int64 values for padding array Link Link

• Fixed an issue where hashcode operation shape function wasn't always returning int64/long dtype Link

• Fixed an issue with reshape operation on empty arrays with -1s Link Link

• Improved performance on CUDA for concat operation Link and CPU/GPU Link

• Improved performance for bias_add operation

  • On CPU for NHWC case Link

  • Generally Link

  • On CUDA for 2D case Link

• Added MKLDNN (DNNL/OneDNN) support for depthwise_conv2d operation for DL4J and SameDiff Link

• Fixed a small SameDiff execution issue for switch operation where the predicate is a constant Link

• Fixed an issue with batchnorm operation when input arrays have unusual strides Link

• Merged nd4j-buffer, nd4j-content modules into nd4j-api Link

• Deleted deprecated nd4j-jackson module (remaining functionality available in nd4j-api) Link

• Deleted unused/unmaintained nd4j-camel and nd4j-gson modules Link

• Optimization for legacy random ops Link

• Optimization for broadcast operations Link, Link, Link, Link, Link

• Performance optimization for multiple operations: softmax, squeeze, expand_dims, tanh Link

• Optimization for transpose/permute operations Link

• Performance enhancement: MKLDNN matmul used for some mmul operation cases Link

• Optimization for gather operation on CPU Link

• Optimization for stack/unstack operations on CPU Link

• Optimization for split operation (CPU and CUDA) Link Link

• ND4J initialization no longer logs number of OpenMP BLAS threads for CUDA Link

• Optimization: Fixed issues with auto-vectorization on multple CPU operations Link

• Optimization for reshape operation Link, Link

• Fixed an issue where INDArray.hashCode() could cause an exception on some datatypes Link

• Optimization for CPU: MKLDNN is now used for softmax, tanh, softmax_bp and tanh_bp operations Link, Link, Link, Link

• Fixed random_exponential operation Link

• Improved performance on C++ SameDiff graph execution via reduced array zeroing where safe to do so Link

• Improved C++ indexing implementation impacting CPU performance on some operations Link

• Fixed an issue where Split operation could have incorrect output shapes for empty arrays Link

• Fixed some issues with SameDiff.equals method Link

• Fixed an issue with reshape operation output shape on empty arrays Link, Link

• Nd4j.gemm now uses Mmul operation internally to avoid potential threading issues with direct BLAS calls on CUDA Link

• Fixed an edge case issue with percentile operation link

• Fixed an edge case issue for cusolved (CUDA) in libnd4j Link

• Fixed an issue with error formatting for segment operations for incorrect lengths Link

• Fixed an issue where ND4J workspaces were not guaranteed to be unique Link

• Fixed some operation implementations when operating on views (Batch/Space to Space/Batch/Depth; batchnorm_bp) Link

• Fixed an issue where exponential distribution random number generation operation could produce infinities extremely rarely (~1 in 10^9 values) Link

• Fixed an issue with long file paths for memory mapped workspaces on Windows Link

• Memory for memory mapped workspaces are now deallocated immediately when workspace is destroyed, instead of waiting for GC to free memory Link

• Fall-back to other BLAS implementation for cases where MKLDNN GEMM implementation is slow Link

• Set nd4j-native source/target to Java 7 Link, Link

DataVec

Features and Enhancements

• datavec-python: added zero-copy support for bytes/byte buffers Link

• datavec-python: Python exceptions are now thrown as Java exceptions Link

• datavec-python: Added support for additional NumPy datatypes Link

• datavec-python: Python version upgraded from 3.7.6 to 3.7.7 Link

Bug Fixes and Optimizations

• Deleted not properly maintained modules: datavec-camel, datavec-perf Link

• Fixed missing BOOL datatype support for arrow conversion functionality Link

• Assorted fixes for datavec-python Link Link, Link

• Fixed an issue with LineRecordReader where initialization was performed unnecessarily (adding performance overhead) Link

RL4J

Features and Enhancements

• Refactoring to decouple configuration and learning methods from their implementations Link

• Added builder patterns for all configuration classes Link

Arbiter

Bug Fixes and Optimizations

• Fixes an issue with GridSearchCandidateGenerator not working correctly for some cases Link, Link

Highlights

In light of the coming 1.0, the project has decided to cut a number of modules before the final release. These modules have not had many users in the past and have created confusion for many users just trying to use a few simple apis. Many of these modules have not been maintained.

There will likely be 1 or 2 more milestone releases before the final 1.0. These should be considered checkpoints.

These modules include:

1. Arbiter

2. Jumpy

3. Datavec modules for video, audio, audio, sound. The computer vision datavec module

   will continue to be available.

4. Tokenizers: The tokenizers for chinese, japanese, korean were imported from other frameworks

   and not really updated.

5. Scalnet, Nd4s: We removed the scala modules due to the small user base. We welcome 3rd party enhancements

   to the framework for syntatic sugar such as kotlin and scala. The framework's focus will be on providing

   the underlying technology rather than the defacto interfaces. If there is interest in something higher level, please discuss it on

ARM support: We have included armcompute modules for core convolution routines. These routines can be found

TVM: We now support running TVM modules. Docs coming soon.

We've updated our shaded modules to newer versions to mitigate security risks. These modules include: 1. jackson 2. guava

Cuda 11: We've upgraded dl4j and associated modules to support cuda 11 and 11.2.

A more modular model import framework supporting tensorflow and onnx: 1. Model mapping procedures loadable as protobuf 2. Defining custom rules for import to work around unsupported or custom layers/operations 3. Op descriptor for all operations in nd4j

This will enable users to override model import behavior to run their own custom models. This means, in most circumstances, there will be no need to modify model import core code anymore. Instead, users will be able to provide definitions and custom rules for their graphs.

Users will be expected to convert their models in an external process. This means running standalone conversions for their models. This extends to keras import as well. Sometimes users convert their models in production directly from keras.

The workflow going forward is to ensure that your model is converted ahead of time to avoid performance issues with converting large models.

Removed ppc from nd4j-native-platform and nd4j-cuda-platform. If you need this architecture, please contact us or build from source.

We've upgraded arrow to 4.0.0 enabling the associated nd4j-arrow and datavec-arrow modules to be used without netty clashes.

Deeplearning4j

Bug fixes

• Improved keras model import support for NWHC as well as NCHW input formats for both rnn and cnn

Nd4j

Features and Enhancements

Bug fixes

Python4j

Features and Enhancements

Rewritten and more stable python execution. This allows better support for multi threaded environments.

Bug fixes

Highlights - 1.0.0-beta4 Release

Main highlight: full multi-datatype support for ND4J and DL4J. In past releases, all N-Dimensional arrays in ND4J were limited to a single datatype (float or double), set globally. Now, arrays of all datatypes may be used simultaneously. The following are supported:

• DOUBLE: double precision floating point, 64-bit (8 byte)

• FLOAT: single precision floating point, 32-bit (4 byte)

• HALF: half precision floating point, 16-bit (2 byte), "FP16"

• LONG: long signed integer, 64 bit (8 byte)

• INT: signed integer, 32 bit (4 byte)

• SHORT: signed short integer, 16 bit (2 byte)

• UBYTE: unsigned byte, 8 bit (1 byte), 0 to 255

• BYTE: signed byte, 8 bit (1 byte), -128 to 127

• BOOL: boolean type, (0/1, true/false). Uses ubyte storage for easier op parallelization

• UTF8: String array type, UTF8 format

ND4J Behaviour changes of note:

• When creating an INDArray from a Java primitive array, the INDArray datatype will be determined by the primitive array type (unless a datatype is specified)

  • For example: Nd4j.createFromArray(double[]) -> DOUBLE datatype INDArray

  • Similarly, Nd4j.scalar(1), Nd4j.scalar(1L), Nd4j.scalar(1.0) and Nd4j.scalar(1.0f) will produce INT, LONG, DOUBLE and FLOAT type scalar INDArrays respectively

• Some operations require matched datatypes for operands

  • For example, if x and y are different datatypes, a cast may be required: x.add(y.castTo(x.dataType()))

• Some operations have datatype restrictions: for example, sum on a UTF8 array is not supported, nor is variance on a BOOL array. For some operations on boolean arrays (such as sum), casting to an integer or floating point type first may make sense.

DL4J Behaviour changes of note:

• MultiLayerNetwork/ComputationGraph no longer depend in any way on ND4J global datatype.

  • The datatype of a network (DataType for it's parameters and activations) can be set during construction using NeuralNetConfigutation.Builder().dataType(DataType)

  • Networks can be converted from one type to another (double to float, float to half etc) using MultiLayerNetwork/ComputationGraph.convertDataType(DataType) method

Main new methods:

• Nd4j.create(), zeros(), ones(), linspace(), etc methods with DataType argument

• INDArray.castTo(DataType) method - to convert INDArrays from one datatype to another

• New Nd4j.createFromArray(...) methods for

ND4J/DL4J: CUDA - 10.1 support added, CUDA 9.0 support dropped

CUDA versions supported in 1.0.0-beta4: CUDA 9.2, 10.0, 10.1.

ND4J: Mac/OSX CUDA support dropped

Mac (OSX) CUDA binaries are no longer provided. Linux (x86_64, ppc64le) and Windows (x86_64) CUDA support remains. OSX CPU support (x86_64) is still available.

DL4J/ND4J: MKL-DNN Support Added DL4J (and ND4J conv2d etc ops) now support MKL-DNN by default when running on CPU/native backend. MKL-DNN support is implemented for the following layer types:

• ConvolutionLayer and Convolution1DLayer (and Conv2D/Conv2DDerivative ND4J ops)

• SubsamplingLayer and Subsampling1DLayer (and MaxPooling2D/AvgPooling2D/Pooling2DDerivative ND4J ops)

• BatchNormalization layer (and BatchNorm ND4J op)

• LocalResponseNormalization layer (and LocalResponseNormalization ND4J op)

• Convolution3D layer (and Conv3D/Conv3DDerivative ND4J ops)

MKL-DNN support for other layer types (such as LSTM) will be added in a future release.

MKL-DNN can be disabled globally (ND4J and DL4J) using Nd4jCpu.Environment.getInstance().setUseMKLDNN(false);

MKL-DNN can be disabled globally for specific ops by setting ND4J_MKL_FALLBACK environment variable to the name of the operations to have MKL-DNN support disabled for. For example: ND4J_MKL_FALLBACK=conv2d,conv2d_bp

ND4J: Improved Performance due to Memory Management Changes

Prior releases of ND4J used periodic garbage collection (GC) to release memory that was not allocated in a memory workspace. (Note that DL4J uses workspaces for almost all operations by default hence periodic GC could frequently be disabled when training DL4J networks). However, the reliance on garbage collection resulted in a performance overhead that scaled with the number of objects in the JVM heap.

In 1.0.0-beta4, the periodic garbage collection is disabled by default; instead, GC will be called only when it is required to reclaim memory from arrays that are allocated outside of workspaces.

To re-enable periodic GC (as per the default in beta3) and set the GC frequency to every 5 seconds (5000ms) you can use:

ND4J: Improved Rank 0/1 Array Support

In prior versions of ND4J, scalars and vectors would sometimes be rank 2 instead of rank 0/1 when getting rows/columns, getting sub-arrays using INDArray.get(NDArrayIndex...) or when creating arrays from Java arrays/scalars. Now, behaviour should be more consistent for these rank 0/1 cases. Note to maintain old behaviour for getRow and getColumn (i.e., return rank 2 array with shape [1,x] and [x,1] respectively), the getRow(long,boolean) and getColumn(long,boolean) methods can be used.

DL4J: Attention layers added

Deeplearning4J

Deeplearning4J: Features and Enhancements

Deeplearning4J: Bug Fixes and Optimizations

ND4J and SameDiff

ND4J/SameDiff: Features and Enhancements

• Added basic ("technology preview") of SameDiff UI. Should be considered early WIP with breaking API changes expected in future releases. Supports plotting of SameDiff graphs as well as various metrics (line charts, histograms, etc)

  • Currenty embedding in the DL4J UI - call UIServer.getInstance() then go to localhost:9000/samediff to access.
• ND4J/SameDiff - new operations added:
• SameDiff TensorFlow Import

ND4J/SameDiff: API Changes (Transition Guide): 1.0.0-beta3 to 1.0.0-beta4

• ND4J datatypes - significant changes, see highlights at top of this section

ND4J/SameDiff: Bug Fixes and Optimizations

• SameDiff: Numerous fixes and enhancements

ND4J: Known Issues

• Most CustomOperation operations (such as those used in SameDiff) are CPU only until next release. GPU support was not completed in time for 1.0.0-beta4 release.

DataVec

DataVec: Features and Enhancements

DataVec: Optimizations and Bug Fixes

Arbiter

Arbiter: Enhancements

Arbiter: Fixes

Highlights

A number of bug fixes following the M1 release, thanks to the feedback from the community, allowed us to quickly sort out a few issues. This is a minor bug fix release to address short comings found with M1. Most fixes were related to keras import, the cnn/rnn helpers, and python4j.

Snapshots will also be published every 2 days automatically now to get around sonatype ossrh deletion of snapshots every 3 days. This should increase robustness of the snapshots.

Worked around an issue with github actions pre emptively upgrading visual studio breaking the cuda builds:

Added backwards compatibility for centos 6 via a new linux-x86_64-compat classifier enabling use of older glibcs on centos 7:

A number of bugs were fixed with LSTM and CUDNN:

Known issues

- avoid shuffle operations on gpu. Pre save data on cpu in mini batches. For more help, please post on the forums at https://community.konduit.ai/

Deeplearning4j

Features and Enhancements

• Add batch normalization support for RNNs:

• Disable old helpers by default

• Minor unit test fixes:

• Add keras support for cnn 1d NWHC:

• Move the warning about version check to tracing so it stops logging this during normal usage confusing users:

• Allow 1d convolutions to accept feed forward as input type:

• Remove the old benchmark suite and migrate it to contrib:

• Remove old MKLDNNLSTM helper (it never fully functioned anyways):

Bug fixes

Nd4j

Features and Enhancements

Bug fixes

Datavec

Features and Enhancements

Bug fixes

Python4j

Features and Enhancements

Bug fixes

Samediff

Features and Enhancements

Bug fixes

Highlights - 1.0.0-beta5 Release

• Added model server - remote inference of SameDiff and DL4J models using JSON or (optionally) binary serialization

  • Server: See JsonModelServer

  • Client: See JsonRemoteInference

  • Tests/examples: See Link and Link

• Added Scala 2.12 support, dropped Scala 2.10 support. Modules with Scala dependencies are now released with Scala 2.11 and 2.12 versions

• Apache Spark 1.x support dropped (now only Spark 2.x is supported). Note: Spark version suffix dropped: For upgrading: 1.0.0-beta4_spark2 -> 1.0.0-beta5

• Added FastText support to deeplearning4j-nlp

• CUDA support for all ND4J/SameDiff Operations

  • In 1.0.0-beta4, some operations were CPU only. Now, all operations have full CUDA support

• Added support for new data types in ND4J (and DL4J/SameDiff): BFLOAT16, UINT16, UINT32, UINT64

• ND4J: Implicit broadcasting support added to INDArray (already present in SameDiff - for example shape [3,1]+[3,2]=[3,2])

• CUDA 9.2, 10.0 and 10.1-Update2 still supported

  • NOTE: For CUDA 10.1, CUDA 10.1 update 2 is recommended. CUDA 10.1 and 10.1 Update 1 will still run, but rare internal cuBLAS issues may be encountered in heavily multi-threaded code on some systems

• Dependency upgrades: Jackson (2.5.1 to 2.9.9/2.9.9.3), Commons Compress (1.16.1 to 1.18), Play Framework (2.4.8 to 2.7.3), Guava: (20.0 to 28.0-jre, and shaded to avoid dependency clashes)

• CUDA: now host (RAM) buffers are only allocated when required (previously: host buffers were always allocated), in addition to device (GPU) buffer

Deeplearning4J

Deeplearning4J: Features and Enhancements

• Added FastText - inference and training, including OOV (out of vocabulary) support (Link)

• Scala 2.12 support added, Scala 2.10 support dropped (Link)

• Added model server (DL4J and SameDiff models, JSON and binary communication) - JsonModelServer, JsonRemoteInference, Link, Link

• Added saved model format validation utilities - DL4JModelValidator, DL4JKerasModelValidator (Link)

• Added LabelLastTimeStepPreProcessor (Link)

• BertIterator: added option to prepend token to the output (such as [cls] expected by some models) (Link)

• Added trace level logging to MultiLayerNetwork and ComputationGraph assist with debugging certain issues (Link)

• Upsampling3D: Added NDHWC support (Link)

• MergeVertex now supports broadcasting (Link)

• LSTM and Dropout will now fall back on built-in implementations if an exception is encountered from cuDNN (same as Subsampling/ConvolutionLayer) (Link)

• Improved JavaDoc and cleanup up API for WordVectorSerializer (Link, Link)

Deeplearning4J: Bug Fixes and Optimizations

• Updated deeplearning4j-ui theme (Link)

• Fixed an issue with MergeVertex and CNN3D activations (Link)

• Fixed typo in Yolo2OutputLayer builder/configuration method name (Link)

• Improved ComputationGraph builder InputType validation (Link)

• Removed dl4j-spark-ml module until it can be properly maintained (Link)

• Fixed an issue with BertWordPieceTokenizerFactory and bad character encoding (Link)

• Fixed an issue with LearnedSelfAttentionLayer and variable minibatch size (Link, Link)

• Fixed issue with SharedTrainingMaster controller address when set from environment variable (Link)

• Fixed issue with SameDiffOutputLayer initialization under some circumstances (Link)

• https is now used by default for data and zoo model downloads (Link, Link)

• Fixed an issue where UI WebJars dependencies would check for updates on every single build (Link, Link)

• Fixed issue where Upsampling layer memory report could produce an OOM exception (Link)

• Improved UX/validation for RecordReaderDataSetIterator (Link)

• Fixed an issue where EmbeddingSequenceLayer would not check mask array datatype (Link)

• Improved validation when initializing networks with a non rank-2 (shape [1, numParams]) array (Link)

• Fixed a DataType issue for BertIterator (Link)

• Fixed Word2Vec model backward compatibilty (beta3 and earlier models now loadable again) Link

• Fixed issue where some Keras import models could fail with Could not read abnormally long HDF5 attribute (Link)

• Added validation for RnnOutputLayer - feature/label array lengths (Link)

• Fixed an issue where SameDiffOutputLayer would not support variable minibatch size (Link)

• Fixed DL4J SameDiff layer mask support (Link)

• DL4J UI: Fixed an issue where tab switching did not work when visualizing saved/stored data (Link, Link)

• DL4J UI: Fixed a rare UI threading issue (Link)

• Fixed a Keras import issue with JSON format change (Link)

• Fixed a Keras import issue where updater learning rate schedule could be imported incorrectly (Link)

• Fixed an issue with CnnSentenceDataSetIterator when using UnknownWordHandling.UseUnknownVector (Link, Link)

• Fixes and optimizations to DL4J SameDiff layers (Link)

• MultiLayerNetwork/ComputationGraph will now log the original exception if a second exception occurs during workspace closing, instead of swallowing it (inference/fit operation try/finally blocks) (Link)

• Upgraded dependencies: Jackson (2.5.1 to 2.9.9/2.9.9.3), Commons Compress (1.16.1 to 1.18), Play Framework (2.4.8 to 2.7.3), Guava: (20.0 to 28.0-jre, shaded to avoid dependency clashes) (Link)

• Logging framework can now be configured for DL4J UI (due to Play framework dependency upgrade) (Link)

• Reduced amount of garbage produced by MnistDataFetcher (impacts MNIST and EMNIST DataSetIterators) (Link)

• Activation function backpropagation has been optimized for many activation functions (Link, Link)

Deeplearning4j: Transition Guide, 1.0.0-beta4 to 1.0.0-beta5

• DL4J AsyncDataSetIterator and AsyncMultiDataSetIterator moved to ND4J, use org.nd4j.linalg.dataset.Async(Multi)DataSetIterator instead

• Saved models with custom layers from 1.0.0-alpha and before can no longer be loaded. Workaround: load in 1.0.0-beta4, and re-save the model (Link). Models without custom layers can still be loaded back to 0.5.0

• Apache Spark 1.x support dropped (now only Spark 2.x is supported). Note: Spark version suffix dropped: For upgrading, change versions as follows: 1.0.0-beta4_spark2 -> 1.0.0-beta5

• Scala 2.10 dropped, Scala 2.12 added (for modules with Scala dependencies)

Deeplearning4j: 1.0.0-beta5 Known Issues

• dl4j-spark_2.11 and _2.12 dependencies incorrectly pull in datavec-spark_2.11/2.12 version 1.0.0-SNAPSHOT. Workaround: control version using dependency management as per here or here

• Some layers (such as LSTM) may run slower on 1.0.0-beta5 than 1.0.0-beta4 on CUDA when not using cuDNN, due to added synchronization. This synchronization will be removed in the next release after 1.0.0-beta5

• CUDA 10.1: Rare internal cuBLAS issues may be encountered in heavily multi-threaded code on some systems, when running CUDA 10.1 Update 1 (and maybe 10.1). CUDA 10.1 update 2 is recommended.

ND4J and SameDiff

ND4J/SameDiff: Features and Enhancements

• Added new data types: BFLOAT16, UINT16, UINT32, UINT64 (Link)

• CUDA support for all operations without CUDA implementations (Link, Link, Link, Link, Link)

• Added model server (DL4J and SameDiff models, JSON and binary communication) - JsonModelServer, JsonRemoteInference, Link, Link

• Added support for empty arrays with zeros in shape, for compatibility with TensorFlow import (Link)

• CUDA: now host (RAM) buffers are only allocated when required (previously: host buffers were always allocated), in addition to device (GPU) buffer

• Improved SameDiff training API - added "in line" test set evaluation, returning History object with loss curve, etc (Link)

• Added saved model format validation utilities - Nd4jValidator, Nd4jCommonValidator (Link)

• Added SameDiff ScoreListener (equivalent to DL4J ScoreIterationListener/PerformanceListener) (Link, Link)

• Added SameDiff.convertDataTypes method, for variable dtype conversion (Link)

• Added crop and resize op (Link)

• DL4J AsyncDataSetIterator and AsyncMultiDataSetIterator moved to ND4J Link

• Added basic/MVP SameDiff UI listener (Link)

• Added SameDiff CheckpointListener (Link, Link)

• Added SameDiff name scopes (Link)

• SameDiff: Updater state and training configuration is now written to FlatBuffers format (Link)

• Added c++ benchmark suite callable from Java - call using Nd4j.getExecutioner().runLightBenchmarkSuit() and Nd4j.getExecutioner().runFullBenchmarkSuit() (Link)

• Added SameDiff.save/load methods with InputStream/OutputStream arguments (Link, Link)

• Added axis configuraiton for evaluation instances (Evaluation, RegressionEvaluation, ROC, etc - getAxis and setAxis methods) to allow different data formats (NCHW vs. NHWC for CNNs, for example) (Link)

• SameDiff: Added support to convert constants to placeholders, via SDVariable.convertToConstant() method (Link)

• SameDiff: Added GradCheckUtil.checkActivationGradients method to check activation gradients for SameDiff instance (not just parameter gradients as in existing gradient check methods) (Link)

• Added CheckNumerics op (Link)

• Added FakeQuantWithMinMaxArgs and FakeQuantWithMinMaxVars ops (Link)

• Added INDArray reduction methods with "keep dimensions" option - for example, INDArray.mean(boloean, int... dimension) (Link)

• Added Nd4j SystemInfo class - SystemInfo.getSystemInfo, .writeSystemInfo(File) to aid with debugging issues (Link, Link)

• Added INDArray.toString(NDArrayStrings options), toStringFull() and toString overloads for easier control of array printing (Link)

• Added HashCode op, INDArray.hashCode() (Link)

• SameDiff: added whileLoop, ifCond methods for loops/conditional ops (Link)

• Cleaned up some infrequently used Nd4j methods (Link, Link, Link, Link)

• Added bitwise integer operations: left/right bit shift, left/right cyclical bit shift, bitwise Hamming distance (Link, Link, Link, Link, Link)

• deeplearning4j-nlp: renamed AggregatingSentencePreProcessor to sentencePreProcessor method (Link)

• Upgraded (and shaded) Protobuf version - 3.5.1 to 3.8.0 (Link)

• Switched to c=style error handling for libnd4j native operations (Link)

• Renamed FlatBuffers enum org.nd4j.graph.DataType to org.nd4j.graph.DType to avoid users importing incorrect type when using Nd4j methods (Link, Link)

• Added SameDiff.bitwise namespace for bitwise ops (Link, Link)

ND4J/SameDiff: Bug Fixes and Optimizations

• Updated to JavaCPP/JavaCV 1.5.1-1 (Link)

• SameDiff: Placeholders must now only be provided if required to calculate the requested variables (Link)

• SameDiff: Fixed an issue with duplicate variable name validation (Link)

• SameDiff: Fixed an issue with SDVariable.getArr for scalars (Link)

• Added delayed mode to DeviceLocalNDArray (don't replicate to device until needed) (Link)

• ND4J: Fixed an issue with writing 0d (scalar) NDArrays in numpy .npy format (Link)

• Fixed an issue with Pad operation for some constant cases (Link)

• Fixed some issues with strided_slice operation (Link, Link, Link)

• SameDiff: Fixed issue with DataType inference for some ops using ND4J default datatype (Link)

• INDArray.castTo(DataType) is now a no-op when array is already the correct type (Link)

• SameDiff: Fixed an issue with training mixed precision networks (Link)

• Fixed an issue where Evaluation class was incorrectly reporting macro-averaged precision for binary case (Link)

• Removed trainableParams config/field from SameDiff TrainingConfig (no longer required) (Link)

• Improvements and cleanup to ND4J Javadoc (Link, Link, Link, Link)

• Fixed an issue with Cholesky Lapack op on CUDA (Link, Link)

• Fixed an issue where [1,N] and [N,1] arrays were not considered a matrix (rank 2 array) according to INDArray.isMatrix() (Link)

• Fixed RegressionEvaluation for 4D arrays (CNNs / segmentation) (Link, Link)

• Fixed issue with INDArray.median(int... dimension) (Link)

• Fixed NPE that could occur when executing gather operation backprop (Link)

• Fixed issue with LogSumExp operation Java/C++ mapping (Link)

• Added header validation when reading Numpy .npy files, to ensure file is valid (Link)

• Fixed a possible issue with reading Numpy .npy files on CUDA (Link)

• Fixed an issue when reading Numpy .npy boolean files (Link)

• Various fixes for TensorFlow import (Link)

• Fixed an issue with a small number of Nd4j.create methods not creating arrays corresponding to the java primitive (Link)

• Improved shape validation for some Nd4j.create methods (Link)

• Cleaned up unmaintained Nd4j.createSparse methods (Link)

• Fixed a CUDA issue for CUDA GPUs with CC 3.0 (Link)

• Fixed some possible integer overflows in c++ code (Link)

• Removed deprecated methods: Nd4j.trueScalar and Nd4j.trueVector (Link, Link)

• Fixed an issue where some JVMs could warn about "Illegal reflective access" due to a (now removed) SameDiff dependency (Link)

• SDVariable now no longer extends DifferentialFunction (Link)

• Moved numerous operation calculateOutputShape instances from Java to C++ (Link)

• Fixed an issue where maxpool2d_bp could throw an exception when NaN values are present (Link)

• Fixed an issue with concatenation of empty shapes (with zeros) (Link)

• Removed INDArray.javaTensorAlongDimension (Link)

• LayerNorm operation now properly supports axis arg, NCHW format data (Link)

• libnd4j: cuBLAS hgemm (FP16 gemm) wil only be called for devices with compute capability >= 5.3 due to cuBLAS limitations (Link)

• Nd4j.readNumpy optimized (Link)

• Added configurable alpha parameter to ELU and lrelu_bp operations in c++ (Link)

• Cleaned up SameDiff SDCNN/SDRNN (SameDiff.cnn, .rnn) API/methods (Link, Link)

ND4J: Transition Guide, 1.0.0-beta4 to 1.0.0-beta5

• OldAddOp, OldSubOp, etc removed: Replace with AddOp, SubOp, etc

• Nd4j.trueScalar and trueVector removed; use Nd4j.scalar and Nd4j.createFromArray methods

• INDArray.javaTensorAlongDimension removed; use INDArray.tensorAlongDimension instead

• INDArray.lengthLong() removed; use INDArray.length() instead

ND4J: 1.0.0-beta5 Known Issues

• nd4j-native on some OSX systems can fail with Symbol not found: ___emutls_get_address - See this link

• SBT 1.3.0 can fail with an Illegal character in path error; SBT 1.2.8 is OK. This is an SBT issue, not an ND4J issue. See this link for details

DataVec

DataVec: Features and Enhancements

• ImageRecordReader: Support for 16-bit TIFF added (Link)

• Added SequenceTrimToLengthTransform (Link)

DataVec: Bug Fixes and Optimizations

• Fixed an issue with AnalyzeSpark and String columns (Link)

• Fixed an issue with URL scheme detection in NumberedFileInputScheme (Link)

• Fixed an issue with RandomPathFilter sampling being biased (Link, Link)

RL4J

RL4J: Features and Enhancements

• API cleanup and refactoring (Link, Link, Link, Link)

RL4J: Bug Fixes and Optimizations

• Fixed issue with compression for HistoryProcessor (Link)

Arbiter

Bug Fixes and Optimizations

• Updated EvaluationScoreFunction to use ND4J Evaluation class metrics (Link)

• Fixed incorrect search size in GridSearchCandidateGenerator (Link)

Arbiter: Known Issues

• The Jackson version upgrade necessitated a change to how generic object serialization was performed; Arbiter JSON data stored in 1.0.0-beta4 or earlier format may not be readable in 1.0.0-beta5 (Link)

ND4S

ND4S Features and Enhancements

• Added full data type support to ND4S as per ND4J (Link)

• Added syntactic sugar for SameDiff (implicits, operator overloads) (Link)

Highlights - 1.0.0-beta3 Release

• ND4J/Deeplearning4j: Added support for CUDA 10.0. Dropped support for CUDA 8.0. (1.0.0-beta3 release has CUDA 9.0, 9.2 and 10.0 support)

• SameDiff now supports training and evaluation from DataSetIterator and MultiDataSetIterator. Evaluation classes have been moved to ND4J.

• DL4J Spark training (gradient sharing) is now fully fault tolerant, and has improvements for threshold adaption (potentially more robust convergence). Ports can now be easily configured independently on master/workers.

Deeplearning4J

Deeplearning4J: New Features

• Added OutputAdapter interface and MultiLayerNetwork/ComputationGraph.output method overloads using OutputAdapter (avoids allocating off-heap memory that needs to be cleaned up by GC) Link, Link, Link

• Added ComputationGraph/MultiLayerNetwork rnnTimeStep overload with user-specified workspace. Link

• Added Cnn3DLossLayer Link

• ParallelInference: Instances can now update the model in real-time (without re-init) Link

• ParallelInferenc: Added ParallelInference INPLACE mode Link

• Added validation for incompatible loss/activation function combinations (such as softmax+nOut=1, or sigmoid+mcxent). New validation can be disabled using outputValidation(false) Link

• Spark training: Added full fault tolerance (robust failure recovery) for gradient sharing implementation Link Link

• Spark training now supports configuring ports more flexibly (and differently for different workers) using PortSupplier Link Link

• Spark training: overhauled gradient sharing threshold adaption algorithms; made it possible to customize threshold settings, plus made defaults more robust to initial threshold configuration improving convergence speed in some cases. Link

• Spark training: implemented chunked messaging to reduce memory requirements (and insufficient buffer length issues) for large messages Link

• Spark training: Added MeshBuildMode configuration for improved scalability for large clusters Link

• Spark network data pipelines: added FileBatch, FileBatchRecordReader etc for "small files" (images etc) distributed training use cases Link

• Added FailureTestingListener for fault tolerance/debugging purposes Link

• Upgraded Apache Lucene/Solr to version 7.5.0 (from 7.4.0) Link

• Added system properties (org.deeplearning4j.tempdir and org.nd4j.tempdir) to allow overriding of the temporary directories ND4J and DL4J use Link Link

• Mode MultiLayerNetwork/ComputationGraph.clearLayerStates methods public (was protected) Link

• AbstactLayer.layerConf() method is now public Link

• ParallelWrapper module now no longer has a Scala version suffix for artifact id; new artifact id is deeplearning4j-parallel-wrapper Link

• Improved validation and error mesages for invalid inputs/labels in Yolo2OutputLayer Link

• Spark training: added SharedTrainingMaster.Builder.workerTogglePeriodicGC and .workerPeriodicGCFrequency to easily configure the ND4J garbage collection configuration on workers. Set default GC to 5 seconds on workers Link

• Spark training: added threshold encoding debug mode (logs current threshold and encoding statistics on each worker during training). Enable using SharedTrainingConfiguration.builder.encodingDebugMode(true). Note this operation has computational overhead. Link

Deeplearning4J: Bug Fixes and Optimizations

• Fixed an issue where L1/L2 and updaters (Adam, Nesterov, etc) were applied before dividing gradients by minibatch to obtain average gradient. To maintain old behaviour, use NeuralNetConfiguration.Builder.legacyBatchScaledL2(true) Link.

  • Note that learning rates may need to be decreased for some updaters (such as Adam) to account for this change vs. earlier versions. Some other updaters (such as SGD, NoOp, etc) should be unaffected.

  • Note that deserialized (loaded) configurations/networks saved in 1.0.0-beta2 or earlier will default to old behaviour for backward compatibility. All new networks (created in 1.0.0-beta3) will default to the new behaviour.

• Fixed an issue where EarlyStoppingScoreCalculator would not correctly handle "maximize score" cases instead of minimizing Link

• Fixed order (BGR vs. RGB) for VGG16ImagePreProcessor channel offset values Link

• Fixed bug with variational autoencoders using weight noise Link

• Fixed issue with BaseDataSetIterator not respecting the 'maximum examples' configuration Link

• Optimization: A workspace is now used for ComputationGraph/MultiLayerNetwork evaluation methods (avoids allocating off-heap memory during evaluation that must be cleaned up by garbage collector) Link

• Fixed an issue where shuffling combined with a subset for MnistDataSetIterator would not maintain the same subset between resets Link

• Fixed issue with StackVertex.getOutputType Link

• Fix issue with CNN to/from RNN preprocessors handling of mask arrays Link

• Fixed issue with VGG16 non-pretrained configuration in model zoo Link

• Fixed issue with TransferLearning nOutReplace where multiple layers in a row are modified Link

• Fixed issue with CuDNN workspaces where backpropagation is performed outside of a standard fit call Link

• Fixed an issue with dropout masks being cleared prematurely on output layers in ComputationGraph Link

• RecordReaderMultiDataSetIterator now supports 5D arrays (for 3D CNNs) Link

• Fixed bug in multi input/output ComputationGraphs with TBPTT combined with both masking and different number of input/output arrays Link

• Improved input validation/exceptions for batch normalization layer Link

• Fixed bug with TransferLearning GraphBuilder nOutReplace when combined with subsampling layers Link

• SimpleRnnParamInitializer now properly respects bias initialization configuration Link

• Fixed SqueezeNet zoo model non-pretrained configuration Link

• Fixed Xception zoo model non-pretrained configuration Link

• Fixed an issue with some evaluation signatures for multi-output ComputationGraphs Link

• Improved MultiLayerNetwork/ComputationGraph summary method formatting for large nets Link

• Fixed an issue where gradient normalization could result in NaNs if gradient is exactly 0.0 for all parameters in a layer Link

• Fixed an issue where MultiLayerNetwork/ComputationGraph.setLearningRate could throw an exception for SGD and NoOp updaters Link

• Fixed an issue with StackVertex plus masking in some rare cases Link

• Fixed an issue with JSON deserialization of frozen layers in pre-1.0.0-alpha format Link

• Fixed an issue where GraphBuilder.removeVertex can fail under some limited circumstances Link

• Fixed a bug in CacheableExtractableDataSetFetcher Link

• DL4J Spark training: Fixed issues with thread/device affinity for multi-GPU training + evaluation Link

• DL4J Spark training: Made all Aeron threads daemon threads to prevent Aeron from stopping JVM shutdown when all other threads have completed Link

• Added cudnnAllowFallback configuration for BatchNormalization layer (fallback to built-in implementation if CuDNN fails unexpectedly) Link

• Fixed some rare concurrency issues with multi-worker (multi-GPU) nodes for Spark training Link Link

• Fixed an issue with BatchNormalization layers that prevented the mean/variance estimates from being synced properly on each worker for GradientSharing training, causing convergence issues Link

• Added a check to detect ZipSlip CVE attempts in ArchiveUtils Link

• DL4J Spark training and evaluation: methods now use Hadoop Configuration from Spark context to ensure runtime-set configuration is available in Spark functions reading directly from remote storage (HDFS etc) Link

• MultiLayerNetwork and ComputationGraph now properly support more than Integer.MAX_VALUE parameters Link Link

• Added data validation for Nd4j.readTxt - now throws exception on invalid input instead of returning incorrect values Link

• Fixed an issue with KNN implementation where a deadlock could occur if an invalid distance function (one returning "distances" less than 0) was utilized Link

• Added synchronization to loading of Keras import models to avoid thread safety issues in the underlying HDFS library used for loading Link

• Fixed rare issue for Async(Multi)DataSetIterator with large prefetch values Link

Deeplearning4J: API Changes (Transition Guide): 1.0.0-beta2 to 1.0.0-beta3

• IEvaluation classes in DL4J have been deprecated and moved to ND4J so they are available for SameDiff training. Functionality and APIs are unchanged

• MultiLayerConfiguration/ComputationGraphConfiguration pretrain(boolean) and backprop(boolean) have been deprecated and are no longer used. Use fit and pretrain/pretrainLayer methods instead. Link

• ParallelWrapper module now no longer has a Scala version suffix for artifact id; new artifact id is deeplearning4j-parallel-wrapper which should be used instead Link

• deeplearning4j-nlp-korean module now has Scala version suffix due to scala dependencies; new artifact ID is deeplearning4j-nlp-korean_2.10 and deeplearning4j-nlp-korean_2.11 Link

Deeplearning4J: Known issues: 1.0.0-beta3

• Running multiple Spark training jobs simultaneously on the one physical node (i.e., multiple JVMs from one or more Spark jobs) may cause problems with network communication. A workaround for this is to manually set a unique stream ID manually in the VoidConfiguration. Use a unique (or random) integer value for different jobs Link

Deeplearning4J: Keras Import

• Fixed import issue due to Keras JSON format changes for Keras 2.2.3+ Link

• Added Keras import for timeseries preprocessing Link

• Elephas Link

• Fixed issue with importing models with reshaping after an embedding layer Link

• Added support for Keras masking layers Link

• Fixed JSON deserialization issue with some layers/preprocessors, such as Permute Link

• Fixed issue with Keras import of Nadam configuration Link

ND4J

ND4J: New Features

• Added SameDiff training and evaluation: SameDiff instances can now be trained directly using DataSetIterator and MultiDataSetIterator, and evaluated using IEvaluation instances (that have been moved from ND4J to DL4J) Link

• Added GraphServer implementation: c++ inference server for SameDiff (and Tensorflow, via TF import) with Java API Link

• SameDiff instances can now be loaded from serialized FlatBuffers format (SameDiff.asFlatFile plus fromFlatFile) Link Link

• Added MKL-DNN support for some operations (Conv2d, etc) Link

• Upgraded ND4J (and DataVec) to Arrow 0.11.0 Link, which also fixes Link

• Added Nd4j.where op method (same semantics as numpy.where) Link

• Added Nd4j.stack op method (combine arrays + increase array rank by 1) Link

• Libnd4j new ops:

  • Matrix band part Link

  • Scatter ND, ND-add, ND-sub and ND-update ops Link

  • Sparse softmax cross entropy loss with logits Link

  • Histogram fixed width op Link

  • broadcast_to op Link

  • deconv3d op added Link

  • Unsorted segment ops added Link

  • Segment_X backprop ops added Link

  • batchnorm_new op added that supports multiple axes for mean/variance Link

  • GRU cell backprop added Link

• Nd4j Preconditions class now has methods for formatting INDArray arguments Link, Link

• SameDiff loss functions: cleanup plus forward pass implementation Link

• CudaGridExecutioner now warns that exception stack traces may be delayed to avoid confusion in debugging exceptions occuring during asynchronous execution of ops Link

• JavaCPP and JavaCPP-presets have been upgraded to version 1.4.3 Link

• Improved Javadoc on SDVariable class Link

ND4J: Bug Fixes and Optimizations

• Fixes for android: Remove use of RawIndexer Link

• Libnd4j custom ops: conv op weight layouts are now not dependent on the input format (NCHW/NHWC) - now always [kH, kW, inChannels, outChannels] for 2d CNNs, [kH, kW, kD, inChannels, outChannels] for 3d CNNs. Link, Link

• Libnd4j native op fixes:

  • Dot operation backprop Link, determinant Link

  • Backprop op fix for the broadcast case for some pairwise transform custom op implementations Link

  • Fix for reverse custom op with rank 1 inputs Link

  • ATan2 op is now broadcastable Link

  • Boolean custom op broadcast fixes/additions Link

  • Scatter op edge case fixes Link

  • ArgMin shape function fix Link, negative axis fix Link

  • Unique op fix Link

  • Pad op fix Link

  • Fixed where op shape function Link

  • SVD rank 1 edge case fix Link

  • Range op Link

  • Split and space_to_batch fixes Link

  • Broadcast dynamic shape Link

  • embedding_lookup op now supports multiple input arrays Link

  • Matrix determinant op edge case (rank 0 result) shape fix Link

• SameDiff TensorFlow import: fixes for multiple operations Link, Link, Link, Link

• SameDiff: Improved error handling for multiple outputs case Link

• Fixed issue where INDArray.permute would not correctly throw an exception for invalid length case Link

• Fixed issues with INDArray.get/put with SpecifiedIndex Link, Link

• Minor change to DataSet.merge - signature now accepts any DataSet subtypes Link

• INDArray.transposei operation was not in-place Link

• Fixed issues with INDArray.mmul with MMulTranspose Link

• Added additional order validation for ND4J creation methods (create, rand, etc) Link

• Fix for ND4J binary deserialization (BinarySerde) when deserializing from heap byte buffers Link

• Fixed issue with Nd4j-common ClassPathResource path resolution in some IDEs Link

• Fixed issue where INDArray.get(interval) on rank 1 array would return rank 2 array Link

• Fixed a validation issue with Nd4j.gemm/mmuli on views Link Link

• INDArray.assign(INDArray) no longer allows assigning different shape arrays (other than scalar/vector cases) Link

• NDarrayStrings (and INDArray.toString()) now always uses US locale when formatting numbers Link

• Fixed an issue with GaussianDistribution specific to V100 GPUs Link

• Fixed an issue with bitmap compression/encoding specific to V100 GPUs Link

• Transforms.softmax now throws an error on unsupported shapes instead of simply not applying operation Link

• VersionCheck functionality: handle case where SimpleFileVisitor is not available on earlier versions of Android Link

• SameDiff convolution layer configuration (Conv2dConfig/Conv3dConfig/Pooling3dConfig etc) have had parameter names aligned Link

ND4J: API Changes (Transition Guide): 1.0.0-beta2 to 1.0.0-beta3

• CUDA 8.0 support has been removed. CUDA 9.0, 9.2 and 10.0 support is available in 1.0.0-beta3

• nd4j-base64 module contents have been deprecated; use the equivalent classes in nd4j-api from now on Link

• Some classes in nd4j-jackson module has been deprecated; use the equivalent classes in nd4j-api from now on Link

ND4J: Known issues: 1.0.0-beta3

• Android users may need to manually exclude the (now deprecated) module nd4j-base64. This is due to org.nd4j.serde.base64.Nd4jBase64 class being present in both nd4j-api and nd4j-base64 modules. Both versions have identical content. Use exclude group: 'org.nd4j', module: 'nd4j-base64' to exclude.

DataVec

DataVec: New Features

• Added NativeImageLoader method overloads for org.opencv.core.Mat and String as filename Link

DataVec: Optimizations and Bug Fixes

• Fix for JDBCRecordReader handling of null values Link

• Improved errors/validation for ObjectDetectionRecordReader for invalid input (where image object centers are outside of image bounds) Link

• Fixed issue where FileSplit using methods that are unavailable on earlier versions of Android Link

• Added SerializableHadoopConfiguration and BroadcastHadoopConfigHolder for cases where a Hadoop configuration is required in Spark functions Link Link

• Fixed issue with JDBCRecordReader's handling of real-valued column result types Link

• Added validation and useful exception for CSVRecordReader/LineRecordReader being used without initialization Link

Arbiter

Arbiter: Fixes

• Fixed some issues with dropout layers Link

ND4S

• Added conversion between org.nd4j.linalg.primitives.Pair/Triple and Scala Tuple Link

Deeplearning4J

• Fixed issue with incorrect version dependencies in 0.9.0

• Added EmnistDataSetIterator Link

• Numerical stability improvements to LossMCXENT / LossNegativeLogLikelihood with softmax (should reduce NaNs with very large activations)

ND4J

• Added runtime version checking for ND4J, DL4J, RL4J, Arbiter, DataVec Link

Known Issues

• Deeplearning4j: Use of Evaluation class no-arg constructor (i.e., new Evaluation()) can result in accuracy/stats being reported as 0.0. Other Evaluation class constructors, and ComputationGraph/MultiLayerNetwork.evaluate(DataSetIterator) methods work as expected.

  • This also impacts Spark (distributed) evaluation: workaround is to replace sparkNet.evaluate(testData); with sparkNet.doEvaluation(testData, 64, new Evaluation(10))[0];, where 10 is the number of classes and 64 in the evaluation minibatch size to use.

• SequenceRecordReaderDataSetIterator applies preprocessors (such as normalization) twice to each DataSet (possible workaround: use RecordReaderMultiDataSetIterator + MultiDataSetWrapperIterator)

• TransferLearning: ComputationGraph may incorrectly apply l1/l2 regularization (defined in FinetuneConfiguration) to frozen layers. Workaround: set 0.0 l1/l2 on FineTuneConfiguration, and required l1/l2 on new/non-frozen layers directly. Note that MultiLayerNetwork with TransferLearning appears to be unaffected.

Highlights - 1.0.0-beta Release

• Performance and memory optimizations for DL4J

Deeplearning4J

Deeplearning4J: New Features

• New or enhanced layers:

  • Added Cropping1D layer Link

  • Added Convolution3D, Cropping3D, UpSampling3D, ZeroPadding3D, Subsampling3D layers (all with Keras import support): Link Link

  • Added EmbeddingSequenceLayer (EmbeddingLayer for time series) Link

  • Added OCNNOutputLayer (one-class neural network) - implementation of this paper - Link

  • Added FrozenLayerWithBackprop layer Link

  • Added DepthwiseConvolution2D layer Link

• Added ComputationGraph.output(DataSetIterator) method Link

• Added MultiLayerNetwork/ComputationGraph.layerInputSize methods Link Link

• Added SparkComputationGraph.feedForwardWithKey overload with feature mask support Link

• Added MultiLayerNetwork.calculateGradients method (for easily getting parameter and input gradients, for example for some model interpretabilithy approaches) Link Link

• Added support to get input/activation types for each layer from configuration: ComputationGraphConfiguration.getLayerActivationTypes(InputType...), ComputationGraphConfiguration.GraphBuilder.getLayerActivationTypes(), NeuralNetConfiguration.ListBuilder.getLayerActivationTypes(), MultiLayerConfiguration.getLayerActivationTypes(InputType) methods Link

• Evaluation.stats() now prints confusion matrix in easier to read matrix format, rather than list format Link

• Added ModelSerializer.addObjectToFile, .getObjectFromFile and .listObjectsInFile for storing arbitrary Java objects in same file as saved network Link

• Added SpatialDropout support (with Keras import support) Link

• Added MultiLayerNetwork/ComputationGraph.fit((Multi)DataSetIterator, int numEpochs) overloads Link

• Added performance (hardware) listeners: SystemInfoPrintListener and SystemInfoFilePrintListener Link

Deeplearning4J: Bug Fixes and Optimizations

• Performance and memory optimizations via optimizations of internal use of workspaces Link

• Reflections library has entirely been removed from DL4J and is no longer required for custom layer serialization/deserialization Link, Link

  • Fixes issues with custom and some Keras import layers on Android

• RecordReaderMultiDataSetIterator will no longer try to convert unused columns to numerical values Link

• Added new model zoo models:

  • (to do)

• Fixes for Android compilation (removed duplicate classes, aligned versions, removed some dependencies) Link Link Link

• Fix for RecordReaderMulitDataSetIterator where output could be incorrect for some constructors Link

• Non-frozen layers before a frozen layer will no longer be skipped during backprop (useful for GANs and similar architectures) Link Link

• Fixed issue where ComputationGraph topological sort may not be consistent on all platforms; could sometimes break ComputationGraphs (with multiple valid topological orderings) trained on PC and deployed on Android Link

• Fixed issue with CuDNN batch norm using 1-decay instead of decay Link

• deeplearning4j-cuda no longer throws exceptions if present on classpath with nd4j-native backend set to higher priority Link

• Added RNG control for CifarDataSetIterator Link

• WordVectorSerializer now deletes temp files immediately once done Link

Deeplearning4J: API Changes (Transition Guide): 1.0.0-alpha to 1.0.0-beta

• WorkspaceMode.SINGLE and SEPARATE have been deprecated; use WorkspaceMode.ENABLED instead

• Internal layer API changes: custom layers will need to be updated to the new Layer API - see built-in layers or custom layer example

• Custom layers etc in pre-1.0.0-beta JSON (ModelSerializer) format need to be registered before they can be deserialized due to JSON format change. Built-in layers and models saved in 1.0.0-beta or later do not require this. Use NeuralNetConfiguration.registerLegacyCustomClassesForJSON(Class) for this purpose

• IterationListener has been deprecated in favor of TrainingListener. For existing custom listeners, switch from implements TrainingListener to extends BaseTrainingListener Link

• ExistingDataSetIterator has been deprecated; use fit(DataSetIterator, int numEpochs) method instead

Deelpearning4J: 1.0.0-beta Known Issues

• ComputationGraph TrainingListener onEpochStart and onEpochEnd methods are not being called correctly

• DL4J Zoo Model FaceNetNN4Small2 model configuration is incorrect, causing issues during forward pass

• Early stopping score calculators with values thar should be maximized (accuracy, f1 etc) are not working properly (values are minimized not maximized). Workaround: override ScoreCalculator.calculateScore(...) and return 1.0 - super.calculateScore(...).

Deeplearing4J: Keras Import

Deeplearning4J: Keras Import - API Changes (Transition Guide): 1.0.0-alpha to 1.0.0-beta

ND4J

ND4J: New Features

ND4J: Known Issues

• Not all op gradients implemented for automatic differentiation

• Vast majority of new operations added in 1.0.0-beta do NOT use GPU yet.

ND4J: API Changes (Transition Guide): 1.0.0-alpha to 1.0.0-beta

DataVec

DataVec: New Features

• ImageRecordReader now logs number of inferred label classes (to reduce risk of users missing a problem if something is misconfigured) Link

• Added AnalyzeSpark.getUnique overload for multiple columns Link

• Added performance/timing module Link

DataVec: Optimizations and Bug Fixes

• Reduced ImageRecordReader garbage generation via buffer reuse Link

• Fixes for Android compilation (aligned versions, removed some dependencies) Link Link

• Removed Reflections library use in DataVec Link

• Fix for TransformProcessRecordReader batch support Link

• Fix for TransformProcessRecordReader with filter operations Link

• Fixed issue with ImageRecordReader/ParentPathLabelGenerator incorrectly filtering directories containing . character(s) Link

• ShowImageTransform now initializes frame lazily to avoid blank windows Link

DataVec: API Changes (Transition Guide): 1.0.0-alpha to 1.0.0-beta

• DataVec ClassPathResource has been deprecated; use nd4j-common version instead Link

Arbiter

Arbiter: New Features

• Added LayerSpace for OCNN (one-class neural network)

Arbiter: Fixes

• Fixed timestamp issue that could cause incorrect rendering of first model's results in UI Link

• Execution now waits for last model(s) to complete before returning when a termination condition is hit Link

• As per DL4J etc: use of Reflections library has been removed entirely from Arbiter Link

• Remove use of Eclipse Collections library due to issues with Android compilation Link

• Improved cleanup of completed models to reduce maximum memory requirements for training Link

Highlights - 1.0.0-alpha Release

• ND4J: Added SameDiff - Java automatic differentiation library (alpha release) with Tensorflow import (technology preview) and hundreds of new operations

• ND4J: Added CUDA 9.0 and 9.1 support (with cuDNN), dropped support for CUDA 7.5, continued support for CUDA 8.0

• ND4J: Native binaries (nd4j-native on Maven Central) now ship with AVX/AVX2/AVX-512 support (Windows/Linux)

• DL4J: Large number of new layers and API improvements

• DL4J: Keras 2.0 import support

Deeplearning4J

Deeplearning4J: New Features

• Layers (new and enhanced)

  • Added Yolo2OutputLayer CNN layer for object detection (Link). See also DataVec's ObjectDetectionRecordReader

  • Adds support for 'no bias' layers via hasBias(boolean) config (DenseLayer, EmbeddingLayer, OutputLayer, RnnOutputLayer, CenterLossOutputLayer, ConvolutionLayer, Convolution1DLayer). EmbeddingLayer now defaults to no bias (Link)

  • Adds support for dilated convolutions (aka 'atrous' convolutions) - ConvolutionLayer, SubsamplingLayer, and 1D versions there-of. (Link)

  • Added Upsampling2D layer, Upsampling1D layer (Link, Link)

  • ElementWiseVertex now (additionally) supports Average and Max modes in addition to Add/Subtract/Product (Link)

  • Added SeparableConvolution2D layer (Link)

  • Added Deconvolution2D layer (aka transpose convolution, fractionally strided convolution layer) (Link)

  • Added ReverseTimeSeriesVertex (Link)

  • Added RnnLossLayer - no-parameter version of RnnOutputLayer, or RNN equivalent of LossLayer (Link)

  • Added CnnLossLayer - no-parameter CNN output layer for use cases such as segmentation, denoising, etc. (Link)

  • Added Bidirectional layer wrapper (converts any uni-directional RNN to a bidirectional RNN) (Link)

  • Added SimpleRnn layer (aka "vanilla" RNN layer) (Link)

  • Added LastTimeStep wrapper layer (wraps a RNN layer to get last time step, accounting for masking if present) (Link)

  • Added MaskLayer utility layer that simply zeros out activations on forward pass when a mask array is present (Link)

  • Added alpha-version (not yet stable) SameDiff layer support to DL4J (Note: forward pass, CPU only for now)(Link)

  • Added SpaceToDepth and SpaceToBatch layers (Link, Link)

  • Added Cropping2D layer (Link)

• Added parameter constraints API (LayerConstraint interface), and MaxNormConstraint, MinMaxNormConstraint, NonNegativeConstraint, UnitNormConstraint implementations (Link)

• Significant refactoring of learning rate schedules (Link)

  • Added ISchedule interface; added Exponential, Inverse, Map, Poly, Sigmoid and Step schedule implementations (Link)

  • Added support for both iteration-based and epoch-based schedules via ISchedule. Also added support for custom (user defined) schedules

  • Learning rate schedules are configured on the updaters, via the .updater(IUpdater) method

• Added dropout API (IDropout - previously dropout was available but not a class); added Dropout, AlphaDropout (for use with self-normalizing NNs), GaussianDropout (multiplicative), GaussianNoise (additive). Added support for custom dropout types (Link)

• Added support for dropout schedules via ISchedule interface (Link)

• Added weight/parameter noise API (IWeightNoise interface); added DropConnect and WeightNoise (additive/multiplicative Gaussian noise) implementations (Link); dropconnect and dropout can now be used simultaneously

• Adds layer configuration alias .units(int) equivalent to .nOut(int) (Link)

• Adds ComputationGraphConfiguration GraphBuilder .layer(String, Layer, String...) alias for .addLayer(String, Layer, String...)

• Layer index no longer required for MultiLayerConfiguration ListBuilder (i.e., .list().layer(<layer>) can now be used for configs) (Link)

• Added MultiLayerNetwork.summary(InputType) and ComputationGraph.summary(InputType...) methods (shows layer and activation size information) (Link)

• MultiLayerNetwork, ComputationGraph and layerwise trainable layers now track the number of epochs (Link)

• Added deeplearning4j-ui-standalone module: uber-jar for easy launching of UI server (usage: java -jar deeplearning4j-ui-standalone-1.0.0-alpha.jar -p 9124 -r true -f c:/UIStorage.bin)

• Weight initializations:

  • Added .weightInit(Distribution) convenience/overload (previously: required .weightInit(WeightInit.DISTRIBUTION).dist(Distribution)) (Link)

  • WeightInit.NORMAL (for self-normalizing neural networks) (Link)

  • Ones, Identity weight initialization (Link)

  • Added new distributions (LogNormalDistribution, TruncatedNormalDistribution, OrthogonalDistribution, ConstantDistribution) which can be used for weight initialization (Link)

  • RNNs: Added ability to specify weight initialization for recurrent weights separately to "input" weights (Link)

• Added layer alias: Convolution2D (ConvolutionLayer), Pooling1D (Subsampling1DLayer), Pooling2D (SubsamplingLayer) (Link)

• Added Spark IteratorUtils - wraps a RecordReaderMultiDataSetIterator for use in Spark network training (Link)

• CuDNN-supporting layers (ConvolutionLayer, etc) now warn the user if using CUDA without CuDNN (Link)

• Binary cross entropy (LossBinaryXENT) now implements clipping (1e-5 to (1 - 1e-5) by default) to avoid numerical underflow/NaNs (Link)

• SequenceRecordReaderDataSetIterator now supports multi-label regression (Link)

• TransferLearning FineTuneConfiguration now has methods for setting training/inference workspace modes (Link)

• IterationListener iterationDone method now reports both current iteration and epoch count; removed unnecessary invoke/invoked methods (Link)

• Added MultiLayerNetwork.layerSize(int), ComputationGraph.layerSize(int)/layerSize(String) to easily determine size of layers (Link)

• Added MultiLayerNetwork.toComputationGraph() method (Link)

• Added NetworkUtils convenience methods to easily change the learning rate of an already initialized network (Link)

• Added MultiLayerNetwork.save(File)/.load(File) and ComputationGraph.save(File)/.load(File) convenience methods (Link)

• Added CheckpointListener to periodically save a copy of the model during training (every N iter/epochs, every T time units) (Link)

• Added ComputationGraph output method overloads with mask arrays (Link)

• New LossMultiLabel loss function for multi-label classification (Link)

• Added new model zoo models:

  • Darknet19 (Link)

  • TinyYOLO (Link)

• New iterators, and iterator improvements:

  • Added FileDataSetIterator, FileMultiDataSetIterator for flexibly iterating over directories of saved (Multi)DataSet objects (Link)

  • UCISequenceDataSetIterator (Link)

  • RecordReaderDataSetIterator now has builder pattern for convenience, improved javadoc (Link)

  • Added DataSetIteratorSplitter, MultiDataSetIteratorSplitter (Link, Link)

• Added additional score functions for early stopping (ROC metrics, full set of Evaluation/Regression metrics, etc) (Link)

• Added additional ROC and ROCMultiClass evaluation overloads for MultiLayerNetwork and ComputationGraph (Link)

• Clarified Evaluation.stats() output to refer to "Predictions" instead of "Examples" (former is more correct for RNNs) (Link)

• EarlyStoppingConfiguration now supports Supplier<ScoreCalculator> for use with non-serializable score calculators (Link)

• Improved ModelSerializer exceptions when trying to load a model via wrong method (i.e., try to load ComputationGraph via restoreMultiLayerNetwork) (Link)

• Added SparkDataValidation utility methods to validate saved DataSet and MultiDataSet on HDFS or local (Link)

• ModelSerializer: added restoreMultiLayerNetworkAndNormalizer and restoreComputationGraphAndNormalizer methods (Link)

• ParallelInference now has output overloads with support for input mask arrays (Link)

Deeplearning4J: Bug Fixes and Optimizations

• Lombok is no longer included as a transitive dependency (Link)

• ComputationGraph can now have a vertex as the output (not just layers) (Link, Link)

• Performance improvement for J7FileStatsStorage with large amount of history (Link)

• Fixed UI layer sizes for variational autoencoder layers (Link)

• Fixes to avoid HDF5 library crashes (Link, Link)

• UI Play servers switch to production (PROD) mode (Link)

• Related to the above: users can now set play.crypto.secret system property to manually set the Play application secret; is randomly generated by default (Link).

• SequenceRecordReaderDataSetIterator would apply preprocessor twice (Link)

• Evaluation no-arg constructor could cause NaN evaluation metrics when used on Spark

• CollectScoresIterationListener could recurse endlessly (Link)

• Async(Multi)DataSetIterator calling reset() on underlying iterator could cause issues in some situations (Link)

• In some cases, L2 regularization could be (incorrectly) applied to frozen layers (Link)

• Logging fixes for NearestNeighboursServer (Link)

• Memory optimization for BaseStatsListener (Link)

• ModelGuesser fix for loading Keras models from streams (previously would fail) (Link)

• Various fixes for workspaces in MultiLayerNetwork and ComputationGraph (Link, Link, Link, Link, Link, Link)

• Fix for incorrect condition in DuplicateToTimeSeriesVertex (Link)

• Fix for getMemoryReport exception on some valid ComputationGraph networks (Link)

• RecordReaderDataSetIterator when used with preprocessors could cause an exception under some circumstances (Link)

• CnnToFeedForwardPreProcessor could silently reshape invalid input, as long as the input array length matches the expected length (Link)

• ModelSerializer temporary files would not be deleted if JVM crashes; now are deleted immediately when no longer required (Link)

• RecordReaderMultiDataSetIterator may not add mask arrays under some circumstances, when set to ALIGN_END mode (Link)

• ConvolutionIterationListener previously produced an IndexOutOfBoundsException when all convolution layers are frozen (Link)

• PrecisionRecallCurve.getPointAtRecall could return a point with a correct but sub-optimal precision when multiple points had identical recall (Link)

• Setting dropout(0) on transfer learning FineTuneConfiguration did not remove dropout if present on existing layer (Link)

• Under some rare circumstances, Spark evaluation could lead to a NullPointerException (Link)

• ComputationGraph: disconnected vertices were not always detected in configuration validation (Link)

• Activation layers would not always inherit the global activation function configuration (Link)

• RNN evaluation memory optimization: when TBPTT is configured for training, also use TBPTT-style splitting for evaluation (identical result, less memory) (Link, Link)

• PerformanceListener is now serializable (Link)

• ScoreIterationListener and PerformanceListener now report model iteration, not "iterations since listener creation" (Link)

• Precision/recall curves cached values in ROC class may not be updated after merging ROC instances (Link)

• ROC merging after evaluating a large number of examples may produce IllegalStateException (Link)

• Added checks for invalid input indices to EmbeddingLayer (Link)

• Fixed possible NPE when loading legacy (pre-0.9.0) model configurations from JSON (Link)

• Fixed issues with EvaluationCalibration HTML export chart rendering (Link)

• Fixed possible incorrect redering of UI/StatsStorage charts with J7FileStatsStorage when used with Spark training (Link)

• MnistDataSetIterator would not always reliably detect and automatically fix/redownload on corrupted download data (Link)

• MnistDataSetIterator / EmnistDataSetIterator: updated download location after hosting URL change (Link, Link)

• Fixes to propagation of thread interruptions (Link)

• MultiLayerNetwork/ComputationGraph will no longer throw an ND4JIllegalStateException during initialization if a network contains no parameters (Link, Link)

• Fixes for TSNE posting of data to UI for visualization (Link)

• PerformanceListener now throws a useful exception (in constructor) on invalid frequency argument, instead of runtime ArithmeticException (Link)

• RecordReader(Multi)DataSetIterator now throws more useful exceptions when Writable values are non-numerical (Link)

• UI: Fixed possible character encoding issues for non-English languages when internationalization data .txt files are read from uber JARs (Link)

• UI: Fixed UI incorrectly trying to parse non-DL4J UI resources when loading I18N data (Link)

• Various threading fixes (Link)

• Evaluation: no-arg methods (f1(), precion(), etc) now return single class value for binary case instead of macro-averaged value; clarify values in stats() method and javadoc (Link)

• Early stopping training: TrainingListener opEpochStart/End (etc) methods were not being called correctly (Link)

• Fixes issue where dropout was not always applied to input of RNN layers (Link)

• ModelSerializer: improved validation/exceptions when reading from invalid/empty/closed streams (Link)

• ParallelInference fixes:

  • fixes for variable size inputs (variable length time series, variable size CNN inputs) when using batch mode (Link)

  • fixes undelying model exceptions during output method are now properly propagated back to the user (Link)

  • fixes support for 'pre-batched' inputs (i.e., inputs where minibatch size is > 1) (Link)

• Memory optimization for network weight initialization via in-place random ops (Link)

• Fixes for CuDNN with SAME mode padding (Link, Link)

• Fix for VariationalAutoencoder builder decoder layer size validation (Link)

• Improved Kmeans throughputlink

• Add RPForest to nearest neighbors link