Links
Comment on page

Release Notes

New changes in each release of Eclipse Deeplearning4j.

Version 1.0.0-beta7

Read the announcement at https://blog.konduit.ai/2020/05/14/deeplearning4j-1-0-0-beta7-released/ for the highlights of this release.

Deeplearning4j

Features and Enhancements

  • Added Keras model import support for tf.keras models Link, Link
    • Full inference and training support is available for ops/layers in the tf.keras namespace; inference only for general Tensorflow operations outside of the tf.keras namespace
    • Note also improvements to Keras import for reshape, permute, etc operations due to NHWC and NWC support in DL4J
  • DL4J now supports NHWC (channels last) data format for all CNN 2D layers, in addition to NCHW Link
  • DL4J now supports NWC (channels last - [minibatch, sequence_length, size]) for all RNN and CNN 1D layers, in addition to NCW Link
  • Added Deconvolution3D layer Link
  • Keras import: added ReLU, ELU and Softmax advanced activation layers Link and Swish activation function Link
  • Added DL4J SameDiffLoss class (for easily-defined DL4J ILossFunction's via SameDiff) Link
  • Useful exceptions are now thrown when attempting to perform unsupported operations on FastText Link
  • Added MultiLayerNetwork.evaluate(MultiDataSetIterator) and .evaluateRegression(MultiDataSetIterator) methods Link, Link

Bug Fixes and Optimizations

  • Updaters (Adam, AdaGrad, etc) optimized via C++ operations (significant training performance boost) for DL4J and SameDiff Link, Link
  • Some packages relocated to avoid split packages (that can be a problem for OSGi and Java 9 modules) Link
    • Note: this is a breaking change for some class packages/imports. See this link for details on exact package changes
  • Deeplearning4j UI: Webjars versions locked down using dependency management to avoid check on each build Link
  • Added MKLDNN (DNNL/OneDNN) support for depthwise_conv2d operation for DL4J and SameDiff Link
  • Refactored/merged modules dl4j-perf and dl4j-util into deeplearning4j-core Link
  • Fixed an issue with BertWordPieceTokenizer - potential StackOverflowError with certain inputs Link
  • Fixed an issue with GlobalPooling layer with masks of different datatype to the activations datatype Link
  • Fixed an issue with DL4JModelValidator for ComputationGraph Link
  • Fixed an issue where SameDiff layers in DL4J could throw an exception when used with transfer learning Link
  • Weight initialization for EmbeddingLayer and EmbeddingSequenceLayer now no longer depend on the vocabulary size (only the vector size) Link
  • Fixed an issue with Keras import with bidirectional layers + preprocessors Link
  • DL4J UI: added redirect from /train to /train/overview Link
  • Fixed an issue where RecordReaderDataSetIterator builder collectMetaData configuration was not being applied Link
  • Fixed an issue where MultiLayerNetwork evaluation was not passing metadata to the IEvaluation instances during evaluation Link, Link
  • Fixed an issue with Spark training SharedTrainingMaster when training with a ComputationGraph and MultiDataSets Link
  • Assorted fixes for edge cases for DL4J Keras import Link
  • deelpearning4j-nlp-korean will no longer be released for Scala 2.12 due to required dependency only having Scala 2.11 version avairable Link
  • Fix for ConvolutionalIterationListener for ComputationGraph Link
  • Fixed an issue where dataset and model zoo downloads could get stuck if the server fails to send any data (now: timeout + retry) Link
  • DL4J ModelSerializer no longer writes temporary files when restoring models from InputStream Link
  • Fixes issues with UIServer multi session mode, and potential shutdown race condition Link
  • Fixed an issue where TfidfVectorizer.vectorize() could throw a NPE when fit from LabelAwareIterator Link

ND4J/SameDiff:

Features and Enhancements

  • SameDiff multi-threaded inference enhanced (and fixed) - a single SameDiff instance can now be used for inference safely and efficiently from multiple threads Link Link
  • cuDNN support added to SameDiff (automatically enabled for nd4j-cuda-10.x backend) Link
  • Added ND4J namespaces: Nd4j.cnn, Nd4j.rnn, Nd4j.image Link
  • Added new Image operations namespace operations:
    • rgbToHsv, hsvToRgb Link
    • rgbToYiq, yiqToRgb, rgbToYuv, yuvToRgb Link
    • imageResize Link
  • Added new Random operations namespace operations:
    • gamma, poisson, shuffle Link
  • Added new Math namespace operations:
    • clipByAvgNorm, embeddingLookup Link
    • mergeMaxIndex Link
  • Added new NN namespace operations:
  • Added new CNN namespace operations:
    • upsampling3d Link
  • Added new linalg operations namespace
    • triangular_solve Link
    • tri operation Link
    • triu operation Link
  • Added new RNN operation namespace operations:
    • lstmLayer (note old lstmLayer method renamed to lstmBlock) Link
    • gru Link
  • Added new Loss operations namespace - Nd4j.loss Link
  • Mapped operations for Tensorflow import:
    • HSVToRGB, RGBToHSV, Igamma, Igammac, RandomGamma, RandomPoisson, RandomPoissonV2, RandomShuffle Link
  • Added SameDiff ProfilingListener - writes op performance profiles in Chrome profiler format (load in chrome://tracing/) Link Link
  • Added SameDiff ProfileAnalyzer tool to compare profiles output from ProfilingListener (or Tensorflow) Link Link
  • SameDiff listener API: added frame and iteration information for listener methods Link Link
  • Added (non-backend-specific) method of accessing Nd4j environment: Nd4j.getEnvironment() method (environment info and low-level configuration options) Link Link
  • Improved memory limits/configuration support for libnd4j (c++) Link
  • Added pairwise (broadcastable) power backprop operation Link
  • Updated JavaCPP presets MKL version to 2020.0 from 2019.5 Link
  • Added DynamicCustomOp dargs - datatype arguments Link Link
    • Output datatype configuration for Range op Link, SequenceOp Link, ConfusionMatrix Link
  • Added tensormmul_bp op Link
  • OpenBLAS version upgraded to 0.3.8 Link
  • libnd4j (c++ codebase underlying DL4J, ND4J and SameDiff) refactored to be more easily embeddable in other C++ projects Link
  • ImagePreProcessingScaler now supports preprocessing of labels (for segmentation) Link
  • Additional datatypes now supported for nd4j-tensorflow TensorflowConversion Link
  • SameDiff operation namespaces (sd.math, sd.image, etc) are now code generated to ensure SameDiff and ND4J namespaces are identical (all operations included, same API) Link
  • Added ND4J ArchiveUtils.unzipFileTo(String, String, boolean logFiles) overload to enable/disable extracted file path logging Link
  • Added weight format configuration for following operations: conv1D, conv2D, conv3D, deconv2d, deconv3d, depthwiseConv2d, pointwiseConv2d, sconv2d Link
  • Added backprop operation implementations for mergemax, mergeadd, mergeavg operations Link
  • MKL version upgraded to 2020.0 2020.1; OpenCV upgraded from 4.2.0 to 4.3.0 Link
  • SameDiff: DifferentialFunctionFactory class removed in favor of namespace methods (sd.math, sd.linalg, etc) Link
  • Added lstmLayer_bp operation Link
  • Added gru_bp operation Link
  • linspace operation can now use both targs and arrays for start/end/size arguments Link
  • Assorted dependency updates - OpenBLAS (0.3.9), OpenCV (4.3.0), Leptonica (1.79.0) Link
  • Upgraded assorted dependency versions: javax.activation:activation (1.1 -> 1.1.1), stream analytics (2.7.0->2.9.8), Apache Spark (2.4.3->2.4.5), Jackson databind (2.10.1 -> 2.10.3), Vertx (3.8.3 -> 3.9.0) Link
  • Added nd4j-common-tests ResourceUtils.listClassPathfiles method Link

Bug Fixes and Optimizations

  • Updaters (Adam, AdaGrad, etc) optimized via C++ operations (significant training performance boost) for DL4J and SameDiff Link, Link
  • SameDiff - added CuDNN support Link
  • Some packages relocated to avoid split packages (that can be a problem for OSGi and Java 9 modules) Link
    • Note: this is a breaking change for some class packages/imports. See this link for details on exact package changes
  • Fixed some issues with Tensorflow import of FusedBatchNorm operation Link
  • Fixed an issue where the Roll operation did not match Tensorflow operation Link Link
  • Fixed an issue where ArchiveUtils could fail to create the top level destination directory when it does not exist Link
  • Fixed an issue where resize_bicubic operation did not match Tensorflow for some configuration values Link Link
  • Pad operation now supports long/int64 values for padding array Link Link
  • Fixed an issue where hashcode operation shape function wasn't always returning int64/long dtype Link
  • Fixed an issue with reshape operation on empty arrays with -1s Link Link
  • Improved performance on CUDA for concat operation Link and CPU/GPU Link
  • Improved performance for bias_add operation
    • On CPU for NHWC case Link
    • Generally Link
    • On CUDA for 2D case Link
  • Added MKLDNN (DNNL/OneDNN) support for depthwise_conv2d operation for DL4J and SameDiff Link
  • Fixed a small SameDiff execution issue for switch operation where the predicate is a constant Link
  • Fixed an issue with batchnorm operation when input arrays have unusual strides Link
  • Merged nd4j-buffer, nd4j-content modules into nd4j-api Link
  • Deleted deprecated nd4j-jackson module (remaining functionality available in nd4j-api) Link
  • Deleted unused/unmaintained nd4j-camel and nd4j-gson modules Link
  • Optimization for legacy random ops Link
  • Optimization for broadcast operations Link, Link, Link, Link, Link
  • Performance optimization for multiple operations: softmax, squeeze, expand_dims, tanh Link
  • Optimization for transpose/permute operations Link
  • Performance enhancement: MKLDNN matmul used for some mmul operation cases Link
  • Optimization for gather operation on CPU Link
  • Optimization for stack/unstack operations on CPU Link
  • Optimization for split operation (CPU and CUDA) Link Link
  • ND4J initialization no longer logs number of OpenMP BLAS threads for CUDA Link
  • Optimization: Fixed issues with auto-vectorization on multple CPU operations Link
  • Optimization for reshape operation Link, Link
  • Fixed an issue where INDArray.hashCode() could cause an exception on some datatypes Link
  • Optimization for CPU: MKLDNN is now used for softmax, tanh, softmax_bp and tanh_bp operations Link, Link, Link, Link
  • Fixed random_exponential operation Link
  • Improved performance on C++ SameDiff graph execution via reduced array zeroing where safe to do so Link
  • Improved C++ indexing implementation impacting CPU performance on some operations Link
  • Fixed an issue where Split operation could have incorrect output shapes for empty arrays Link
  • Fixed some issues with SameDiff.equals method Link
  • Fixed an issue with reshape operation output shape on empty arrays Link, Link
  • Nd4j.gemm now uses Mmul operation internally to avoid potential threading issues with direct BLAS calls on CUDA Link
  • Fixed an edge case issue with percentile operation link
  • Fixed an edge case issue for cusolved (CUDA) in libnd4j Link
  • Fixed an issue with error formatting for segment operations for incorrect lengths Link
  • Fixed an issue where ND4J workspaces were not guaranteed to be unique Link
  • Fixed some operation implementations when operating on views (Batch/Space to Space/Batch/Depth; batchnorm_bp) Link
  • Fixed an issue where exponential distribution random number generation operation could produce infinities extremely rarely (~1 in 10^9 values) Link
  • Fixed an issue with long file paths for memory mapped workspaces on Windows Link
  • Memory for memory mapped workspaces are now deallocated immediately when workspace is destroyed, instead of waiting for GC to free memory Link
  • Fall-back to other BLAS implementation for cases where MKLDNN GEMM implementation is slow Link
  • Set nd4j-native source/target to Java 7 Link, Link

DataVec

Features and Enhancements

  • datavec-python: added zero-copy support for bytes/byte buffers Link
  • datavec-python: Python exceptions are now thrown as Java exceptions Link
  • datavec-python: Added support for additional NumPy datatypes Link
  • datavec-python: Python version upgraded from 3.7.6 to 3.7.7 Link

Bug Fixes and Optimizations

  • Deleted not properly maintained modules: datavec-camel, datavec-perf Link
  • Fixed missing BOOL datatype support for arrow conversion functionality Link
  • Assorted fixes for datavec-python Link Link, Link
  • Fixed an issue with LineRecordReader where initialization was performed unnecessarily (adding performance overhead) Link

RL4J

Features and Enhancements

  • Refactoring to decouple configuration and learning methods from their implementations Link
  • Added builder patterns for all configuration classes Link

Arbiter

Bug Fixes and Optimizations

  • Fixes an issue with GridSearchCandidateGenerator not working correctly for some cases Link, Link

Version 1.0.0-beta6

Highlights - 1.0.0-beta6 Release

  • Added support for CUDA 10.2. 1.0.0-beta6 released with CUDA 9.2, 10.0, 10.1 and 10.2 support
  • SameDiff optimizations - memory use for inference and training significantly reduced, with some performance improvements also
  • Deeplearning4j UI - Play framework replaced with Vertx; deeplearning4j-ui dependency now no longer has Scala dependency or Scala version suffix Link
    • Note: No API changes, only artifact ID change: replace deeplearning4j-ui_2.1x with deeplearning4j-ui
  • ND4j namespace operation methods: operations are available through the Nd4j.math, Nd4j.random, Nd4j.bitwise, Nd4j.nn (neural network), for example Nd4j.math.abs(INDArray), Nd4j.random.logNormal etc Link.
    • Note that additional ND4J namespaces API will have additions (new namespaces and methods), and may have some API changes, in the next release
  • OpenMP replaced with thread pool c++ parallelism framework; enabled c++ parallelism for platforms without C++-level threading for operations

Deeplearning4J

Deeplearning4J: Features and Enhancements

  • DNNL (MKL-DNN) upgraded to version 1.1
  • Added causal convolution mode for Convolution1D layer (ConvolutionMode.Causal) and added causal conv1d support for Keras import Link
  • Keras import now supports scaled identity weight initialization Link
  • Added Mish activation function Link, Link
  • BertIterator now has a BertIterator.featurizeSentences(List<String>) method for inference Link, Link
  • BertIterator now supports sentence pairs for supervised training Link
  • Added Sparse multi-class cross entropy for both Deeplearning4j and Keras import Link, Link
  • Deeplearning4j UI: migrated from Play to Vertx for web serving backend, also removing dependency on Scala libraries; no API changes, only artifact ID change - replace deeplearning4j-ui_2.1x with deeplearning4j-ui Link, Link
  • Added TimeDistributed wrapper layer Link

Deeplearning4J: Bug Fixes and Optimizations

  • KDTree implementation optimized Link
  • Deeplearning4j zoo models and datasets hosting location updated Link
  • Fixed nIn validation for Deconv2D layer Link
  • Fixed an issue with incorrect Deconvolution2d results for Keras import models Link
  • Added DNNL/MKLDNN support for batch normalization layer Link, Link
  • Fixed various integer casts to avoid overflows for very large arrays (with dimensions or length > Integer.MAX_VALUE) Link
  • Fixed an issue with UNet non-pretrained model architecture (last layer kernel size) Link
  • Deeplearning4j SameDiff layers now use DL4J workspaces for better performance and reduced memory consumption Link
  • Updated broken links in afew error messages Link
  • Cleaned up a few unused dependencies in various modules Link
  • Cleaned up duplicate SamplingDataSetIterator class Link
  • Fixed an issue where ComputationGraph instances with a single input going into multiple embedding layers could throw a NPE Link
  • Fixed an issue where loss function weights were not automatically cast to network datatype, resulting in an exception if not already correct type Link
  • Shaded Jackson version upgraded from 2.9.9/2.9.9.3 to 2.10.1 Link
  • Fixed an issue with KNN where getMostPopulatedClusters actually returned the least populated clusters Link

Deeplearning4j: Transition Guide, 1.0.0-beta5 to 1.0.0-beta6

  • Deeplearning4j UI artifact ID has changed: deeplearning4j-ui_2.1x (beta5 and earlier) with deeplearning4j-ui

ND4J and SameDiff

ND4J/SameDiff: Features and Enhancements

  • Added suport for CUDA 10.2 Link
  • DNNL (MKL-DNN) upgraded to version 1.1 Link
  • Added ND4j namespaces to match SameDiff: Nd4j.math, Nd4j.random, Nd4j.bitwise, Nd4j.nn (neural network) Link
  • Added SameDiff.calculateGradientsAndOutputs method Link Link
  • Additional SameDiff single batch .output method overloads for DataSet/MultiDataSet added Link
  • TensorFlow import ops coverage enhanced (significant number of additional ops supported) Link, Link, Link, Link, Link
  • PRelu op added Link
  • adjust_contrast, igamma and igammac ops added Link
  • ND4J/SameDiff: BitCast, CompareAndBitpack, DivideNoNan, DrawBoundingBoxes, FakeQuantWithMinMaxVarsPerChannel ops added Link
  • non_max_suppression_overlaps op added Link
  • ImagePreProcessingScaler now supports segmentation use cases Link
  • concat operation now supports the concatenation axis being specified via the last input array Link
  • Added Gamma and Poisson RNG distributions Link
  • SameDiff’s use of DeviceLocal for variables/constants etc is now configurable Link
  • Uniform distribution op now supports random integer generation, not just random floating point generation Link
  • SameDiff: Added simple OpBenchmarkListener for benchmarking purposes Link
  • Added the ability to disable platform helpers (DNNL/MKLDNN etc) via Nd4jCPU.Environment.getInstance().allowHelpers(false); and Nd4jCuda.Environment.getInstance().allowHelpers(false); Link
  • Added draw_bounding_boxes operation Link
  • Added resize_bicubic operation Link
  • Added causal padding mode to conv1d operation Link
  • DNNL (MKLDNN) is included and enabled by default for non-AVX builds Link
  • Added SameDiff ArraySavingListener for debugging purposes Link

ND4J/SameDiff: Bug Fixes and Optimizations

  • OpenMP replaced with ThreadPool abstraction, enables parallelism for platforms without OpenMP support Link
  • SameDiff memory management overheauled for (in some cases significantlny) reduced memory consumption and improved performance Link, Link
  • Switched to Clang instead of gcc for OSX compilation to avoid compiler-related issues Link
  • Removed SameDiff.outputs() “best guess” output inference due to being unreliable, in favor of explicit SameDiff.setOutputs(String...) call Link
  • Fixed an issue with Nd4j.hstack on 1D arrays Link
  • SameDiff no longer allows empty arrays for variables Link
  • Fixed an issue with Nadam updater LR schedules not being cloned Link
  • Cleaned up IActivation interface Link
  • Added new LSTM op implementation with DNNL/MKLDNN support (forward pass only so far) Link
  • SameDiff API cleaned up; deprecated methods removed Link
  • Switched SameDiff variable initialization to non-lazy, to avoid unexpected behaviour when mixing execution and ND4J RNG seed setting Link
  • SameDiff.zero and .one methods now create constants, not vairables Link
  • Moved CUDA build version and device logging to Java logging, from c++ stdout to enable disabling logging (via ND4J config or slf4j config) Link
  • Added DNNL/MKLDNN support for batch normalization Link
  • SameDiff: Fixed an issue where listeners weren’t being called for gradient calculation Link
  • Added DNNL/MKLDNN support for deconv2d/3d operations Link
  • Fixed an issue with biasadd_bp operation and NHWC data format Link
  • Fixed an issue with certain strided slice backprop configurations Link, Link
  • Fixed an issue with LogSumExp reduction operation backprop for along dimension case Link, Link
  • INDArray.toString() now has correct brackets for rank 1+ scalars to avoid ambiguity Link
  • Fixed an issue where some ND4J methods could fail when the library is compiled on Java 9+ but run on Java 8 Link
  • Fixed empty array input case for is_strictly_increasing, non_decreasing and non_max_suppression ops Link, Link
  • Fixed empty input arrays for legacy ops (transform, scalar, pairwise, broadcast) Link
  • CUDA compute capability 3.0 is supported again Link
  • Improved performance for Scatter operations (1D case) + index validation Link
  • Fixed an issue where SameDiff TrainingConfig serialization would fail if evaluation instances are set Link, Link
  • SameDiff execution will now throw an exception when assertion operations in the graph fail Link
  • PolyGamma function now returns NaNs when passed double for args requiring integer values Link
  • Fixed some issues for pad and mirror_pad ops to ensure they conform with Tensorflow for imported networks Link
  • Updated and fixed some issues for TensorFlow graph runner Link
  • Improved performance for Reverse operation Link
  • Removed/cleanup up unused ND4J list functionality Link
  • Fixed reduce bool operation results (such as any, all, IsInf, etc) for empty array inputs Link

ND4J: Transition Guide, 1.0.0-beta5 to 1.0.0-beta6

  • SameDiff.outputs() now requires user to call SameDiff.setOutputs(String...) first; previous “best guess” output inference was unreliable Link
  • SameDiff.zero and .one methods now create constants, not vairables Link

DataVec

DataVec: Bug Fixes and Optimizations

  • NativeImageLoader now checks for empty input streams and throws an exception instead of crashing Link
  • NDArrayScalarOpTransform now supports modulus operator Link

RL4J

RL4J: Features and Enhancements

  • Added AsyncTrainingListener Link
  • Replaced multiple uses of java.util.Random with ND4J Random Link
  • Added Observable and LegacyMDPWrapper Link

RL4J: Bug Fixes and Optimizations

  • Refactored RL4J video recording to separate VideoRecorder class Link
  • Fixed an issue with target for DQN Link, Link
  • Refactoring for DQN and double DQN for improved maintainability Link
  • Internal refactoring and various bug fixes Link

PyDataVec

PyDataVec Features and Enhancements

  • PyDataVec TransformProcess now supports non-inplace operations Link

PyDataVec Bug Fixes and Optimizations

  • Fixed various issues with PyDataVec Link
  • Fixed an issue with data locality that could cause incorrect results under some circumstances when running on CUDA Link

Version 1.0.0-beta5

Highlights - 1.0.0-beta5 Release

  • Added model server - remote inference of SameDiff and DL4J models using JSON or (optionally) binary serialization
  • Added Scala 2.12 support, dropped Scala 2.10 support. Modules with Scala dependencies are now released with Scala 2.11 and 2.12 versions
  • Apache Spark 1.x support dropped (now only Spark 2.x is supported). Note: Spark version suffix dropped: For upgrading: 1.0.0-beta4_spark2 -> 1.0.0-beta5
  • Added FastText support to deeplearning4j-nlp
  • CUDA support for all ND4J/SameDiff Operations
    • In 1.0.0-beta4, some operations were CPU only. Now, all operations have full CUDA support
  • Added support for new data types in ND4J (and DL4J/SameDiff): BFLOAT16, UINT16, UINT32, UINT64
  • ND4J: Implicit broadcasting support added to INDArray (already present in SameDiff - for example shape [3,1]+[3,2]=[3,2])
  • CUDA 9.2, 10.0 and 10.1-Update2 still supported
    • NOTE: For CUDA 10.1, CUDA 10.1 update 2 is recommended. CUDA 10.1 and 10.1 Update 1 will still run, but rare internal cuBLAS issues may be encountered in heavily multi-threaded code on some systems
  • Dependency upgrades: Jackson (2.5.1 to 2.9.9/2.9.9.3), Commons Compress (1.16.1 to 1.18), Play Framework (2.4.8 to 2.7.3), Guava: (20.0 to 28.0-jre, and shaded to avoid dependency clashes)
  • CUDA: now host (RAM) buffers are only allocated when required (previously: host buffers were always allocated), in addition to device (GPU) buffer

Deeplearning4J

Deeplearning4J: Features and Enhancements

  • Added FastText - inference and training, including OOV (out of vocabulary) support (Link)
  • Scala 2.12 support added, Scala 2.10 support dropped (Link)
  • Added model server (DL4J and SameDiff models, JSON and binary communication) - JsonModelServer, JsonRemoteInference, Link, Link
  • Added saved model format validation utilities - DL4JModelValidator, DL4JKerasModelValidator (Link)
  • Added LabelLastTimeStepPreProcessor (Link)
  • BertIterator: added option to prepend token to the output (such as [cls] expected by some models) (Link)
  • Added trace level logging to MultiLayerNetwork and ComputationGraph assist with debugging certain issues (Link)
  • Upsampling3D: Added NDHWC support (Link)
  • MergeVertex now supports broadcasting (Link)
  • LSTM and Dropout will now fall back on built-in implementations if an exception is encountered from cuDNN (same as Subsampling/ConvolutionLayer) (Link)
  • Improved JavaDoc and cleanup up API for WordVectorSerializer (Link, Link)

Deeplearning4J: Bug Fixes and Optimizations

  • Updated deeplearning4j-ui theme (Link)
  • Fixed an issue with MergeVertex and CNN3D activations (Link)
  • Fixed typo in Yolo2OutputLayer builder/configuration method name (Link)
  • Improved ComputationGraph builder InputType validation (Link)
  • Removed dl4j-spark-ml module until it can be properly maintained (Link)
  • Fixed an issue with BertWordPieceTokenizerFactory and bad character encoding (Link)
  • Fixed an issue with LearnedSelfAttentionLayer and variable minibatch size (Link, Link)
  • Fixed issue with SharedTrainingMaster controller address when set from environment variable (Link)
  • Fixed issue with SameDiffOutputLayer initialization under some circumstances (Link)
  • https is now used by default for data and zoo model downloads (Link, Link)
  • Fixed an issue where UI WebJars dependencies would check for updates on every single build (Link, Link)
  • Fixed issue where Upsampling layer memory report could produce an OOM exception (Link)
  • Improved UX/validation for RecordReaderDataSetIterator (Link)
  • Fixed an issue where EmbeddingSequenceLayer would not check mask array datatype (Link)
  • Improved validation when initializing networks with a non rank-2 (shape [1, numParams]) array (Link)
  • Fixed a DataType issue for BertIterator (Link)
  • Fixed Word2Vec model backward compatibilty (beta3 and earlier models now loadable again) Link
  • Fixed issue where some Keras import models could fail with Could not read abnormally long HDF5 attribute (Link)
  • Added validation for RnnOutputLayer - feature/label array lengths (Link)
  • Fixed an issue where SameDiffOutputLayer would not support variable minibatch size (Link)
  • Fixed DL4J SameDiff layer mask support (Link)
  • DL4J UI: Fixed an issue where tab switching did not work when visualizing saved/stored data (Link, Link)
  • DL4J UI: Fixed a rare UI threading issue (Link)
  • Fixed a Keras import issue with JSON format change (Link)
  • Fixed a Keras import issue where updater learning rate schedule could be imported incorrectly (Link)
  • Fixed an issue with CnnSentenceDataSetIterator when using UnknownWordHandling.UseUnknownVector (Link, Link)
  • Fixes and optimizations to DL4J SameDiff layers (Link)
  • MultiLayerNetwork/ComputationGraph will now log the original exception if a second exception occurs during workspace closing, instead of swallowing it (inference/fit operation try/finally blocks) (Link)
  • Upgraded dependencies: Jackson (2.5.1 to 2.9.9/2.9.9.3), Commons Compress (1.16.1 to 1.18), Play Framework (2.4.8 to 2.7.3), Guava: (20.0 to 28.0-jre, shaded to avoid dependency clashes) (Link)
  • Logging framework can now be configured for DL4J UI (due to Play framework dependency upgrade) (Link)
  • Reduced amount of garbage produced by MnistDataFetcher (impacts MNIST and EMNIST DataSetIterators) (Link)
  • Activation function backpropagation has been optimized for many activation functions (Link, Link)

Deeplearning4j: Transition Guide, 1.0.0-beta4 to 1.0.0-beta5

  • DL4J AsyncDataSetIterator and AsyncMultiDataSetIterator moved to ND4J, use org.nd4j.linalg.dataset.Async(Multi)DataSetIterator instead
  • Saved models with custom layers from 1.0.0-alpha and before can no longer be loaded. Workaround: load in 1.0.0-beta4, and re-save the model (Link). Models without custom layers can still be loaded back to 0.5.0
  • Apache Spark 1.x support dropped (now only Spark 2.x is supported). Note: Spark version suffix dropped: For upgrading, change versions as follows: 1.0.0-beta4_spark2 -> 1.0.0-beta5
  • Scala 2.10 dropped, Scala 2.12 added (for modules with Scala dependencies)

Deeplearning4j: 1.0.0-beta5 Known Issues

  • dl4j-spark_2.11 and _2.12 dependencies incorrectly pull in datavec-spark_2.11/2.12 version 1.0.0-SNAPSHOT. Workaround: control version using dependency management as per here or here
  • Some layers (such as LSTM) may run slower on 1.0.0-beta5 than 1.0.0-beta4 on CUDA when not using cuDNN, due to added synchronization. This synchronization will be removed in the next release after 1.0.0-beta5
  • CUDA 10.1: Rare internal cuBLAS issues may be encountered in heavily multi-threaded code on some systems, when running CUDA 10.1 Update 1 (and maybe 10.1). CUDA 10.1 update 2 is recommended.

ND4J and SameDiff

ND4J/SameDiff: Features and Enhancements

  • Added new data types: BFLOAT16, UINT16, UINT32, UINT64 (Link)
  • CUDA support for all operations without CUDA implementations (Link, Link, Link, Link, Link)
  • Added model server (DL4J and SameDiff models, JSON and binary communication) - JsonModelServer, JsonRemoteInference, Link, Link
  • Added support for empty arrays with zeros in shape, for compatibility with TensorFlow import (Link)
  • CUDA: now host (RAM) buffers are only allocated when required (previously: host buffers were always allocated), in addition to device (GPU) buffer
  • Improved SameDiff training API - added "in line" test set evaluation, returning History object with loss curve, etc (Link)
  • Added saved model format validation utilities - Nd4jValidator, Nd4jCommonValidator (Link)
  • Added SameDiff ScoreListener (equivalent to DL4J ScoreIterationListener/PerformanceListener) (Link, Link)
  • Added SameDiff.convertDataTypes method, for variable dtype conversion (Link)
  • Added crop and resize op (Link)
  • DL4J AsyncDataSetIterator and AsyncMultiDataSetIterator moved to ND4J Link
  • Added basic/MVP SameDiff UI listener (Link)
  • Added SameDiff CheckpointListener (Link, Link)
  • Added SameDiff name scopes (Link)
  • SameDiff: Updater state and training configuration is now written to FlatBuffers format (Link)
  • Added c++ benchmark suite callable from Java - call using Nd4j.getExecutioner().runLightBenchmarkSuit() and Nd4j.getExecutioner().runFullBenchmarkSuit() (Link)
  • Added SameDiff.save/load methods with InputStream/OutputStream arguments (Link, Link)
  • Added axis configuraiton for evaluation instances (Evaluation, RegressionEvaluation, ROC, etc - getAxis and setAxis methods) to allow different data formats (NCHW vs. NHWC for CNNs, for example) (Link)
  • SameDiff: Added support to convert constants to placeholders, via SDVariable.convertToConstant() method (Link)
  • SameDiff: Added GradCheckUtil.checkActivationGradients method to check activation gradients for SameDiff instance (not just parameter gradients as in existing gradient check methods) (Link)
  • Added CheckNumerics op (Link)
  • Added FakeQuantWithMinMaxArgs and FakeQuantWithMinMaxVars ops (Link)
  • Added INDArray reduction methods with "keep dimensions" option - for example, INDArray.mean(boloean, int... dimension) (Link)