1.0.0-beta6

Highlights - 1.0.0-beta6 Release

• Added support for CUDA 10.2. 1.0.0-beta6 released with CUDA 9.2, 10.0, 10.1 and 10.2 support

• SameDiff optimizations - memory use for inference and training significantly reduced, with some performance improvements also

• Deeplearning4j UI - Play framework replaced with Vertx; deeplearning4j-ui dependency now no longer has Scala dependency or Scala version suffix Link

  • Note: No API changes, only artifact ID change: replace deeplearning4j-ui_2.1x with deeplearning4j-ui

• ND4j namespace operation methods: operations are available through the Nd4j.math, Nd4j.random, Nd4j.bitwise, Nd4j.nn (neural network), for example Nd4j.math.abs(INDArray), Nd4j.random.logNormal etc Link.

  • Note that additional ND4J namespaces API will have additions (new namespaces and methods), and may have some API changes, in the next release

• OpenMP replaced with thread pool c++ parallelism framework; enabled c++ parallelism for platforms without C++-level threading for operations

Deeplearning4J

Deeplearning4J: Features and Enhancements

• DNNL (MKL-DNN) upgraded to version 1.1

• Added causal convolution mode for Convolution1D layer (ConvolutionMode.Causal) and added causal conv1d support for Keras import Link

• Keras import now supports scaled identity weight initialization Link

• Added Mish activation function Link, Link

• BertIterator now has a BertIterator.featurizeSentences(List<String>) method for inference Link, Link

• BertIterator now supports sentence pairs for supervised training Link

• Added Sparse multi-class cross entropy for both Deeplearning4j and Keras import Link, Link

• Deeplearning4j UI: migrated from Play to Vertx for web serving backend, also removing dependency on Scala libraries; no API changes, only artifact ID change - replace deeplearning4j-ui_2.1x with deeplearning4j-ui Link, Link

• Added TimeDistributed wrapper layer Link

Deeplearning4J: Bug Fixes and Optimizations

• KDTree implementation optimized Link

• Deeplearning4j zoo models and datasets hosting location updated Link

• Fixed nIn validation for Deconv2D layer Link

• Fixed an issue with incorrect Deconvolution2d results for Keras import models Link

• Added DNNL/MKLDNN support for batch normalization layer Link, Link

• Fixed various integer casts to avoid overflows for very large arrays (with dimensions or length > Integer.MAX_VALUE) Link

• Fixed an issue with UNet non-pretrained model architecture (last layer kernel size) Link

• Deeplearning4j SameDiff layers now use DL4J workspaces for better performance and reduced memory consumption Link

• Updated broken links in afew error messages Link

• Cleaned up a few unused dependencies in various modules Link

• Cleaned up duplicate SamplingDataSetIterator class Link

• Fixed an issue where ComputationGraph instances with a single input going into multiple embedding layers could throw a NPE Link

• Fixed an issue where loss function weights were not automatically cast to network datatype, resulting in an exception if not already correct type Link

• Shaded Jackson version upgraded from 2.9.9/2.9.9.3 to 2.10.1 Link

• Fixed an issue with KNN where getMostPopulatedClusters actually returned the least populated clusters Link

Deeplearning4j: Transition Guide, 1.0.0-beta5 to 1.0.0-beta6

• Deeplearning4j UI artifact ID has changed: deeplearning4j-ui_2.1x (beta5 and earlier) with deeplearning4j-ui

ND4J and SameDiff

ND4J/SameDiff: Features and Enhancements

• Added suport for CUDA 10.2 Link

• DNNL (MKL-DNN) upgraded to version 1.1 Link

• Added ND4j namespaces to match SameDiff: Nd4j.math, Nd4j.random, Nd4j.bitwise, Nd4j.nn (neural network) Link

• Added SameDiff.calculateGradientsAndOutputs method Link Link

• Additional SameDiff single batch .output method overloads for DataSet/MultiDataSet added Link

• TensorFlow import ops coverage enhanced (significant number of additional ops supported) Link, Link, Link, Link, Link

• PRelu op added Link

• adjust_contrast, igamma and igammac ops added Link

• ND4J/SameDiff: BitCast, CompareAndBitpack, DivideNoNan, DrawBoundingBoxes, FakeQuantWithMinMaxVarsPerChannel ops added Link

• non_max_suppression_overlaps op added Link

• ImagePreProcessingScaler now supports segmentation use cases Link

• concat operation now supports the concatenation axis being specified via the last input array Link

• Added Gamma and Poisson RNG distributions Link

• SameDiff’s use of DeviceLocal for variables/constants etc is now configurable Link

• Uniform distribution op now supports random integer generation, not just random floating point generation Link

• SameDiff: Added simple OpBenchmarkListener for benchmarking purposes Link

• Added the ability to disable platform helpers (DNNL/MKLDNN etc) via Nd4jCPU.Environment.getInstance().allowHelpers(false); and Nd4jCuda.Environment.getInstance().allowHelpers(false); Link

• Added draw_bounding_boxes operation Link

• Added resize_bicubic operation Link

• Added causal padding mode to conv1d operation Link

• DNNL (MKLDNN) is included and enabled by default for non-AVX builds Link

• Added SameDiff ArraySavingListener for debugging purposes Link

ND4J/SameDiff: Bug Fixes and Optimizations

• OpenMP replaced with ThreadPool abstraction, enables parallelism for platforms without OpenMP support Link

• SameDiff memory management overheauled for (in some cases significantlny) reduced memory consumption and improved performance Link, Link

• Switched to Clang instead of gcc for OSX compilation to avoid compiler-related issues Link

• Removed SameDiff.outputs() “best guess” output inference due to being unreliable, in favor of explicit SameDiff.setOutputs(String...) call Link

• Fixed an issue with Nd4j.hstack on 1D arrays Link

• SameDiff no longer allows empty arrays for variables Link

• Fixed an issue with Nadam updater LR schedules not being cloned Link

• Cleaned up IActivation interface Link

• Added new LSTM op implementation with DNNL/MKLDNN support (forward pass only so far) Link

• SameDiff API cleaned up; deprecated methods removed Link

• Switched SameDiff variable initialization to non-lazy, to avoid unexpected behaviour when mixing execution and ND4J RNG seed setting Link

• SameDiff.zero and .one methods now create constants, not vairables Link

• Moved CUDA build version and device logging to Java logging, from c++ stdout to enable disabling logging (via ND4J config or slf4j config) Link

• Added DNNL/MKLDNN support for batch normalization Link

• SameDiff: Fixed an issue where listeners weren’t being called for gradient calculation Link

• Added DNNL/MKLDNN support for deconv2d/3d operations Link

• Fixed an issue with biasadd_bp operation and NHWC data format Link

• Fixed an issue with certain strided slice backprop configurations Link, Link

• Fixed an issue with LogSumExp reduction operation backprop for along dimension case Link, Link

• INDArray.toString() now has correct brackets for rank 1+ scalars to avoid ambiguity Link

• Fixed an issue where some ND4J methods could fail when the library is compiled on Java 9+ but run on Java 8 Link

• Fixed empty array input case for is_strictly_increasing, non_decreasing and non_max_suppression ops Link, Link

• Fixed empty input arrays for legacy ops (transform, scalar, pairwise, broadcast) Link

• CUDA compute capability 3.0 is supported again Link

• Improved performance for Scatter operations (1D case) + index validation Link

• Fixed an issue where SameDiff TrainingConfig serialization would fail if evaluation instances are set Link, Link

• SameDiff execution will now throw an exception when assertion operations in the graph fail Link

• PolyGamma function now returns NaNs when passed double for args requiring integer values Link

• Fixed some issues for pad and mirror_pad ops to ensure they conform with Tensorflow for imported networks Link

• Updated and fixed some issues for TensorFlow graph runner Link

• Improved performance for Reverse operation Link

• Removed/cleanup up unused ND4J list functionality Link

• Fixed reduce bool operation results (such as any, all, IsInf, etc) for empty array inputs Link

ND4J: Transition Guide, 1.0.0-beta5 to 1.0.0-beta6

• SameDiff.outputs() now requires user to call SameDiff.setOutputs(String...) first; previous “best guess” output inference was unreliable Link

• SameDiff.zero and .one methods now create constants, not vairables Link

DataVec

DataVec: Bug Fixes and Optimizations

• NativeImageLoader now checks for empty input streams and throws an exception instead of crashing Link

• NDArrayScalarOpTransform now supports modulus operator Link

RL4J

RL4J: Features and Enhancements

• Added AsyncTrainingListener Link

• Replaced multiple uses of java.util.Random with ND4J Random Link

• Added Observable and LegacyMDPWrapper Link

RL4J: Bug Fixes and Optimizations

• Refactored RL4J video recording to separate VideoRecorder class Link

• Fixed an issue with target for DQN Link, Link

• Refactoring for DQN and double DQN for improved maintainability Link

• Internal refactoring and various bug fixes Link

PyDataVec

PyDataVec Features and Enhancements

• PyDataVec TransformProcess now supports non-inplace operations Link

PyDataVec Bug Fixes and Optimizations

• Fixed various issues with PyDataVec Link

• Fixed an issue with data locality that could cause incorrect results under some circumstances when running on CUDA Link