Highlights

Adds proper support for java 9 modules: https://github.com/eclipse/deeplearning4j/pull/9631 https://github.com/eclipse/deeplearning4j/pull/9626

As part of the same work flatbuffers has been upgraded to 1.12.1. This affects the samediff file format and the user interfaces. Flatbuffers as a file format is forwards and backwards compatible but if you have any issues please do let us know. The relevant files have been updated using the flatc compiler.

Removed rl4j: in continuing to cut unmaintained modules, the 1.0 will focus the framework on a few key use cases. This invites other folks to build external modules for a tightly maintained core that focuses on deployment, framework interop and training models in java.

Added new model zoo module called omnihub for dl4j and new samediff models. These can be found here: https://github.com/KonduitAI/omnihub-zoo See more in the new omnihub section.

Migrated the snapshots to sonatype's new repository https://s01.oss.sonatype.org. More context can be found here: https://twitter.com/Brian_Fox/status/1357414532512104448 https://github.com/eclipse/deeplearning4j/pull/9618

Consolidated tests to platform-tests to allow for easy testing of behavior against different backends.

Adds proper support for jetson nano with curated binaries and an updated cuda 10.2

Adds Spark 3 support: https://github.com/eclipse/deeplearning4j/pull/9444

Reduce binary size using selective compilation: https://github.com/eclipse/deeplearning4j/pull/9443

https://github.com/eclipse/deeplearning4j/pull/9451 Remove scala 11 support. Only supporting scala 2.12: https://github.com/eclipse/deeplearning4j/pull/9440

Extensive enhancements for samediff model training: https://github.com/eclipse/deeplearning4j/pull/9501

Nd4j/Samdiff/Libnd4j

Features and Enhancements

1. Add beginnings of graph optimization framework: https://github.com/eclipse/deeplearning4j/pull/9402

2. Many onnx model import improvements (add new ops): https://github.com/eclipse/deeplearning4j/pull/9411 https://github.com/eclipse/deeplearning4j/pull/9489https://github.com/eclipse/deeplearning4j/pull/9475 https://github.com/eclipse/deeplearning4j/pull/9526 https://github.com/eclipse/deeplearning4j/pull/9502https://github.com/eclipse/deeplearning4j/pull/9587 https://github.com/eclipse/deeplearning4j/pull/9599

3. Add new op subset frameworks: allows selective inclusion of operations to enable users to reduce binary size: https://github.com/eclipse/deeplearning4j/pull/9443 https://github.com/eclipse/deeplearning4j/pull/9451 https://github.com/eclipse/deeplearning4j/pull/9569

4. Update onednn to 2.2: https://github.com/eclipse/deeplearning4j/pull/9423 https://github.com/eclipse/deeplearning4j/pull/9425

5. Add updated jetson nano support: https://github.com/eclipse/deeplearning4j/pull/9432

6. Enhance codegen exposing more functions for samediff: https://github.com/eclipse/deeplearning4j/pull/9478 https://github.com/eclipse/deeplearning4j/pull/9503 https://github.com/eclipse/deeplearning4j/pull/9500

7. Add new samediff eager mode (mainly used for model import use cases): https://github.com/eclipse/deeplearning4j/pull/9538 https://github.com/eclipse/deeplearning4j/pull/9535 https://github.com/eclipse/deeplearning4j/pull/9533

8. Add dimensions as input variables: https://github.com/eclipse/deeplearning4j/pull/9584

Bug Fixes

1. Update samediff api to allow dimensions as variables

2. Fix cuda shuffle: https://github.com/eclipse/deeplearning4j/pull/9472 https://github.com/eclipse/deeplearning4j/pull/9459

3. Fix up conditions/matching: https://github.com/eclipse/deeplearning4j/pull/9551

4. ImageResize updates to improve compatibility with onnx: https://github.com/eclipse/deeplearning4j/pull/9495

5. Rewrite compat sparse to dense op: https://github.com/eclipse/deeplearning4j/pull/9566

6. Fix creation of string scalar ndarrays: https://github.com/eclipse/deeplearning4j/pull/9556

7. Fix serialization with conv/pooling3d: https://github.com/eclipse/deeplearning4j/pull/9648

Deeplearning4j

Features and Enhancements

1. Add Spark 3 support: https://github.com/eclipse/deeplearning4j/pull/9553

2. Added Deconvolution3D for keras import https://github.com/eclipse/deeplearning4j/pull/9399

3. Add full channels last support for 3d convolutions: https://github.com/eclipse/deeplearning4j/pull/9578

Bug Fixes

1. Fix confusion matrix count increments: https://github.com/eclipse/deeplearning4j/pull/9553

2. Fix Conv3D data format serialization: https://github.com/eclipse/deeplearning4j/pull/9648

Datavec

Features and Enhancements

1. Add LabelsSource to BagOfWordsVectorizer (thanks to XAI!): https://github.com/eclipse/deeplearning4j/pull/9624

2. Performance enhancement for mnist related datasetiterators: https://github.com/eclipse/deeplearning4j/pull/9612

Bug Fixes

1. Fix memory leak in datavec-arrow: https://github.com/eclipse/deeplearning4j/pull/9441

Omnihub

Launches new Omnihub module. Allows access to models from: https://github.com/KonduitAI/omnihub-zoo

A pretrained omnihub module will provide access to pretrained samediff and dl4j modules. This will also supplant the old dl4j zoo.

Modules will be made available from a Pretrained class:https://github.com/eclipse/deeplearning4j/blob/feb8eee5eb07239c49a4d14786114dc0394aad4e/omnihub/src/main/java/org/eclipse/deeplearning4j/omnihub/models/Pretrained.java#L30

Python4j

Clean up tests/consolidate tests to platform-tests

Highlights

A number of bug fixes following the M1 release, thanks to the feedback from the community, allowed us to quickly sort out a few issues. This is a minor bug fix release to address short comings found with M1. Most fixes were related to keras import, the cnn/rnn helpers, and python4j.

Snapshots will also be published every 2 days automatically now https://github.com/eclipse/deeplearning4j/pull/9355 to get around sonatype ossrh deletion of snapshots every 3 days. This should increase robustness of the snapshots.

Worked around an issue with github actions pre emptively upgrading visual studio breaking the cuda builds: https://github.com/eclipse/deeplearning4j/pull/9364

Added backwards compatibility for centos 6 via a new linux-x86_64-compat classifier enabling use of older glibcs on centos 7:

https://github.com/eclipse/deeplearning4j/pull/9368 https://github.com/eclipse/deeplearning4j/pull/9368https://github.com/eclipse/deeplearning4j/pull/9373

A number of bugs were fixed with LSTM and CUDNN: https://github.com/eclipse/deeplearning4j/pull/9372

Known issues

https://github.com/eclipse/deeplearning4j/issues/9142 - avoid shuffle operations on gpu. Pre save data on cpu in mini batches. For more help, please post on the forums at https://community.konduit.ai/

Deeplearning4j

Features and Enhancements

• Add batch normalization support for RNNs: https://github.com/eclipse/deeplearning4j/pull/9338

• Disable old helpers by default https://github.com/eclipse/deeplearning4j/pull/9343

• Minor unit test fixes: https://github.com/eclipse/deeplearning4j/pull/9346

• Add keras support for cnn 1d NWHC: https://github.com/eclipse/deeplearning4j/pull/9353

• Move the warning about version check to tracing so it stops logging this during normal usage confusing users: https://github.com/eclipse/deeplearning4j/pull/9356

• Allow 1d convolutions to accept feed forward as input type: https://github.com/eclipse/deeplearning4j/pull/9365

• Remove the old benchmark suite and migrate it to contrib: https://github.com/eclipse/deeplearning4j/pull/9374

• Remove old MKLDNNLSTM helper (it never fully functioned anyways): https://github.com/eclipse/deeplearning4j/pull/9381

Bug fixes

• Fixed an issue with helper reflection ensuring the classes would be loaded properly https://github.com/eclipse/deeplearning4j/pull/9333 https://github.com/eclipse/deeplearning4j/pull/9350

• Fix minor workspace activation bug: https://github.com/eclipse/deeplearning4j/pull/9341

• Fixed compilation error when running anything more than jdk 8 and NIO buffers: https://github.com/eclipse/deeplearning4j/pull/9351

• Move logback to be a test dependency for some modules: https://github.com/eclipse/deeplearning4j/pull/9362

• Keras model import fixes for GlobalPooling: https://github.com/eclipse/deeplearning4j/pull/9378 https://github.com/eclipse/deeplearning4j/pull/9384

Nd4j

Features and Enhancements

• Add Eigen op as public ensuring easier use when running eigenvalue decomposition https://github.com/eclipse/deeplearning4j/pull/9328

Bug fixes

• Fixes minor issue with choice(..) op https://github.com/eclipse/deeplearning4j/pull/9360 thanks to https://github.com/Romira915

• Minor applyScalar typo fix: https://github.com/eclipse/deeplearning4j/pull/9385

Datavec

Features and Enhancements

Bug fixes

Fixed serialization bug with StringToTimeTransform: https://github.com/eclipse/deeplearning4j/pull/9377 thanks to community member https://github.com/yumg

Python4j

Features and Enhancements

• Made python4j's python path setting more robust by migrating from set path calls to add path calls: https://github.com/eclipse/deeplearning4j/pull/9386

Bug fixes

• Fixes bug with numpy import array jvm crashes: https://github.com/eclipse/deeplearning4j/pull/9348

Samediff

Features and Enhancements

Bug fixes

Fixed inconsistent conventions between SameDiffVariable getArr and getArrForName().. https://github.com/eclipse/deeplearning4j/pull/9357

Highlights - 1.0.0-beta6 Release

• Added support for CUDA 10.2. 1.0.0-beta6 released with CUDA 9.2, 10.0, 10.1 and 10.2 support

• SameDiff optimizations - memory use for inference and training significantly reduced, with some performance improvements also

• Deeplearning4j UI - Play framework replaced with Vertx; deeplearning4j-ui dependency now no longer has Scala dependency or Scala version suffix Link

  • Note: No API changes, only artifact ID change: replace deeplearning4j-ui_2.1x with deeplearning4j-ui

• ND4j namespace operation methods: operations are available through the Nd4j.math, Nd4j.random, Nd4j.bitwise, Nd4j.nn (neural network), for example Nd4j.math.abs(INDArray), Nd4j.random.logNormal etc Link.

  • Note that additional ND4J namespaces API will have additions (new namespaces and methods), and may have some API changes, in the next release

• OpenMP replaced with thread pool c++ parallelism framework; enabled c++ parallelism for platforms without C++-level threading for operations

Deeplearning4J

Deeplearning4J: Features and Enhancements

• DNNL (MKL-DNN) upgraded to version 1.1

• Added causal convolution mode for Convolution1D layer (ConvolutionMode.Causal) and added causal conv1d support for Keras import Link

• Keras import now supports scaled identity weight initialization Link

• Added Mish activation function Link, Link

• BertIterator now has a BertIterator.featurizeSentences(List<String>) method for inference Link, Link

• BertIterator now supports sentence pairs for supervised training Link

• Added Sparse multi-class cross entropy for both Deeplearning4j and Keras import Link, Link

• Deeplearning4j UI: migrated from Play to Vertx for web serving backend, also removing dependency on Scala libraries; no API changes, only artifact ID change - replace deeplearning4j-ui_2.1x with deeplearning4j-ui Link, Link

• Added TimeDistributed wrapper layer Link

Deeplearning4J: Bug Fixes and Optimizations

• KDTree implementation optimized Link

• Deeplearning4j zoo models and datasets hosting location updated Link

• Fixed nIn validation for Deconv2D layer Link

• Fixed an issue with incorrect Deconvolution2d results for Keras import models Link

• Added DNNL/MKLDNN support for batch normalization layer Link, Link

• Fixed various integer casts to avoid overflows for very large arrays (with dimensions or length > Integer.MAX_VALUE) Link

• Fixed an issue with UNet non-pretrained model architecture (last layer kernel size) Link

• Deeplearning4j SameDiff layers now use DL4J workspaces for better performance and reduced memory consumption Link

• Updated broken links in afew error messages Link

• Cleaned up a few unused dependencies in various modules Link

• Cleaned up duplicate SamplingDataSetIterator class Link

• Fixed an issue where ComputationGraph instances with a single input going into multiple embedding layers could throw a NPE Link

• Fixed an issue where loss function weights were not automatically cast to network datatype, resulting in an exception if not already correct type Link

• Shaded Jackson version upgraded from 2.9.9/2.9.9.3 to 2.10.1 Link

• Fixed an issue with KNN where getMostPopulatedClusters actually returned the least populated clusters Link

Deeplearning4j: Transition Guide, 1.0.0-beta5 to 1.0.0-beta6

• Deeplearning4j UI artifact ID has changed: deeplearning4j-ui_2.1x (beta5 and earlier) with deeplearning4j-ui

ND4J and SameDiff

ND4J/SameDiff: Features and Enhancements

• Added suport for CUDA 10.2 Link

• DNNL (MKL-DNN) upgraded to version 1.1 Link

• Added ND4j namespaces to match SameDiff: Nd4j.math, Nd4j.random, Nd4j.bitwise, Nd4j.nn (neural network) Link

• Added SameDiff.calculateGradientsAndOutputs method Link Link

• Additional SameDiff single batch .output method overloads for DataSet/MultiDataSet added Link

• TensorFlow import ops coverage enhanced (significant number of additional ops supported) Link, Link, Link, Link, Link

• PRelu op added Link

• adjust_contrast, igamma and igammac ops added Link

• ND4J/SameDiff: BitCast, CompareAndBitpack, DivideNoNan, DrawBoundingBoxes, FakeQuantWithMinMaxVarsPerChannel ops added Link

• non_max_suppression_overlaps op added Link

• ImagePreProcessingScaler now supports segmentation use cases Link

• concat operation now supports the concatenation axis being specified via the last input array Link

• Added Gamma and Poisson RNG distributions Link

• SameDiff’s use of DeviceLocal for variables/constants etc is now configurable Link

• Uniform distribution op now supports random integer generation, not just random floating point generation Link

• SameDiff: Added simple OpBenchmarkListener for benchmarking purposes Link

• Added the ability to disable platform helpers (DNNL/MKLDNN etc) via Nd4jCPU.Environment.getInstance().allowHelpers(false); and Nd4jCuda.Environment.getInstance().allowHelpers(false); Link

• Added draw_bounding_boxes operation Link

• Added resize_bicubic operation Link

• Added causal padding mode to conv1d operation Link

• DNNL (MKLDNN) is included and enabled by default for non-AVX builds Link

• Added SameDiff ArraySavingListener for debugging purposes Link

ND4J/SameDiff: Bug Fixes and Optimizations

• OpenMP replaced with ThreadPool abstraction, enables parallelism for platforms without OpenMP support Link

• SameDiff memory management overheauled for (in some cases significantlny) reduced memory consumption and improved performance Link, Link

• Switched to Clang instead of gcc for OSX compilation to avoid compiler-related issues Link

• Removed SameDiff.outputs() “best guess” output inference due to being unreliable, in favor of explicit SameDiff.setOutputs(String...) call Link

• Fixed an issue with Nd4j.hstack on 1D arrays Link

• SameDiff no longer allows empty arrays for variables Link

• Fixed an issue with Nadam updater LR schedules not being cloned Link

• Cleaned up IActivation interface Link

• Added new LSTM op implementation with DNNL/MKLDNN support (forward pass only so far) Link

• SameDiff API cleaned up; deprecated methods removed Link

• Switched SameDiff variable initialization to non-lazy, to avoid unexpected behaviour when mixing execution and ND4J RNG seed setting Link

• SameDiff.zero and .one methods now create constants, not vairables Link

• Moved CUDA build version and device logging to Java logging, from c++ stdout to enable disabling logging (via ND4J config or slf4j config) Link

• Added DNNL/MKLDNN support for batch normalization Link

• SameDiff: Fixed an issue where listeners weren’t being called for gradient calculation Link

• Added DNNL/MKLDNN support for deconv2d/3d operations Link

• Fixed an issue with biasadd_bp operation and NHWC data format Link

• Fixed an issue with certain strided slice backprop configurations Link, Link

• Fixed an issue with LogSumExp reduction operation backprop for along dimension case Link, Link

• INDArray.toString() now has correct brackets for rank 1+ scalars to avoid ambiguity Link

• Fixed an issue where some ND4J methods could fail when the library is compiled on Java 9+ but run on Java 8 Link

• Fixed empty array input case for is_strictly_increasing, non_decreasing and non_max_suppression ops Link, Link

• Fixed empty input arrays for legacy ops (transform, scalar, pairwise, broadcast) Link

• CUDA compute capability 3.0 is supported again Link

• Improved performance for Scatter operations (1D case) + index validation Link

• Fixed an issue where SameDiff TrainingConfig serialization would fail if evaluation instances are set Link, Link

• SameDiff execution will now throw an exception when assertion operations in the graph fail Link

• PolyGamma function now returns NaNs when passed double for args requiring integer values Link

• Fixed some issues for pad and mirror_pad ops to ensure they conform with Tensorflow for imported networks Link

• Updated and fixed some issues for TensorFlow graph runner Link

• Improved performance for Reverse operation Link

• Removed/cleanup up unused ND4J list functionality Link

• Fixed reduce bool operation results (such as any, all, IsInf, etc) for empty array inputs Link

ND4J: Transition Guide, 1.0.0-beta5 to 1.0.0-beta6

• SameDiff.outputs() now requires user to call SameDiff.setOutputs(String...) first; previous “best guess” output inference was unreliable Link

• SameDiff.zero and .one methods now create constants, not vairables Link

DataVec

DataVec: Bug Fixes and Optimizations

• NativeImageLoader now checks for empty input streams and throws an exception instead of crashing Link

• NDArrayScalarOpTransform now supports modulus operator Link

RL4J

RL4J: Features and Enhancements

• Added AsyncTrainingListener Link

• Replaced multiple uses of java.util.Random with ND4J Random Link

• Added Observable and LegacyMDPWrapper Link

RL4J: Bug Fixes and Optimizations

• Refactored RL4J video recording to separate VideoRecorder class Link

• Fixed an issue with target for DQN Link, Link

• Refactoring for DQN and double DQN for improved maintainability Link

• Internal refactoring and various bug fixes Link

PyDataVec

PyDataVec Features and Enhancements

• PyDataVec TransformProcess now supports non-inplace operations Link

PyDataVec Bug Fixes and Optimizations

• Fixed various issues with PyDataVec Link

• Fixed an issue with data locality that could cause incorrect results under some circumstances when running on CUDA Link

Highlights

Significant changes for views being used within nd4j and samediff via the create_view op allow for users to directly leverage views in combination with graph operations reducing allocations and allowing for increased performance.

Cuda 11.4 and 11.6 added. Add nd4j-cuda-11.4 and nd4j-cuda-11.6 to your dependencies.

New Onnx ops support added (https://github.com/eclipse/deeplearning4j/pull/9663):

1. AliasWithName

2. CumSum

3. GenerateProposals

4. Loop

5. ResizeNearest

6. SequenceAt

7. SequenceConstruct

8. SequenceEmpty

9. SequenceErase

10. SequenceInsert

11. SequenceLength

3D Convolution Pooling serialization fixes for keras

Fix up training gradient propagation: https://github.com/eclipse/deeplearning4j/pull/9664

Significant changes for migration of NEC Aurora backend to VEDA:

1. https://github.com/eclipse/deeplearning4j/pull/9659

2. https://github.com/eclipse/deeplearning4j/pull/9678

3. https://github.com/eclipse/deeplearning4j/pull/9726

4. https://github.com/eclipse/deeplearning4j/pull/9723

5. https://github.com/eclipse/deeplearning4j/pull/9705

More fixes for mac osx arm64, please feel free to report an issue if anything comes up: https://github.com/eclipse/deeplearning4j/pull/9731

Upgrade protobuf to 3.21.2

Migrate project to java 11 and newer: https://github.com/eclipse/deeplearning4j/pull/9738

Enhanced performance for python4j: https://github.com/eclipse/deeplearning4j/pull/9688 (more to be added) thanks to https://github.com/subes for the suggestions for the improvements.

Removes spark 2 support.

Nd4j/Samdiff/Libnd4j

Features and Enhancements

1. Add label serialization for multi

2. Allow using fast path for results as well: https://github.com/eclipse/deeplearning4j/pull/9729

3. Fix up concurrency issue in ticketing framework: https://github.com/eclipse/deeplearning4j/pull/9721 Thanks PZA from: https://github.com/wehowsky

4. Indexing changes: https://github.com/eclipse/deeplearning4j/pull/9690

5. Add graalvm support for onnx/tensorflow import framework annotations: https://github.com/eclipse/deeplearning4j/pull/9718

6. Add more op definitions for sd.linalg: https://github.com/eclipse/deeplearning4j/pull/9681

7. Allow more tolerance for samediff serialization for model import issues: https://github.com/eclipse/deeplearning4j/pull/9745

Bug Fixes

1. Enable Gather Gradient: https://github.com/eclipse/deeplearning4j/pull/9674

2. Fix up tri, samediff training: https://github.com/eclipse/deeplearning4j/pull/9672

3. Fix up SVD: https://github.com/eclipse/deeplearning4j/pull/9673

4. Fix up tri + cuda: https://github.com/eclipse/deeplearning4j/pull/9730

5. Fix up null checks with samediff variable getArr(): https://github.com/eclipse/deeplearning4j/pull/9701

6. Fix up buffer overflow where databufferand length do not match: https://github.com/eclipse/deeplearning4j/pull/9713

Deeplearning4j

Features and Enhancements

1. Add label saving for computation graph, multiayernetwork; https://github.com/eclipse/deeplearning4j/pull/9672

Bug Fixes

1. Fix confusion matrix count increments: https://github.com/eclipse/deeplearning4j/pull/9553

2. Fix Conv3D data format serialization: https://github.com/eclipse/deeplearning4j/pull/9648

3. Add keras import aliases for more recent versions: https://github.com/eclipse/deeplearning4j/pull/9704

4. Misc bug fixes for views in deeplearning4j-nn: https://github.com/eclipse/deeplearning4j/pull/9689

5. Fix up wordvectorserializer: https://github.com/eclipse/deeplearning4j/pull/9728 Thanks to https://github.com/j-d-o!

Datavec

Omnihub

Python4j

1. Significant performance improvements for python4j: https://github.com/eclipse/deeplearning4j/pull/9688

Highlights - 1.0.0-beta5 Release

• Added model server - remote inference of SameDiff and DL4J models using JSON or (optionally) binary serialization

  • Server: See JsonModelServer

  • Client: See JsonRemoteInference

  • Tests/examples: See Link and Link

• Added Scala 2.12 support, dropped Scala 2.10 support. Modules with Scala dependencies are now released with Scala 2.11 and 2.12 versions

• Apache Spark 1.x support dropped (now only Spark 2.x is supported). Note: Spark version suffix dropped: For upgrading: 1.0.0-beta4_spark2 -> 1.0.0-beta5

• Added FastText support to deeplearning4j-nlp

• CUDA support for all ND4J/SameDiff Operations

  • In 1.0.0-beta4, some operations were CPU only. Now, all operations have full CUDA support

• Added support for new data types in ND4J (and DL4J/SameDiff): BFLOAT16, UINT16, UINT32, UINT64

• ND4J: Implicit broadcasting support added to INDArray (already present in SameDiff - for example shape [3,1]+[3,2]=[3,2])

• CUDA 9.2, 10.0 and 10.1-Update2 still supported

  • NOTE: For CUDA 10.1, CUDA 10.1 update 2 is recommended. CUDA 10.1 and 10.1 Update 1 will still run, but rare internal cuBLAS issues may be encountered in heavily multi-threaded code on some systems

• Dependency upgrades: Jackson (2.5.1 to 2.9.9/2.9.9.3), Commons Compress (1.16.1 to 1.18), Play Framework (2.4.8 to 2.7.3), Guava: (20.0 to 28.0-jre, and shaded to avoid dependency clashes)

• CUDA: now host (RAM) buffers are only allocated when required (previously: host buffers were always allocated), in addition to device (GPU) buffer

Deeplearning4J

Deeplearning4J: Features and Enhancements

• Added FastText - inference and training, including OOV (out of vocabulary) support (Link)

• Scala 2.12 support added, Scala 2.10 support dropped (Link)

• Added model server (DL4J and SameDiff models, JSON and binary communication) - JsonModelServer, JsonRemoteInference, Link, Link

• Added saved model format validation utilities - DL4JModelValidator, DL4JKerasModelValidator (Link)

• Added LabelLastTimeStepPreProcessor (Link)

• BertIterator: added option to prepend token to the output (such as [cls] expected by some models) (Link)

• Added trace level logging to MultiLayerNetwork and ComputationGraph assist with debugging certain issues (Link)

• Upsampling3D: Added NDHWC support (Link)

• MergeVertex now supports broadcasting (Link)

• LSTM and Dropout will now fall back on built-in implementations if an exception is encountered from cuDNN (same as Subsampling/ConvolutionLayer) (Link)

• Improved JavaDoc and cleanup up API for WordVectorSerializer (Link, Link)

Deeplearning4J: Bug Fixes and Optimizations

• Updated deeplearning4j-ui theme (Link)

• Fixed an issue with MergeVertex and CNN3D activations (Link)

• Fixed typo in Yolo2OutputLayer builder/configuration method name (Link)

• Improved ComputationGraph builder InputType validation (Link)

• Removed dl4j-spark-ml module until it can be properly maintained (Link)

• Fixed an issue with BertWordPieceTokenizerFactory and bad character encoding (Link)

• Fixed an issue with LearnedSelfAttentionLayer and variable minibatch size (Link, Link)

• Fixed issue with SharedTrainingMaster controller address when set from environment variable (Link)

• Fixed issue with SameDiffOutputLayer initialization under some circumstances (Link)

• https is now used by default for data and zoo model downloads (Link, Link)

• Fixed an issue where UI WebJars dependencies would check for updates on every single build (Link, Link)

• Fixed issue where Upsampling layer memory report could produce an OOM exception (Link)

• Improved UX/validation for RecordReaderDataSetIterator (Link)

• Fixed an issue where EmbeddingSequenceLayer would not check mask array datatype (Link)

• Improved validation when initializing networks with a non rank-2 (shape [1, numParams]) array (Link)

• Fixed a DataType issue for BertIterator (Link)

• Fixed Word2Vec model backward compatibilty (beta3 and earlier models now loadable again) Link

• Fixed issue where some Keras import models could fail with Could not read abnormally long HDF5 attribute (Link)

• Added validation for RnnOutputLayer - feature/label array lengths (Link)

• Fixed an issue where SameDiffOutputLayer would not support variable minibatch size (Link)

• Fixed DL4J SameDiff layer mask support (Link)

• DL4J UI: Fixed an issue where tab switching did not work when visualizing saved/stored data (Link, Link)

• DL4J UI: Fixed a rare UI threading issue (Link)

• Fixed a Keras import issue with JSON format change (Link)

• Fixed a Keras import issue where updater learning rate schedule could be imported incorrectly (Link)

• Fixed an issue with CnnSentenceDataSetIterator when using UnknownWordHandling.UseUnknownVector (Link, Link)

• Fixes and optimizations to DL4J SameDiff layers (Link)

• MultiLayerNetwork/ComputationGraph will now log the original exception if a second exception occurs during workspace closing, instead of swallowing it (inference/fit operation try/finally blocks) (Link)

• Upgraded dependencies: Jackson (2.5.1 to 2.9.9/2.9.9.3), Commons Compress (1.16.1 to 1.18), Play Framework (2.4.8 to 2.7.3), Guava: (20.0 to 28.0-jre, shaded to avoid dependency clashes) (Link)

• Logging framework can now be configured for DL4J UI (due to Play framework dependency upgrade) (Link)

• Reduced amount of garbage produced by MnistDataFetcher (impacts MNIST and EMNIST DataSetIterators) (Link)

• Activation function backpropagation has been optimized for many activation functions (Link, Link)

Deeplearning4j: Transition Guide, 1.0.0-beta4 to 1.0.0-beta5

• DL4J AsyncDataSetIterator and AsyncMultiDataSetIterator moved to ND4J, use org.nd4j.linalg.dataset.Async(Multi)DataSetIterator instead

• Saved models with custom layers from 1.0.0-alpha and before can no longer be loaded. Workaround: load in 1.0.0-beta4, and re-save the model (Link). Models without custom layers can still be loaded back to 0.5.0

• Apache Spark 1.x support dropped (now only Spark 2.x is supported). Note: Spark version suffix dropped: For upgrading, change versions as follows: 1.0.0-beta4_spark2 -> 1.0.0-beta5

• Scala 2.10 dropped, Scala 2.12 added (for modules with Scala dependencies)

Deeplearning4j: 1.0.0-beta5 Known Issues

• dl4j-spark_2.11 and _2.12 dependencies incorrectly pull in datavec-spark_2.11/2.12 version 1.0.0-SNAPSHOT. Workaround: control version using dependency management as per here or here

• Some layers (such as LSTM) may run slower on 1.0.0-beta5 than 1.0.0-beta4 on CUDA when not using cuDNN, due to added synchronization. This synchronization will be removed in the next release after 1.0.0-beta5

• CUDA 10.1: Rare internal cuBLAS issues may be encountered in heavily multi-threaded code on some systems, when running CUDA 10.1 Update 1 (and maybe 10.1). CUDA 10.1 update 2 is recommended.

ND4J and SameDiff

ND4J/SameDiff: Features and Enhancements

• Added new data types: BFLOAT16, UINT16, UINT32, UINT64 (Link)

• CUDA support for all operations without CUDA implementations (Link, Link, Link, Link, Link)

• Added model server (DL4J and SameDiff models, JSON and binary communication) - JsonModelServer, JsonRemoteInference, Link, Link

• Added support for empty arrays with zeros in shape, for compatibility with TensorFlow import (Link)

• CUDA: now host (RAM) buffers are only allocated when required (previously: host buffers were always allocated), in addition to device (GPU) buffer

• Improved SameDiff training API - added "in line" test set evaluation, returning History object with loss curve, etc (Link)

• Added saved model format validation utilities - Nd4jValidator, Nd4jCommonValidator (Link)

• Added SameDiff ScoreListener (equivalent to DL4J ScoreIterationListener/PerformanceListener) (Link, Link)

• Added SameDiff.convertDataTypes method, for variable dtype conversion (Link)

• Added crop and resize op (Link)

• DL4J AsyncDataSetIterator and AsyncMultiDataSetIterator moved to ND4J Link

• Added basic/MVP SameDiff UI listener (Link)

• Added SameDiff CheckpointListener (Link, Link)

• Added SameDiff name scopes (Link)

• SameDiff: Updater state and training configuration is now written to FlatBuffers format (Link)

• Added c++ benchmark suite callable from Java - call using Nd4j.getExecutioner().runLightBenchmarkSuit() and Nd4j.getExecutioner().runFullBenchmarkSuit() (Link)

• Added SameDiff.save/load methods with InputStream/OutputStream arguments (Link, Link)

• Added axis configuraiton for evaluation instances (Evaluation, RegressionEvaluation, ROC, etc - getAxis and setAxis methods) to allow different data formats (NCHW vs. NHWC for CNNs, for example) (Link)

• SameDiff: Added support to convert constants to placeholders, via SDVariable.convertToConstant() method (Link)

• SameDiff: Added GradCheckUtil.checkActivationGradients method to check activation gradients for SameDiff instance (not just parameter gradients as in existing gradient check methods) (Link)

• Added CheckNumerics op (Link)

• Added FakeQuantWithMinMaxArgs and FakeQuantWithMinMaxVars ops (Link)

• Added INDArray reduction methods with "keep dimensions" option - for example, INDArray.mean(boloean, int... dimension) (Link)

• Added Nd4j SystemInfo class - SystemInfo.getSystemInfo, .writeSystemInfo(File) to aid with debugging issues (Link, Link)

• Added INDArray.toString(NDArrayStrings options), toStringFull() and toString overloads for easier control of array printing (Link)

• Added HashCode op, INDArray.hashCode() (Link)

• SameDiff: added whileLoop, ifCond methods for loops/conditional ops (Link)

• Cleaned up some infrequently used Nd4j methods (Link, Link, Link, Link)

• Added bitwise integer operations: left/right bit shift, left/right cyclical bit shift, bitwise Hamming distance (Link, Link, Link, Link, Link)

• deeplearning4j-nlp: renamed AggregatingSentencePreProcessor to sentencePreProcessor method (Link)

• Upgraded (and shaded) Protobuf version - 3.5.1 to 3.8.0 (Link)

• Switched to c=style error handling for libnd4j native operations (Link)

• Renamed FlatBuffers enum org.nd4j.graph.DataType to org.nd4j.graph.DType to avoid users importing incorrect type when using Nd4j methods (Link, Link)

• Added SameDiff.bitwise namespace for bitwise ops (Link, Link)

ND4J/SameDiff: Bug Fixes and Optimizations

• Updated to JavaCPP/JavaCV 1.5.1-1 (Link)

• SameDiff: Placeholders must now only be provided if required to calculate the requested variables (Link)

• SameDiff: Fixed an issue with duplicate variable name validation (Link)

• SameDiff: Fixed an issue with SDVariable.getArr for scalars (Link)

• Added delayed mode to DeviceLocalNDArray (don't replicate to device until needed) (Link)

• ND4J: Fixed an issue with writing 0d (scalar) NDArrays in numpy .npy format (Link)

• Fixed an issue with Pad operation for some constant cases (Link)

• Fixed some issues with strided_slice operation (Link, Link, Link)

• SameDiff: Fixed issue with DataType inference for some ops using ND4J default datatype (Link)

• INDArray.castTo(DataType) is now a no-op when array is already the correct type (Link)

• SameDiff: Fixed an issue with training mixed precision networks (Link)

• Fixed an issue where Evaluation class was incorrectly reporting macro-averaged precision for binary case (Link)

• Removed trainableParams config/field from SameDiff TrainingConfig (no longer required) (Link)

• Improvements and cleanup to ND4J Javadoc (Link, Link, Link, Link)

• Fixed an issue with Cholesky Lapack op on CUDA (Link, Link)

• Fixed an issue where [1,N] and [N,1] arrays were not considered a matrix (rank 2 array) according to INDArray.isMatrix() (Link)

• Fixed RegressionEvaluation for 4D arrays (CNNs / segmentation) (Link, Link)

• Fixed issue with INDArray.median(int... dimension) (Link)

• Fixed NPE that could occur when executing gather operation backprop (Link)

• Fixed issue with LogSumExp operation Java/C++ mapping (Link)

• Added header validation when reading Numpy .npy files, to ensure file is valid (Link)

• Fixed a possible issue with reading Numpy .npy files on CUDA (Link)

• Fixed an issue when reading Numpy .npy boolean files (Link)

• Various fixes for TensorFlow import (Link)

• Fixed an issue with a small number of Nd4j.create methods not creating arrays corresponding to the java primitive (Link)

• Improved shape validation for some Nd4j.create methods (Link)

• Cleaned up unmaintained Nd4j.createSparse methods (Link)

• Fixed a CUDA issue for CUDA GPUs with CC 3.0 (Link)

• Fixed some possible integer overflows in c++ code (Link)

• Removed deprecated methods: Nd4j.trueScalar and Nd4j.trueVector (Link, Link)

• Fixed an issue where some JVMs could warn about "Illegal reflective access" due to a (now removed) SameDiff dependency (Link)

• SDVariable now no longer extends DifferentialFunction (Link)

• Moved numerous operation calculateOutputShape instances from Java to C++ (Link)

• Fixed an issue where maxpool2d_bp could throw an exception when NaN values are present (Link)

• Fixed an issue with concatenation of empty shapes (with zeros) (Link)

• Removed INDArray.javaTensorAlongDimension (Link)

• LayerNorm operation now properly supports axis arg, NCHW format data (Link)

• libnd4j: cuBLAS hgemm (FP16 gemm) wil only be called for devices with compute capability >= 5.3 due to cuBLAS limitations (Link)

• Nd4j.readNumpy optimized (Link)

• Added configurable alpha parameter to ELU and lrelu_bp operations in c++ (Link)

• Cleaned up SameDiff SDCNN/SDRNN (SameDiff.cnn, .rnn) API/methods (Link, Link)

ND4J: Transition Guide, 1.0.0-beta4 to 1.0.0-beta5

• OldAddOp, OldSubOp, etc removed: Replace with AddOp, SubOp, etc

• Nd4j.trueScalar and trueVector removed; use Nd4j.scalar and Nd4j.createFromArray methods

• INDArray.javaTensorAlongDimension removed; use INDArray.tensorAlongDimension instead

• INDArray.lengthLong() removed; use INDArray.length() instead

ND4J: 1.0.0-beta5 Known Issues

• nd4j-native on some OSX systems can fail with Symbol not found: ___emutls_get_address - See this link

• SBT 1.3.0 can fail with an Illegal character in path error; SBT 1.2.8 is OK. This is an SBT issue, not an ND4J issue. See this link for details

DataVec

DataVec: Features and Enhancements

• ImageRecordReader: Support for 16-bit TIFF added (Link)

• Added SequenceTrimToLengthTransform (Link)

DataVec: Bug Fixes and Optimizations

• Fixed an issue with AnalyzeSpark and String columns (Link)

• Fixed an issue with URL scheme detection in NumberedFileInputScheme (Link)

• Fixed an issue with RandomPathFilter sampling being biased (Link, Link)

RL4J

RL4J: Features and Enhancements

• API cleanup and refactoring (Link, Link, Link, Link)

RL4J: Bug Fixes and Optimizations

• Fixed issue with compression for HistoryProcessor (Link)

Arbiter

Bug Fixes and Optimizations

• Updated EvaluationScoreFunction to use ND4J Evaluation class metrics (Link)

• Fixed incorrect search size in GridSearchCandidateGenerator (Link)

Arbiter: Known Issues

• The Jackson version upgrade necessitated a change to how generic object serialization was performed; Arbiter JSON data stored in 1.0.0-beta4 or earlier format may not be readable in 1.0.0-beta5 (Link)

ND4S

ND4S Features and Enhancements

• Added full data type support to ND4S as per ND4J (Link)

• Added syntactic sugar for SameDiff (implicits, operator overloads) (Link)

Version 1.0.0-beta7

Read the announcement at https://blog.konduit.ai/2020/05/14/deeplearning4j-1-0-0-beta7-released/ for the highlights of this release.

Deeplearning4j

Features and Enhancements

• Added Keras model import support for tf.keras models Link, Link

  • Full inference and training support is available for ops/layers in the tf.keras namespace; inference only for general Tensorflow operations outside of the tf.keras namespace

  • Note also improvements to Keras import for reshape, permute, etc operations due to NHWC and NWC support in DL4J

• DL4J now supports NHWC (channels last) data format for all CNN 2D layers, in addition to NCHW Link

• DL4J now supports NWC (channels last - [minibatch, sequence_length, size]) for all RNN and CNN 1D layers, in addition to NCW Link

• Added Deconvolution3D layer Link

• Keras import: added ReLU, ELU and Softmax advanced activation layers Link and Swish activation function Link

• Added DL4J SameDiffLoss class (for easily-defined DL4J ILossFunction's via SameDiff) Link

• Useful exceptions are now thrown when attempting to perform unsupported operations on FastText Link

• Added MultiLayerNetwork.evaluate(MultiDataSetIterator) and .evaluateRegression(MultiDataSetIterator) methods Link, Link

Bug Fixes and Optimizations

• Updaters (Adam, AdaGrad, etc) optimized via C++ operations (significant training performance boost) for DL4J and SameDiff Link, Link

• Some packages relocated to avoid split packages (that can be a problem for OSGi and Java 9 modules) Link

  • Note: this is a breaking change for some class packages/imports. See this link for details on exact package changes

• Deeplearning4j UI: Webjars versions locked down using dependency management to avoid check on each build Link

• Added MKLDNN (DNNL/OneDNN) support for depthwise_conv2d operation for DL4J and SameDiff Link

• Refactored/merged modules dl4j-perf and dl4j-util into deeplearning4j-core Link

• Fixed an issue with BertWordPieceTokenizer - potential StackOverflowError with certain inputs Link

• Fixed an issue with GlobalPooling layer with masks of different datatype to the activations datatype Link

• Fixed an issue with DL4JModelValidator for ComputationGraph Link

• Fixed an issue where SameDiff layers in DL4J could throw an exception when used with transfer learning Link

• Weight initialization for EmbeddingLayer and EmbeddingSequenceLayer now no longer depend on the vocabulary size (only the vector size) Link

• Fixed an issue with Keras import with bidirectional layers + preprocessors Link

• DL4J UI: added redirect from /train to /train/overview Link

• Fixed an issue where RecordReaderDataSetIterator builder collectMetaData configuration was not being applied Link

• Fixed an issue where MultiLayerNetwork evaluation was not passing metadata to the IEvaluation instances during evaluation Link, Link

• Fixed an issue with Spark training SharedTrainingMaster when training with a ComputationGraph and MultiDataSets Link

• Assorted fixes for edge cases for DL4J Keras import Link

• deelpearning4j-nlp-korean will no longer be released for Scala 2.12 due to required dependency only having Scala 2.11 version avairable Link

• Fix for ConvolutionalIterationListener for ComputationGraph Link

• Fixed an issue where dataset and model zoo downloads could get stuck if the server fails to send any data (now: timeout + retry) Link

• DL4J ModelSerializer no longer writes temporary files when restoring models from InputStream Link

• Fixes issues with UIServer multi session mode, and potential shutdown race condition Link

• Fixed an issue where TfidfVectorizer.vectorize() could throw a NPE when fit from LabelAwareIterator Link

ND4J/SameDiff:

Features and Enhancements

• SameDiff multi-threaded inference enhanced (and fixed) - a single SameDiff instance can now be used for inference safely and efficiently from multiple threads Link Link

• cuDNN support added to SameDiff (automatically enabled for nd4j-cuda-10.x backend) Link

• Added ND4J namespaces: Nd4j.cnn, Nd4j.rnn, Nd4j.image Link

• Added new Image operations namespace operations:

  • rgbToHsv, hsvToRgb Link

  • rgbToYiq, yiqToRgb, rgbToYuv, yuvToRgb Link

  • imageResize Link

• Added new Random operations namespace operations:

  • gamma, poisson, shuffle Link

• Added new Math namespace operations:

  • clipByAvgNorm, embeddingLookup Link

  • mergeMaxIndex Link

• Added new NN namespace operations:

  • cReLU Link

• Added new CNN namespace operations:

  • upsampling3d Link

• Added new linalg operations namespace

  • triangular_solve Link

  • tri operation Link

  • triu operation Link

• Added new RNN operation namespace operations:

  • lstmLayer (note old lstmLayer method renamed to lstmBlock) Link

  • gru Link

• Added new Loss operations namespace - Nd4j.loss Link

• Mapped operations for Tensorflow import:

  • HSVToRGB, RGBToHSV, Igamma, Igammac, RandomGamma, RandomPoisson, RandomPoissonV2, RandomShuffle Link

• Added SameDiff ProfilingListener - writes op performance profiles in Chrome profiler format (load in chrome://tracing/) Link Link

• Added SameDiff ProfileAnalyzer tool to compare profiles output from ProfilingListener (or Tensorflow) Link Link

• SameDiff listener API: added frame and iteration information for listener methods Link Link

• Added (non-backend-specific) method of accessing Nd4j environment: Nd4j.getEnvironment() method (environment info and low-level configuration options) Link Link

• Improved memory limits/configuration support for libnd4j (c++) Link

• Added pairwise (broadcastable) power backprop operation Link

• Updated JavaCPP presets MKL version to 2020.0 from 2019.5 Link

• Added DynamicCustomOp dargs - datatype arguments Link Link

  • Output datatype configuration for Range op Link, SequenceOp Link, ConfusionMatrix Link

• Added tensormmul_bp op Link

• OpenBLAS version upgraded to 0.3.8 Link

• libnd4j (c++ codebase underlying DL4J, ND4J and SameDiff) refactored to be more easily embeddable in other C++ projects Link

• ImagePreProcessingScaler now supports preprocessing of labels (for segmentation) Link

• Additional datatypes now supported for nd4j-tensorflow TensorflowConversion Link

• SameDiff operation namespaces (sd.math, sd.image, etc) are now code generated to ensure SameDiff and ND4J namespaces are identical (all operations included, same API) Link

• Added ND4J ArchiveUtils.unzipFileTo(String, String, boolean logFiles) overload to enable/disable extracted file path logging Link

• Added weight format configuration for following operations: conv1D, conv2D, conv3D, deconv2d, deconv3d, depthwiseConv2d, pointwiseConv2d, sconv2d Link

• Added backprop operation implementations for mergemax, mergeadd, mergeavg operations Link

• MKL version upgraded to 2020.0 2020.1; OpenCV upgraded from 4.2.0 to 4.3.0 Link

• SameDiff: DifferentialFunctionFactory class removed in favor of namespace methods (sd.math, sd.linalg, etc) Link

• Added lstmLayer_bp operation Link

• Added gru_bp operation Link

• linspace operation can now use both targs and arrays for start/end/size arguments Link

• Assorted dependency updates - OpenBLAS (0.3.9), OpenCV (4.3.0), Leptonica (1.79.0) Link

• Upgraded assorted dependency versions: javax.activation:activation (1.1 -> 1.1.1), stream analytics (2.7.0->2.9.8), Apache Spark (2.4.3->2.4.5), Jackson databind (2.10.1 -> 2.10.3), Vertx (3.8.3 -> 3.9.0) Link

• Added nd4j-common-tests ResourceUtils.listClassPathfiles method Link

Bug Fixes and Optimizations

• Updaters (Adam, AdaGrad, etc) optimized via C++ operations (significant training performance boost) for DL4J and SameDiff Link, Link

• SameDiff - added CuDNN support Link

• Some packages relocated to avoid split packages (that can be a problem for OSGi and Java 9 modules) Link

  • Note: this is a breaking change for some class packages/imports. See this link for details on exact package changes

• Fixed some issues with Tensorflow import of FusedBatchNorm operation Link

• Fixed an issue where the Roll operation did not match Tensorflow operation Link Link

• Fixed an issue where ArchiveUtils could fail to create the top level destination directory when it does not exist Link

• Fixed an issue where resize_bicubic operation did not match Tensorflow for some configuration values Link Link

• Pad operation now supports long/int64 values for padding array Link Link

• Fixed an issue where hashcode operation shape function wasn't always returning int64/long dtype Link

• Fixed an issue with reshape operation on empty arrays with -1s Link Link

• Improved performance on CUDA for concat operation Link and CPU/GPU Link

• Improved performance for bias_add operation

  • On CPU for NHWC case Link

  • Generally Link

  • On CUDA for 2D case Link

• Added MKLDNN (DNNL/OneDNN) support for depthwise_conv2d operation for DL4J and SameDiff Link

• Fixed a small SameDiff execution issue for switch operation where the predicate is a constant Link

• Fixed an issue with batchnorm operation when input arrays have unusual strides Link

• Merged nd4j-buffer, nd4j-content modules into nd4j-api Link

• Deleted deprecated nd4j-jackson module (remaining functionality available in nd4j-api) Link

• Deleted unused/unmaintained nd4j-camel and nd4j-gson modules Link

• Optimization for legacy random ops Link

• Optimization for broadcast operations Link, Link, Link, Link, Link

• Performance optimization for multiple operations: softmax, squeeze, expand_dims, tanh Link

• Optimization for transpose/permute operations Link

• Performance enhancement: MKLDNN matmul used for some mmul operation cases Link

• Optimization for gather operation on CPU Link

• Optimization for stack/unstack operations on CPU Link

• Optimization for split operation (CPU and CUDA) Link Link

• ND4J initialization no longer logs number of OpenMP BLAS threads for CUDA Link

• Optimization: Fixed issues with auto-vectorization on multple CPU operations Link

• Optimization for reshape operation Link, Link

• Fixed an issue where INDArray.hashCode() could cause an exception on some datatypes Link

• Optimization for CPU: MKLDNN is now used for softmax, tanh, softmax_bp and tanh_bp operations Link, Link, Link, Link

• Fixed random_exponential operation Link

• Improved performance on C++ SameDiff graph execution via reduced array zeroing where safe to do so Link

• Improved C++ indexing implementation impacting CPU performance on some operations Link

• Fixed an issue where Split operation could have incorrect output shapes for empty arrays Link

• Fixed some issues with SameDiff.equals method Link

• Fixed an issue with reshape operation output shape on empty arrays Link, Link

• Nd4j.gemm now uses Mmul operation internally to avoid potential threading issues with direct BLAS calls on CUDA Link

• Fixed an edge case issue with percentile operation link

• Fixed an edge case issue for cusolved (CUDA) in libnd4j Link

• Fixed an issue with error formatting for segment operations for incorrect lengths Link

• Fixed an issue where ND4J workspaces were not guaranteed to be unique Link

• Fixed some operation implementations when operating on views (Batch/Space to Space/Batch/Depth; batchnorm_bp) Link

• Fixed an issue where exponential distribution random number generation operation could produce infinities extremely rarely (~1 in 10^9 values) Link

• Fixed an issue with long file paths for memory mapped workspaces on Windows Link

• Memory for memory mapped workspaces are now deallocated immediately when workspace is destroyed, instead of waiting for GC to free memory Link

• Fall-back to other BLAS implementation for cases where MKLDNN GEMM implementation is slow Link

• Set nd4j-native source/target to Java 7 Link, Link

DataVec

Features and Enhancements

• datavec-python: added zero-copy support for bytes/byte buffers Link

• datavec-python: Python exceptions are now thrown as Java exceptions Link

• datavec-python: Added support for additional NumPy datatypes Link

• datavec-python: Python version upgraded from 3.7.6 to 3.7.7 Link

Bug Fixes and Optimizations

• Deleted not properly maintained modules: datavec-camel, datavec-perf Link

• Fixed missing BOOL datatype support for arrow conversion functionality Link

• Assorted fixes for datavec-python Link Link, Link

• Fixed an issue with LineRecordReader where initialization was performed unnecessarily (adding performance overhead) Link

RL4J

Features and Enhancements

• Refactoring to decouple configuration and learning methods from their implementations Link

• Added builder patterns for all configuration classes Link

Arbiter

Bug Fixes and Optimizations

• Fixes an issue with GridSearchCandidateGenerator not working correctly for some cases Link, Link

Eclipse DeepLearning4J

Eclipse Deeplearning4j is a suite of tools for running deep learning on the JVM. It's the only framework that allows you to train models from java while interoperating with the python ecosystem through a mix of python execution via our cpython bindings, model import support, and interop of other runtimes such as tensorflow-java and onnxruntime.

Consider going to our Quickstart for an overview of where to get started. If you have dependency issues please use our Required Dependencies guide.

The use cases include importing and retraining models (Pytorch, Tensorflow, Keras) models and deploying in JVM Micro service environments, mobile devices, IoT, and Apache Spark. It is a great compliment to your python environment for running models built in python, deployed to or packaged for other environments.

Deeplearning4j has several submodules including:

1. Samediff: a tensorflow/pytorch like framework for execution of complex graphs. This framework is lower level, but very flexible. It's also the base api for running onnx and tensorflow graphs.

2. Nd4j: numpy ++ for java. Contains a mix of numpy operations and tensorflow/pytorch operations.

3. Libnd4j: A lightweight, standalone c++ library enable math code to run on different devices. Optimizable for running on a wide variety of devices.

4. Python4j: A python script execution framework easing deployment of python scripts in to production.

5. Apache Spark Integration: An integration with the Apache Spark framework enabling execution of deep learning pipelines on spark

6. Datavec: A data transformation library converting raw input data to tensors suitable for running neural networks on.

How to use this website

This website follows the divio framework layout. This website has several sections of documentation following this layout. Below is an overview of the sections of the site:

1. Multi project contains all cross project documentation such as end to end training and other whole project related documentation. This should be the default entry point for those getting started.

2. Deeplearning4j contains all of the documentation related to the core deeplearning4j apis such as the multi layer network and the computation graph. Consider this the high level framework for building neural networks. If you would like something lower level like tensorflow or pytorch, consider using samediff

3. Samediff contains all the documentation related to the samediff submodule of ND4j. Samediff is a lower level api for building neural networks similar to pytorch or tensorflow with built in automatic differentiation.

4. Datavec contains all the documentation related to our data transformation library datavec.

5. Python4j contains all the documentation related to our cpython execution framework python4j.

6. Libnd4j contains all the documentation related to our underlying C++ framework libnd4j.

7. Apache Spark contains all of the documentation related to our Apache Spark integration.

8. Concepts/Theory contains all of the documentation related to general mathematical or computer science theory needed to understand various aspects of the framework.

Open Source

The libraries are completely open-source, Apache 2.0 under open governance at the Eclipse foundation. The Eclipse Deeplearning4j project welcomes all contributions. See our community and our Contribution guide to get involved.

JVM/Python/C++

Deeplearning4j can either be a compliment to your existing workflows in python and c++ or a standalone library for you to build and deploy models. Use what components you find useful.

Highlights

In light of the coming 1.0, the project has decided to cut a number of modules before the final release. These modules have not had many users in the past and have created confusion for many users just trying to use a few simple apis. Many of these modules have not been maintained.

There will likely be 1 or 2 more milestone releases before the final 1.0. These should be considered checkpoints.

These modules include:

1. Arbiter

2. Jumpy

3. Datavec modules for video, audio, audio, sound. The computer vision datavec module

   will continue to be available.

4. Tokenizers: The tokenizers for chinese, japanese, korean were imported from other frameworks

   and not really updated.

5. Scalnet, Nd4s: We removed the scala modules due to the small user base. We welcome 3rd party enhancements

   to the framework for syntatic sugar such as kotlin and scala. The framework's focus will be on providing

   the underlying technology rather than the defacto interfaces. If there is interest in something higher level, please discuss it on community forums

ARM support: We have included armcompute modules for core convolution routines. These routines can be found here

TVM: We now support running TVM modules. Docs coming soon.

We've updated our shaded modules to newer versions to mitigate security risks. These modules include: 1. jackson 2. guava

Cuda 11: We've upgraded dl4j and associated modules to support cuda 11 and 11.2.

A more modular model import framework supporting tensorflow and onnx: 1. Model mapping procedures loadable as protobuf 2. Defining custom rules for import to work around unsupported or custom layers/operations 3. Op descriptor for all operations in nd4j

This will enable users to override model import behavior to run their own custom models. This means, in most circumstances, there will be no need to modify model import core code anymore. Instead, users will be able to provide definitions and custom rules for their graphs.

Users will be expected to convert their models in an external process. This means running standalone conversions for their models. This extends to keras import as well. Sometimes users convert their models in production directly from keras.

The workflow going forward is to ensure that your model is converted ahead of time to avoid performance issues with converting large models.

Removed ppc from nd4j-native-platform and nd4j-cuda-platform. If you need this architecture, please contact us or build from source.

Added more support for avx/mkldnn/cudnn linked acceleration in our c++ library. We now have the ability to distribute more combinations of pre compiled math kernels via different combinations of classifiers. See the ADR here for more details.

The class loader is now overridable. This is useful for OSGI and application server environments.

We've upgraded arrow to 4.0.0 enabling the associated nd4j-arrow and datavec-arrow modules to be used without netty clashes.

Deeplearning4j

Bug fixes

• Improved keras model import support for NWHC as well as NCHW input formats for both rnn and cnn

• Added Adabelief updater

• Added maximum merge for Keras import

• Keras cropping 2d validation fixes

• Lenet input shape fix

• Fix for obtaining the UI port from a property

Nd4j

Features and Enhancements

• CTC Loss: We now have basic support for CTC loss in nd4j. This will enable the import of CTC loss based models for speech recognition as well as OCR.

• tensormmul_bp now run from c++

• Arm compute added for conv2d and pooling operations

• Add IndexUtils containing ravelMultiIndex and unravelIndex methods

• Updates sortCooolIndicesGeneric to take any datatype

• Add TVM runner

Bug fixes

• compare_and_bitpack now functions properly

• Fix null pointer in cuda op executioner

• Fix for samediff array cache removal during training

• Fix for SD_FORBID_HELPERS environment variable

• Fixed cuda bug in summary stats (mean, variance,)

Python4j

Features and Enhancements

Rewritten and more stable python execution. This allows better support for multi threaded environments.

Bug fixes

Contributors: https://github.com/eclipse/deeplearning4j/issues?q=is%3Apr+author%3Amjlorenzo305

• RBM and AutoEncoder key fixes:

  • Ensured visual bias updated and applied during pretraining.

  • RBM HiddenUnit is the activation function for this layer; thus, established derivative calculations for backprop according to respective HiddenUnit.

• RNG performance issues fixed for CUDA backend

• OpenBLAS issues fixed for macOS, powerpc, linux.

• DataVec is back to Java 7 now.

• Multiple minor bugs fixed for ND4J/DL4J

Highlights - 1.0.0-beta4 Release

• DOUBLE: double precision floating point, 64-bit (8 byte)

• FLOAT: single precision floating point, 32-bit (4 byte)

• HALF: half precision floating point, 16-bit (2 byte), "FP16"

• LONG: long signed integer, 64 bit (8 byte)

• INT: signed integer, 32 bit (4 byte)

• SHORT: signed short integer, 16 bit (2 byte)

• UBYTE: unsigned byte, 8 bit (1 byte), 0 to 255

• BYTE: signed byte, 8 bit (1 byte), -128 to 127

• BOOL: boolean type, (0/1, true/false). Uses ubyte storage for easier op parallelization

• UTF8: String array type, UTF8 format

ND4J Behaviour changes of note:

• When creating an INDArray from a Java primitive array, the INDArray datatype will be determined by the primitive array type (unless a datatype is specified)

  • For example: Nd4j.createFromArray(double[]) -> DOUBLE datatype INDArray

  • Similarly, Nd4j.scalar(1), Nd4j.scalar(1L), Nd4j.scalar(1.0) and Nd4j.scalar(1.0f) will produce INT, LONG, DOUBLE and FLOAT type scalar INDArrays respectively

• Some operations require matched datatypes for operands

  • For example, if x and y are different datatypes, a cast may be required: x.add(y.castTo(x.dataType()))

• Some operations have datatype restrictions: for example, sum on a UTF8 array is not supported, nor is variance on a BOOL array. For some operations on boolean arrays (such as sum), casting to an integer or floating point type first may make sense.

DL4J Behaviour changes of note:

• MultiLayerNetwork/ComputationGraph no longer depend in any way on ND4J global datatype.

  • The datatype of a network (DataType for it's parameters and activations) can be set during construction using NeuralNetConfigutation.Builder().dataType(DataType)

  • Networks can be converted from one type to another (double to float, float to half etc) using MultiLayerNetwork/ComputationGraph.convertDataType(DataType) method

Main new methods:

• Nd4j.create(), zeros(), ones(), linspace(), etc methods with DataType argument

• INDArray.castTo(DataType) method - to convert INDArrays from one datatype to another

• New Nd4j.createFromArray(...) methods for

ND4J/DL4J: CUDA - 10.1 support added, CUDA 9.0 support dropped

CUDA versions supported in 1.0.0-beta4: CUDA 9.2, 10.0, 10.1.

ND4J: Mac/OSX CUDA support dropped

Mac (OSX) CUDA binaries are no longer provided. Linux (x86_64, ppc64le) and Windows (x86_64) CUDA support remains. OSX CPU support (x86_64) is still available.

DL4J/ND4J: MKL-DNN Support Added DL4J (and ND4J conv2d etc ops) now support MKL-DNN by default when running on CPU/native backend. MKL-DNN support is implemented for the following layer types:

• ConvolutionLayer and Convolution1DLayer (and Conv2D/Conv2DDerivative ND4J ops)

• SubsamplingLayer and Subsampling1DLayer (and MaxPooling2D/AvgPooling2D/Pooling2DDerivative ND4J ops)

• BatchNormalization layer (and BatchNorm ND4J op)

• LocalResponseNormalization layer (and LocalResponseNormalization ND4J op)

• Convolution3D layer (and Conv3D/Conv3DDerivative ND4J ops)

MKL-DNN support for other layer types (such as LSTM) will be added in a future release.

MKL-DNN can be disabled globally (ND4J and DL4J) using Nd4jCpu.Environment.getInstance().setUseMKLDNN(false);

MKL-DNN can be disabled globally for specific ops by setting ND4J_MKL_FALLBACK environment variable to the name of the operations to have MKL-DNN support disabled for. For example: ND4J_MKL_FALLBACK=conv2d,conv2d_bp

ND4J: Improved Performance due to Memory Management Changes

Prior releases of ND4J used periodic garbage collection (GC) to release memory that was not allocated in a memory workspace. (Note that DL4J uses workspaces for almost all operations by default hence periodic GC could frequently be disabled when training DL4J networks). However, the reliance on garbage collection resulted in a performance overhead that scaled with the number of objects in the JVM heap.

In 1.0.0-beta4, the periodic garbage collection is disabled by default; instead, GC will be called only when it is required to reclaim memory from arrays that are allocated outside of workspaces.

To re-enable periodic GC (as per the default in beta3) and set the GC frequency to every 5 seconds (5000ms) you can use:

ND4J: Improved Rank 0/1 Array Support

In prior versions of ND4J, scalars and vectors would sometimes be rank 2 instead of rank 0/1 when getting rows/columns, getting sub-arrays using INDArray.get(NDArrayIndex...) or when creating arrays from Java arrays/scalars. Now, behaviour should be more consistent for these rank 0/1 cases. Note to maintain old behaviour for getRow and getColumn (i.e., return rank 2 array with shape [1,x] and [x,1] respectively), the getRow(long,boolean) and getColumn(long,boolean) methods can be used.

DL4J: Attention layers added

Deeplearning4J

Deeplearning4J: Features and Enhancements

Deeplearning4J: Bug Fixes and Optimizations

ND4J and SameDiff

ND4J/SameDiff: Features and Enhancements

• Added basic ("technology preview") of SameDiff UI. Should be considered early WIP with breaking API changes expected in future releases. Supports plotting of SameDiff graphs as well as various metrics (line charts, histograms, etc)

  • Currenty embedding in the DL4J UI - call UIServer.getInstance() then go to localhost:9000/samediff to access.
• ND4J/SameDiff - new operations added:
• SameDiff TensorFlow Import

ND4J/SameDiff: API Changes (Transition Guide): 1.0.0-beta3 to 1.0.0-beta4

• ND4J datatypes - significant changes, see highlights at top of this section

ND4J/SameDiff: Bug Fixes and Optimizations

• SameDiff: Numerous fixes and enhancements

ND4J: Known Issues

• Most CustomOperation operations (such as those used in SameDiff) are CPU only until next release. GPU support was not completed in time for 1.0.0-beta4 release.

DataVec

DataVec: Features and Enhancements

DataVec: Optimizations and Bug Fixes

Arbiter

Arbiter: Enhancements

Arbiter: Fixes

Deeplearning4J

• Fixed issue with incorrect version dependencies in 0.9.0

• Numerical stability improvements to LossMCXENT / LossNegativeLogLikelihood with softmax (should reduce NaNs with very large activations)

ND4J

Known Issues

• Deeplearning4j: Use of Evaluation class no-arg constructor (i.e., new Evaluation()) can result in accuracy/stats being reported as 0.0. Other Evaluation class constructors, and ComputationGraph/MultiLayerNetwork.evaluate(DataSetIterator) methods work as expected.

  • This also impacts Spark (distributed) evaluation: workaround is to replace sparkNet.evaluate(testData); with sparkNet.doEvaluation(testData, 64, new Evaluation(10))[0];, where 10 is the number of classes and 64 in the evaluation minibatch size to use.

• SequenceRecordReaderDataSetIterator applies preprocessors (such as normalization) twice to each DataSet (possible workaround: use RecordReaderMultiDataSetIterator + MultiDataSetWrapperIterator)

• TransferLearning: ComputationGraph may incorrectly apply l1/l2 regularization (defined in FinetuneConfiguration) to frozen layers. Workaround: set 0.0 l1/l2 on FineTuneConfiguration, and required l1/l2 on new/non-frozen layers directly. Note that MultiLayerNetwork with TransferLearning appears to be unaffected.

• Weighted loss functions: Loss functions now support a per-output weight array (row vector)

• Improved error messages on invalid configuration or data; improved validation on both

• Removed Jackson as core dependency (shaded); users can now use any version of Jackson without issue

• Added LossLayer: version of OutputLayer that only applies loss function (unlike OutputLayer: it has no weights/biases)

• Functionality required to build triplet embedding model (L2 vertex, LossLayer, Stack/Unstack vertices etc)

• Reduced DL4J and ND4J ‘cold start’ initialization/start-up time

• Pretrain default changed to false and backprop default changed to true. No longer needed to set these when setting up a network configuration unless defaults need to be changed.

• Numerous bug fixes across DL4J and ND4J

• Performance improvements for nd4j-native & nd4j-cuda backends

• Standalone Word2Vec/ParagraphVectors overhaul:

  • Performance improvements

  • ParaVec inference available for both PV-DM & PV-DBOW

  • Parallel tokenization support was added, to address computation-heavy tokenizers.

• Native RNG introduced for better reproducibility within multi-threaded execution environment.

• Additional RNG calls added: Nd4j.choice(), and BernoulliDistribution op.

• Off-gpu storage introduced, to keep large things, like Word2Vec model in host memory. Available via WordVectorSerializer.loadStaticModel()

• Two new options for performance tuning on nd4j-native backend: setTADThreshold(int) & setElementThreshold(int)

0.6.0 -> 0.7.0 Transition Notes

Notable changes for upgrading codebases based on 0.6.0 to 0.7.0:

• UI: new UI package name is deeplearning4j-ui_2.10 or deeplearning4j-ui_2.11 (previously: deeplearning4j-ui). Scala version suffix is necessary due to Play framework (written in Scala) being used now.

• DataVec ImageRecordReader: labels are now sorted alphabetically by default before assigning an integer class index to each - previously (0.6.0 and earlier) they were according to file iteration order. Use .setLabels(List) to manually specify the order if required.

• CNNs: configuration validation is now less strict. With new ConvolutionMode option, 0.6.0 was equivalent to ‘Strict’ mode, but new default is ‘Truncate’
• Xavier weight initialization change for CNNs and LSTMs: Xavier now aligns better with original Glorot paper and other libraries. Xavier weight init. equivalent to 0.6.0 is available as XAVIER_LEGACY

• DataVec: Custom RecordReader and SequenceRecordReader classes require additional methods, for the new metadata functionality. Refer to existing record reader implementations for how to implement these methods.

• Word2Vec/ParagraphVectors:

  • Few new builder methods:

    • allowParallelTokenization(boolean)

    • useHierarchicSoftmax(boolean)

  • Behaviour change: batchSize: now batch size is ALSO used as threshold to execute number of computational batches for sg/cbow

• Activation function refactor

  • New activation functions added: hard sigmoid, randomized leaky rectified linear units (RReLU)

• Multiple fixes/improvements for Keras model import

• Added P-norm pooling for CNNs (option as part of SubsamplingLayer configuration)

• Iteration count persistence: stored/persisted properly in model configuration + fixes to learning rate schedules for Spark network training

• LSTM: gate activation function can now be configured (previously: hard-coded to sigmoid)

• UI:

  • Added Chinese translation

  • Fixes for UI + pretrain layers

  • Improvements in front-end for handling NaNs

  • Added UIServer.stop() method

  • Fixed score vs. iteration moving average line (with subsampling)

• Solved Jaxb/Jackson issue with Spring Boot based applications

• RecordReaderDataSetIterator now supports NDArrayWritable for the labels (set regression == true; used for multi-label classification + images, etc)

0.7.1 -> 0.7.2 Transition Notes

• Activation functions (built-in): now specified using Activation enumeration, not String (String-based configuration has been deprecated)

Deeplearning4J

• VPTree performance significantly improved

• Convolution performance improvements, including activation caching

• Evaluation improvements

  • ComputationGraph and SparkComputationGraph evaluation convenience methods added (evaluateROC, etc)

  • RegressionEvaluation, ROCBinary etc now support per-output masking (in addition to per-example/per-time-step masking)
• Optimizations: updaters, bias calculation

• New loss functions:

ND4J

• Native parallel sort was added

• New ops added: SELU/SELUDerivative, TAD-based comparisons, percentile/median, Reverse, Tan/TanDerivative, SinH, CosH, Entropy, ShannonEntropy, LogEntropy, AbsoluteMin/AbsoluteMax/AbsoluteSum, Atan2

• New distance functions added: CosineDistance, HammingDistance, JaccardDistance

DataVec

• TransformProcess and Transforms now support NDArrayWritables and NDArrayWritable columns

• Multiple new Transform classes

Arbiter

• • UI now uses Play framework, integrates with DL4J UI (replaces Dropwizard backend). Dependency issues/clashing versions fixed.

  • Supports DL4J StatsStorage and StatsStorageRouter mechanisms (FileStatsStorage, Remote UI via RemoveUIStatsStorageRouter)

  • General UI improvements (additional information, formatting fixes)

0.8.0 -> 0.9.0 Transition Notes

Deeplearning4j

• Updater configuration methods such as .momentum(double) and .epsilon(double) have been deprecated. Instead: use .updater(new Nesterovs(0.9)) and .updater(Adam.builder().beta1(0.9).beta2(0.999).build()) etc to configure

DataVec

• CsvRecordReader constructors: now uses characters for delimiters, instead of Strings (i.e., ',' instead of ",")

Arbiter

• Arbiter UI is now a separate module, with Scala version suffixes: arbiter-ui_2.10 and arbiter-ui_2.11

Version 0.8.0

• Spark 2.0 support (DL4J and DataVec; see transition notes below)

• New layers
• New ComputationGraph vertices

  • L2 distance vertex

  • L2 normalization vertex

• Per-output masking is now supported for most loss functions (for per output masking, use a mask array equal in size/shape to the labels array; previous masking functionality was per-example for RNNs)

• L1 and L2 regularization can now be configured for biases (via l1Bias and l2Bias configuration options)

• Evaluation improvements:

  • For both MultiLayerNetwork and SparkDl4jMultiLayer: added evaluateRegression, evaluateROC, evaluateROCMultiClass convenience methods

  • TSNE re-added to new UI

  • Training UI: now usable without an internet connection (no longer relies on externally hosted fonts)

  • UI: improvements to error handling for ‘no data’ condition

• Epsilon configuration now used for Adam and RMSProp updaters

• Fix for bidirectional LSTMs + variable-length time series (using masking)

• Spark + Kryo: now test serialization + throw exception if misconfigured (instead of logging an error that can be missed)

• MultiLayerNetwork now adds default layer names if no name is specified

• DataVec:

  • JSON/YAML support for DataAnalysis, custom Transforms etc

  • ImageRecordReader refactored to reduce garbage collection load (hence improve performance with large training sets)

  • Faster quality analysis.

• Arbiter: added new layer types to match DL4J

  • Performance improvement for Word2Vec/ParagraphVectors tokenization & training.

• Batched inference introduced for ParagraphVectors

• Nd4j improvements

  • New native operations available for ND4j: firstIndex, lastIndex, remainder, fmod, or, and, xor.

  • OpProfiler NAN_PANIC & INF_PANIC now also checks result of BLAS calls.

  • Nd4.getMemoryManager() now provides methods to tweak GC behavior.

• Alpha version of parameter server for Word2Vec/ParagraphVectors were introduced for Spark. Please note: It’s not recommended for production use yet.

• Performance improvements for CNN inference

0.7.2 -> 0.8.0 Transition Notes

• Spark versioning schemes: with the addition of Spark 2 support, the versions for Deeplearning4j and DataVec Spark modules has changed

  • For Spark 1: use <version>0.8.0_spark_1</version>

  • For Spark 2: use <version>0.8.0_spark_2</version>

  • Also note: Modules with Spark 2 support are released with Scala 2.11 support only. Spark 1 modules are released with both Scala 2.10 and 2.11 support

0.8.0 Known Issues (At Launch)

• Keras 1D convolutional and pooling layers cannot be imported yet. Will be supported in forthcoming release.

• Keras v2 model configurations cannot be imported yet. Will be supported in forthcoming release.

• Initial multi-GPU support viable for standalone and Spark.

• Refactored the Spark API significantly

• Added CuDNN wrapper

• Performance improvements for ND4J

• Introducing DataVec: Lots of new functionality for transforming, preprocessing, cleaning data. (This replaces Canova)

• New DataSetIterators for feeding neural nets with existing data: ExistingDataSetIterator, Floats(Double)DataSetIterator, IteratorDataSetIterator

• New learning algorithms for word2vec and paravec: CBOW and PV-DM respectively

• New native ops for better performance: DropOut, DropOutInverted, CompareAndSet, ReplaceNaNs

• Shadow asynchronous datasets prefetch enabled by default for both MultiLayerNetwork and ComputationGraph

• Better memory handling with JVM GC and CUDA backend, resulting in significantly lower memory footprint