Highlights

A number of bug fixes following the M1 release, thanks to the feedback from the community, allowed us to quickly sort out a few issues. This is a minor bug fix release to address short comings found with M1. Most fixes were related to keras import, the cnn/rnn helpers, and python4j.

Snapshots will also be published every 2 days automatically now https://github.com/eclipse/deeplearning4j/pull/9355 to get around sonatype ossrh deletion of snapshots every 3 days. This should increase robustness of the snapshots.

Worked around an issue with github actions pre emptively upgrading visual studio breaking the cuda builds: https://github.com/eclipse/deeplearning4j/pull/9364

Added backwards compatibility for centos 6 via a new linux-x86_64-compat classifier enabling use of older glibcs on centos 7:

https://github.com/eclipse/deeplearning4j/pull/9368 https://github.com/eclipse/deeplearning4j/pull/9368https://github.com/eclipse/deeplearning4j/pull/9373

A number of bugs were fixed with LSTM and CUDNN: https://github.com/eclipse/deeplearning4j/pull/9372

Known issues

https://github.com/eclipse/deeplearning4j/issues/9142 - avoid shuffle operations on gpu. Pre save data on cpu in mini batches. For more help, please post on the forums at https://community.konduit.ai/

Deeplearning4j

Features and Enhancements

• Add batch normalization support for RNNs: https://github.com/eclipse/deeplearning4j/pull/9338

• Disable old helpers by default https://github.com/eclipse/deeplearning4j/pull/9343

• Minor unit test fixes: https://github.com/eclipse/deeplearning4j/pull/9346

• Add keras support for cnn 1d NWHC: https://github.com/eclipse/deeplearning4j/pull/9353

• Move the warning about version check to tracing so it stops logging this during normal usage confusing users: https://github.com/eclipse/deeplearning4j/pull/9356

• Allow 1d convolutions to accept feed forward as input type: https://github.com/eclipse/deeplearning4j/pull/9365

• Remove the old benchmark suite and migrate it to contrib: https://github.com/eclipse/deeplearning4j/pull/9374

• Remove old MKLDNNLSTM helper (it never fully functioned anyways): https://github.com/eclipse/deeplearning4j/pull/9381

Bug fixes

• Fixed an issue with helper reflection ensuring the classes would be loaded properly https://github.com/eclipse/deeplearning4j/pull/9333 https://github.com/eclipse/deeplearning4j/pull/9350

• Fix minor workspace activation bug: https://github.com/eclipse/deeplearning4j/pull/9341

• Fixed compilation error when running anything more than jdk 8 and NIO buffers: https://github.com/eclipse/deeplearning4j/pull/9351

• Move logback to be a test dependency for some modules: https://github.com/eclipse/deeplearning4j/pull/9362

• Keras model import fixes for GlobalPooling: https://github.com/eclipse/deeplearning4j/pull/9378 https://github.com/eclipse/deeplearning4j/pull/9384

Nd4j

Features and Enhancements

• Add Eigen op as public ensuring easier use when running eigenvalue decomposition https://github.com/eclipse/deeplearning4j/pull/9328

Bug fixes

• Fixes minor issue with choice(..) op https://github.com/eclipse/deeplearning4j/pull/9360 thanks to https://github.com/Romira915

• Minor applyScalar typo fix: https://github.com/eclipse/deeplearning4j/pull/9385

Datavec

Features and Enhancements

Bug fixes

Fixed serialization bug with StringToTimeTransform: https://github.com/eclipse/deeplearning4j/pull/9377 thanks to community member https://github.com/yumg

Python4j

Features and Enhancements

• Made python4j's python path setting more robust by migrating from set path calls to add path calls: https://github.com/eclipse/deeplearning4j/pull/9386

Bug fixes

• Fixes bug with numpy import array jvm crashes: https://github.com/eclipse/deeplearning4j/pull/9348

Samediff

Features and Enhancements

Bug fixes

Fixed inconsistent conventions between SameDiffVariable getArr and getArrForName().. https://github.com/eclipse/deeplearning4j/pull/9357

Highlights

Adds proper support for java 9 modules: https://github.com/eclipse/deeplearning4j/pull/9631 https://github.com/eclipse/deeplearning4j/pull/9626

As part of the same work flatbuffers has been upgraded to 1.12.1. This affects the samediff file format and the user interfaces. Flatbuffers as a file format is forwards and backwards compatible but if you have any issues please do let us know. The relevant files have been updated using the flatc compiler.

Removed rl4j: in continuing to cut unmaintained modules, the 1.0 will focus the framework on a few key use cases. This invites other folks to build external modules for a tightly maintained core that focuses on deployment, framework interop and training models in java.

Added new model zoo module called omnihub for dl4j and new samediff models. These can be found here: https://github.com/KonduitAI/omnihub-zoo See more in the new omnihub section.

Migrated the snapshots to sonatype's new repository https://s01.oss.sonatype.org. More context can be found here: https://twitter.com/Brian_Fox/status/1357414532512104448 https://github.com/eclipse/deeplearning4j/pull/9618

Consolidated tests to platform-tests to allow for easy testing of behavior against different backends.

Adds proper support for jetson nano with curated binaries and an updated cuda 10.2

Adds Spark 3 support: https://github.com/eclipse/deeplearning4j/pull/9444

Reduce binary size using selective compilation: https://github.com/eclipse/deeplearning4j/pull/9443

https://github.com/eclipse/deeplearning4j/pull/9451 Remove scala 11 support. Only supporting scala 2.12: https://github.com/eclipse/deeplearning4j/pull/9440

Extensive enhancements for samediff model training: https://github.com/eclipse/deeplearning4j/pull/9501

Nd4j/Samdiff/Libnd4j

Features and Enhancements

1. Add beginnings of graph optimization framework: https://github.com/eclipse/deeplearning4j/pull/9402

2. Many onnx model import improvements (add new ops): https://github.com/eclipse/deeplearning4j/pull/9411 https://github.com/eclipse/deeplearning4j/pull/9489https://github.com/eclipse/deeplearning4j/pull/9475 https://github.com/eclipse/deeplearning4j/pull/9526 https://github.com/eclipse/deeplearning4j/pull/9502https://github.com/eclipse/deeplearning4j/pull/9587 https://github.com/eclipse/deeplearning4j/pull/9599

3. Add new op subset frameworks: allows selective inclusion of operations to enable users to reduce binary size: https://github.com/eclipse/deeplearning4j/pull/9443 https://github.com/eclipse/deeplearning4j/pull/9451 https://github.com/eclipse/deeplearning4j/pull/9569

4. Update onednn to 2.2: https://github.com/eclipse/deeplearning4j/pull/9423 https://github.com/eclipse/deeplearning4j/pull/9425

5. Add updated jetson nano support: https://github.com/eclipse/deeplearning4j/pull/9432

6. Enhance codegen exposing more functions for samediff: https://github.com/eclipse/deeplearning4j/pull/9478 https://github.com/eclipse/deeplearning4j/pull/9503 https://github.com/eclipse/deeplearning4j/pull/9500

7. Add new samediff eager mode (mainly used for model import use cases): https://github.com/eclipse/deeplearning4j/pull/9538 https://github.com/eclipse/deeplearning4j/pull/9535 https://github.com/eclipse/deeplearning4j/pull/9533

8. Add dimensions as input variables: https://github.com/eclipse/deeplearning4j/pull/9584

Bug Fixes

1. Update samediff api to allow dimensions as variables

2. Fix cuda shuffle: https://github.com/eclipse/deeplearning4j/pull/9472 https://github.com/eclipse/deeplearning4j/pull/9459

3. Fix up conditions/matching: https://github.com/eclipse/deeplearning4j/pull/9551

4. ImageResize updates to improve compatibility with onnx: https://github.com/eclipse/deeplearning4j/pull/9495

5. Rewrite compat sparse to dense op: https://github.com/eclipse/deeplearning4j/pull/9566

6. Fix creation of string scalar ndarrays: https://github.com/eclipse/deeplearning4j/pull/9556

7. Fix serialization with conv/pooling3d: https://github.com/eclipse/deeplearning4j/pull/9648

Deeplearning4j

Features and Enhancements

1. Add Spark 3 support: https://github.com/eclipse/deeplearning4j/pull/9553

2. Added Deconvolution3D for keras import https://github.com/eclipse/deeplearning4j/pull/9399

3. Add full channels last support for 3d convolutions: https://github.com/eclipse/deeplearning4j/pull/9578

Bug Fixes

1. Fix confusion matrix count increments: https://github.com/eclipse/deeplearning4j/pull/9553

2. Fix Conv3D data format serialization: https://github.com/eclipse/deeplearning4j/pull/9648

Datavec

Features and Enhancements

1. Add LabelsSource to BagOfWordsVectorizer (thanks to XAI!): https://github.com/eclipse/deeplearning4j/pull/9624

2. Performance enhancement for mnist related datasetiterators: https://github.com/eclipse/deeplearning4j/pull/9612

Bug Fixes

1. Fix memory leak in datavec-arrow: https://github.com/eclipse/deeplearning4j/pull/9441

Omnihub

Launches new Omnihub module. Allows access to models from: https://github.com/KonduitAI/omnihub-zoo

A pretrained omnihub module will provide access to pretrained samediff and dl4j modules. This will also supplant the old dl4j zoo.

Modules will be made available from a Pretrained class:https://github.com/eclipse/deeplearning4j/blob/feb8eee5eb07239c49a4d14786114dc0394aad4e/omnihub/src/main/java/org/eclipse/deeplearning4j/omnihub/models/Pretrained.java#L30

Python4j

Clean up tests/consolidate tests to platform-tests

Highlights - 1.0.0-beta2 Release

• ND4J/Deeplearning4j: Added support for CUDA 9.2. Dropped support for CUDA 9.1. (1.0.0-beta2 release has CUDA 8.0, 9.0 and 9.2 support)

• Deeplearning4j: New SameDiff layers with training support - Link Link

• Deeplearning4j resource (datasets, pretrained models) storage directory can now be configured via DL4JResources.setBaseDirectory method or org.deeplearning4j.resources.directory system property

• ND4J: all indexing is now done with longs instead of ints to allow for arrays with dimensions and lengths greater than Integer.MAX_VALUE (approx. 2.1 billion)

• ND4J: nd4j-native-platform will now use Intel MKL-DNN as the default/bundled BLAS implementation (replacing OpenBLAS as the previous default)

• Deeplearning4j: Added Out-of-memory (OOM) crash dump reporting functionality. Provides a dump with memory use and configuration if training/inference OOMs (to assist with debugging and tuning memory configuration).

• Deeplearning4j - new layers: Locally connected 1d Link, Locally connected 2d Link

Deeplearning4J

Deeplearning4J: New Features

• Added new SameDiff layers (automatic differentiation - only single class, forward pass definition required) to DL4J with full training support - SameDiffLayer, SameDiffVertex, SameDiffOutputLayer, SameDiffLambdaLayer, SameDiffLambdaVertex - note that these are CPU-only execution for now Link Link Link

• Resource (datasets, pretrained models) storage directory can now be configured via DL4JResources.setBaseDirectory method or org.deeplearning4j.resources.directory system property. Note that it is also possible to set a different base location for downloads (for local mirrors of DL4J resources) Link

• Added Out-of-memory (OOM) crash dump reporting functionality. Provides a dump with memory use and configuration if training/inference OOMs. Same information is available (without a crash) for MultiLayerNetwork/ComputationGraph.memoryInfo methods. Can be disabled (or output directory set) using system properties - Link

• Added Composite[Multi]DataSetPreProcessor to enable multiple [Multi]DataSetPreProcessors to be applied in a single iterator Link

• Added ComputationGraph evaluate methods for multi-output networks: evaluate(DataSetIterator, Map<Integer,IEvaluation[]>) and evaluate(MultiDataSetIterator, Map<Integer,IEvaluation[]>) Link

• Added JointMultiDataSetIterator - utility iterator used to create MultiDataSetIterator from multiple DataSetIterators Link

• GraphVertices may now have trainable parameters directly (not just enclose layers with trainable parameters) Link

• Added MultiLayerNetwork/ComputationGraph getLearningRate methods Link

• Added RandomDataSetIterator and RandomMultiDataSetIterator (mainly for testing/debugging) Link Link

• Added cyclical "1cycle" schedule for learning rate schedules etc - Link

• RDD repartitioning for Spark training is more configurable (adds Repartitioner interface) Link

• Added ComputationGraph.getIterationCount() and .getEpochCount() for consistency with MultiLayerNetwork Link

• Added locally connected 1d layer Link Link

• Spark "data loader" API (mainly for Spark) Link Link Link

• Spark evaluation: added evaluation method overloads that allow specifying the number of evaluation workers (less than number of Spark threads) Link

• CnnSentenceDataSetIterator now has a Format argument, and supports outputting data for RNNs and 1D CNNs Link

• Added ComputationGraph/MultiLayerNetwork.pretrain((Multi)DataSetIterator, int epochs) method overloads Link

• MultiLayerNetwork and ComputationGraph now have output method overloads where the network output can be placed in the user-specified workspace, instead of being detached Link Link. This can be used to avoid creating INDArrays that need to be garbage collected before native memory can be freed.

• EmbeddingSequenceLayer now supports [minibatch,1,seqLength] format sequence data in addition to [minibatch,seqLength] format data Link

• CuDNN batch norm implementation will now be used for rank 2 input, not just rank 4 input Link

• Environment variables and system properties for DL4J have been centralized into DL4JResources and DL4JEnvironmentVars classes, with proper descriptions Link Link

• MultiLayerNetwork and ComputationGraph output/feedForward/fit methods are now thread-safe via synchronization. Note that concurrent use is not recommended due to performance (instead: use ParallelInference); however the now-synchronized methods should avoid obscure errors due to concurrent modifications Link

• BarnesHutTSNE now throws a useful exception in the case where the distance metric is undefined (for example, all zeros plus cosine similarity) Link

Deeplearning4J: Bug Fixes and Optimizations

• ComputationGraph.addListeners was not working correctly if listeners were already present Link, Link

• TinyImageNetDataSetIterator did not validate/correctly use input shape configuration Link, Link

• BatchNormalization layer now correctly asserts that nOut is set if required (instead of unfriendly shape errors later) Link

• Fixed issue where OutputLayer may not initialize parameter constraints correctly Link

• Fixed performance issue with Nesterov updater using CPU-only op for CUDA execution Link

• Removed TerminationCondition for DL4J optimizers - was not used in practice, and had minor overhead Link

• Fixed issue where EvaluativeListener could hit a workspace validation exception when workspaces are enabled Link

• Fixed issue where TrainingListener.onEpochStart/onEpochEnd were not being called correctly for ComputationGraph Link

• Fixed workspace issue with TensorFlowCnnToFeedForwardPreProcessor Link

• Performance optimization for BatchNormalization when using CuDNN Link

• Performance optimization: Dropout will be applied in-place when safe to do so, avoiding a copy Link

• Added CuDNN implementation of Dropout Link

• Reduced memory use for CuDNN: CuDNN working memory is now shared and reused between layers within a network Link

• CuDNN batch normalization implementation would fail with FP16 datatype Link

• Fixed issue Bidirectional LSTM may incorrectly use workspaces causing an exception Link

• Fixed issue with early stopping where scores to be maximized (accuracy, f1, etc) were not properly triggering termination conditions Link

• Fixed issue where label mask counter could be incorrectly incremented in ComputationGraph.computeGradientAndScore() Link

• ComputationGraph was not setting lastEtlTime field during training Link

• Fixed issue with AutoEncoder layer when workspaces are enabled Link

• Fixed issue with EmbeddingSequenceLayer use of mask arrays Link

• Lombok is now provided scope everywhere, isn't on user classpath when using DL4J Link

• Fixed issue where WordVectorSerializer.readParagraphVectors(File) initialization of label source Link

• Spark training (gradient sharing) now properly handles empty partition edge case when encountered during training Link

• Errors are propagated better/more consistently for Spark gradient sharing training Link

• Fixed issue with 1D CNN layers with mask arrays and stride > 1 (masks not being correctly downsized) Link

• DL4J Batch norm implementation was not correctly adding epsilon value during inference, only during training (CuDNN unaffected) Link

• CuDNN subsampling layers with max pooling and ConvolutionMode.SAME may have taken padding value (0) as the maximum for border values when all non-padding values are less than 0 Link

• Spark training with gradient sharing now passes listeners to workers correctly Link

• Fixed rare (and non-terminal) concurrent modification issue with UI and FileStatsStorage Link

• CuDNN convolution layer now supports dilation > 2 (previously: used DL4J conv layer implementation as a fallback) Link

• Yolo2OutputLayer now implements computeScoreForExamples() Link

• SequenceRecordReeaderDataSetIterator now handles the "no labels" case correctly Link

• Fixed issue where BarnesHutTSNE could hit a workspace validation exception Link

• EMNIST iterator could produce incorrect data in some cases after a reset Link

Deeplearning4J: API Changes (Transition Guide): 1.0.0-beta to 1.0.0-beta2

• GravesLSTM has been deprecated in favor of LSTM due to lack of CuDNN support but otherwise similar accuracy to in practice. Use LSTM class instead.

• deeplearning4j-modelexport-solr: now uses Lucene/Solr version 7.4.0 (was 7.3.0) Link

• Mask arrays for CNN2d layers must be in broadcastable 4d format: [minibatch,depth or 1, height or 1, width or 1] - previously they were 2d with shape [minibatch,height] or [minibatch,width]. This provents ambiguity in later cases (pooling layers), and allows for more complex masking scenarios (such as masking for different image sizes in same minibatch). Link

• Some older/deprecated Model and Layer methods have been removed. (validateInput(), initParams()). Some custom layers may need to be updated as a result Link

Deelpearning4J: 1.0.0-beta2 Known Issues

• Windows users are unable to load the HDF5 files used in SvhnLabelProvider (used in HouseNumberDetection example). Linux/Mac users are unaffected. A workaround for windows users is to add the sonatype snapshot dependency org.bytedeco.javacpp-presets:hdf5-platform:jar:1.10.2-1.4.3-SNAPSHOT Link

Deeplearing4J: Keras Import

• Keras model import now imports every Keras application

• Supports GlobalPooling3D layer import

• Supports RepeatVector layer import

• Supports LocallyConnected1D and LocallyConnected2D layers

• Keras Lambda layers can now be imported by registering custom SameDiff layers

• All Keras optimizers are now supported

• All advanced activation functions can now be imported.

• Many minor bugs have been fixed, including proper weight setting for all configurations of BatchNormalization, improvements to Reshape SeparableConvolution2D, and full support of Bidirectional layers.

ND4J

ND4J: New Features

• ND4J: all indexing is now done with longs instead of ints to allow for arrays with dimensions and lengths greater than Integer.MAX_VALUE (approx. 2.1 billion)

• Added the ability to write Numpy .npy format using Nd4j.writeAsNumpy(INDArray,File) and convert an INDArray to a numpy strict in-memory using Nd4j.convertToNumpy(INDArray) Link

• ND4j-common ClassPathResource: added ClassPathResource.copyDirectory(File) Link

• SameDiff: A significant number of new ops, and backprop implementations for existing ops

• Added Nd4j.randomBernoulli/Binomial/Exponential convenience methods Link

• Added way to disable/suppress ND4J initialization logging via org.nd4j.log.initialization system property Link

• SameDiff class - most op/constructor methods now have complete/useful javadoc Link

• Workspaces can now be disabled globally, ignoring workspace configuration. This is mainly used for debugging; use Nd4j.getWorkspaceManager().setDebugMode(DebugMode.DISABLED) or Nd4j.getWorkspaceManager().setDebugMode(DebugMode.SPILL_EVERYTHING); to enable this. Link [Link]

• Added EnvironmentalAction API for environment variable processing Link

• ND4J environment variables and system properties have been centralized in ND4jEnvironmentVars and ND4jSystemProperties classes Link and Link

ND4J: Bug Fixes and Optimizations

• SameDiff: a significant number of bug fixes for execution and individual ops

• Fixed issue where INDArray.toDoubleArray() with true scalars (rank 0 arrays) Link

• Fixed issue with DataSet.sample() not working for rank 3+ features Link

• IActivation implementations now validate/enforce same shape for activations and gradients Link

• Fixed issue with muliColumnVector where vector is 1d Link

• ImagePreProcessingScaler now supports serialization via NormalizerSerializerStrategy and ModelSerializer Link

• Performance optimization for threshold encoding used in DL4J's Spark gradient sharing distributed training implementation Link

• SameDiff: Fixed issue where memory wasn't always released after execution Link

• DataSet.save() and MultiDataSet.save() methods now save example metadata when present Link

• Fixed issue with KFoldIterator when dataset does not divide equally into folds with no remainder Link

• Fixed issue where version check functionality could fail to load resources if resources are on a path with spaces Link

ND4J: Known Issues

ND4J: API Changes (Transition Guide): 1.0.0-beta to 1.0.0-beta2

• CUDA 9.1 support has been removed. CUDA 8.0, 9.0 and 9.2 support is available

• Due to long indexing changes, long/long[] should be used in place of int/int[] in some places (such as INDArray.size(int), INDArray.shape())

• Simplified DataSetIterator API: totalExamples(), cursor() and numExamples() - these were unsupported on most DataSetIterator implementations, and not used in practice for training. Custom iterators should remove these methods also Link

• Long-deprecated DataSet.getFeatureMatrix() has been removed. Use DataSet.getFeatures() instead. Link

• Unused and not properly tested/maintained utility class BigDecimalMath has been removed. Users should find an aternative library for this functionality, if required.

• Not properly maintained complex number support classes (IComplexNumber, IComplexNDArray) have been removed entirely Link

DataVec

DataVec: New Features

• Added AnalyzeLocal class to mirror functionality of AnalyzeSpark (but without Spark dependency) Link

• Added JacksonLineSequenceRecordReader: RecordReader used for multi-example JSON/XML where each line in a file is an independent example Link

• Added RecordConvert.toRecord(Schema, List<Object>) Link

• Added missing FloatColumnCondition Link

• Added CSVLineSequenceRecordReader for "each line in CSV is a sequence, and sequence is single-valued/univariate" Link

• Added CSVMultiSequenceRecordReader for "multiple multi-valued sequences in a single CSV" data Link

DataVec: Optimizations and Bug Fixes

• Fixed issue with NativeImageLoader on Android Link

• Fixed issue with ExcelRecordReader Link

• Fixed issue where bad args for CSVRecordReader.next(int) could cause an unnecessarily large list to be generated Link

DataVec: API Changes (Transition Guide): 1.0.0-beta to 1.0.0-beta2

Arbiter

Arbiter: New Features

• Added DataSource interface. Unlike old DataProvider, this does not require JSON serializability (only a no-arg constructor) Link

• Added numerous enhancements and missing configuration options (constraints, dilation, etc) Link Link

Arbiter: Fixes

• DataProvider has been deprecated. Use DataSource instead.

RL4J

• stepCounter, epochCounter and historyProcessor can now be set Link

• Random seed is now loaded for ACPolicy is loaded Link

Highlights - 1.0.0-beta4 Release

Main highlight: full multi-datatype support for ND4J and DL4J. In past releases, all N-Dimensional arrays in ND4J were limited to a single datatype (float or double), set globally. Now, arrays of all datatypes may be used simultaneously. The following datatypes are supported:

• DOUBLE: double precision floating point, 64-bit (8 byte)

• FLOAT: single precision floating point, 32-bit (4 byte)

• HALF: half precision floating point, 16-bit (2 byte), "FP16"

• LONG: long signed integer, 64 bit (8 byte)

• INT: signed integer, 32 bit (4 byte)

• SHORT: signed short integer, 16 bit (2 byte)

• UBYTE: unsigned byte, 8 bit (1 byte), 0 to 255

• BYTE: signed byte, 8 bit (1 byte), -128 to 127

• BOOL: boolean type, (0/1, true/false). Uses ubyte storage for easier op parallelization

• UTF8: String array type, UTF8 format

ND4J Behaviour changes of note:

• When creating an INDArray from a Java primitive array, the INDArray datatype will be determined by the primitive array type (unless a datatype is specified)

  • For example: Nd4j.createFromArray(double[]) -> DOUBLE datatype INDArray

  • Similarly, Nd4j.scalar(1), Nd4j.scalar(1L), Nd4j.scalar(1.0) and Nd4j.scalar(1.0f) will produce INT, LONG, DOUBLE and FLOAT type scalar INDArrays respectively

• Some operations require matched datatypes for operands

  • For example, if x and y are different datatypes, a cast may be required: x.add(y.castTo(x.dataType()))

• Some operations have datatype restrictions: for example, sum on a UTF8 array is not supported, nor is variance on a BOOL array. For some operations on boolean arrays (such as sum), casting to an integer or floating point type first may make sense.

DL4J Behaviour changes of note:

• MultiLayerNetwork/ComputationGraph no longer depend in any way on ND4J global datatype.

  • The datatype of a network (DataType for it's parameters and activations) can be set during construction using NeuralNetConfigutation.Builder().dataType(DataType)

  • Networks can be converted from one type to another (double to float, float to half etc) using MultiLayerNetwork/ComputationGraph.convertDataType(DataType) method

Main new methods:

• Nd4j.create(), zeros(), ones(), linspace(), etc methods with DataType argument

• INDArray.castTo(DataType) method - to convert INDArrays from one datatype to another

• New Nd4j.createFromArray(...) methods for

ND4J/DL4J: CUDA - 10.1 support added, CUDA 9.0 support dropped

CUDA versions supported in 1.0.0-beta4: CUDA 9.2, 10.0, 10.1.

ND4J: Mac/OSX CUDA support dropped

Mac (OSX) CUDA binaries are no longer provided. Linux (x86_64, ppc64le) and Windows (x86_64) CUDA support remains. OSX CPU support (x86_64) is still available.

DL4J/ND4J: MKL-DNN Support Added DL4J (and ND4J conv2d etc ops) now support MKL-DNN by default when running on CPU/native backend. MKL-DNN support is implemented for the following layer types:

• ConvolutionLayer and Convolution1DLayer (and Conv2D/Conv2DDerivative ND4J ops)

• SubsamplingLayer and Subsampling1DLayer (and MaxPooling2D/AvgPooling2D/Pooling2DDerivative ND4J ops)

• BatchNormalization layer (and BatchNorm ND4J op)

• LocalResponseNormalization layer (and LocalResponseNormalization ND4J op)

• Convolution3D layer (and Conv3D/Conv3DDerivative ND4J ops)

MKL-DNN support for other layer types (such as LSTM) will be added in a future release.

MKL-DNN can be disabled globally (ND4J and DL4J) using Nd4jCpu.Environment.getInstance().setUseMKLDNN(false);

MKL-DNN can be disabled globally for specific ops by setting ND4J_MKL_FALLBACK environment variable to the name of the operations to have MKL-DNN support disabled for. For example: ND4J_MKL_FALLBACK=conv2d,conv2d_bp

ND4J: Improved Performance due to Memory Management Changes

Prior releases of ND4J used periodic garbage collection (GC) to release memory that was not allocated in a memory workspace. (Note that DL4J uses workspaces for almost all operations by default hence periodic GC could frequently be disabled when training DL4J networks). However, the reliance on garbage collection resulted in a performance overhead that scaled with the number of objects in the JVM heap.

In 1.0.0-beta4, the periodic garbage collection is disabled by default; instead, GC will be called only when it is required to reclaim memory from arrays that are allocated outside of workspaces.

To re-enable periodic GC (as per the default in beta3) and set the GC frequency to every 5 seconds (5000ms) you can use:

Nd4j.getMemoryManager().togglePeriodicGc(true);
Nd4j.getMemoryManager().setAutoGcWindow(5000);

ND4J: Improved Rank 0/1 Array Support

In prior versions of ND4J, scalars and vectors would sometimes be rank 2 instead of rank 0/1 when getting rows/columns, getting sub-arrays using INDArray.get(NDArrayIndex...) or when creating arrays from Java arrays/scalars. Now, behaviour should be more consistent for these rank 0/1 cases. Note to maintain old behaviour for getRow and getColumn (i.e., return rank 2 array with shape [1,x] and [x,1] respectively), the getRow(long,boolean) and getColumn(long,boolean) methods can be used.

DL4J: Attention layers added

• AttentionVertex

• LearnedSelfAttentionLayer

• RecurrentAttentionLayer

• SelfAttentionLayer

Deeplearning4J

Deeplearning4J: Features and Enhancements

• Added MKL-DNN support for Conv/Pool/BatchNorm/LRN layers. MKL-DNN will be used automatically when using nd4j-native backend. (Link, Link)

• L1/L2 regularization now made into a class; weight decay added, with better control as to when/how it is applied. See this page for more details on the difference between L2 and weight decay. In general, weight decay should be preferred to L2 regularization. (Link, Link)

• Added dot product attention layers: AttentionVertex, LearnedSelfAttentionLayer, RecurrentAttentionLayer and SelfAttentionLayer

• The parameter/activation datatypes for new models can be set for new networks using the dataType(DataType) method on NeuralNetConfiguration.Builder (Link)

• MultiLayerNetwork/ComputationGraph can be converted between (floating point) datatypes FP16/32/64 for the parameters and activations using the MultiLayerNetwork/ComputationGraph.convertDataType(DataType) methods (Link, Link)

• EmbeddingLayer and EmbeddingSequenceLayer builders now have .weightInit(INDArray) and .weightInit(Word2Vec) methods for initializing parameters from pretrained word vectors (Link)

• PerformanceListener can now be configured to report garbage collection information (number/duration) Link

• Evaluation class will now check for NaNs in the predicted output and throw an exception instead treating argMax(NaNs) as having value 0 (Link)

• Added ModelAdapter for ParallelInference for convenience and for use cases such as YOLO (allows improved performance by avoiding detached (out-of-workspace) arrays) (Link)

• Added GELU Activation function (Link)

• Added BertIterator (a MultiDataSetIterator for BERT training - supervised and unsupervised) Link

• Added validation to MultiLayerNetwork/ComputationGraph that throws an exception when attempting to perform Regression evaluation on a classifier, or vice-versa (Link, Link)

• Added ComputationGraph.output(List<String> layers, boolean train, INDArray[] features, INDArray[] featureMasks) method to get the activations for a specific set of layers/vertices only (without redundant calculations) (Link)

• Weight initialization for networks is now implemented as classes (not just enumerations) and hence is now extesible via IWeightInit interface (Link); i.e., custom weight initializations are now supported (Link, Link)

• Added Capsule Network layers (no GPU acceleration until next release) - CapsuleLayer, CapsuleStrengthLayer and PrimaryCapsules (Link)

• Added Cifar10DataSetIterator to replace CifarDataSetIterator (Link, Link)

• Keras import: Importing models from InputStream is now supported (Link, Link)

• Layer/NeuralNetConfiguration builders now have getter/setter methods also, for better Kotlin support (Link)

• Most JavaScript dependencies and fonts for UI have been migrated to WebJars (Link)

• CheckpointListener now has static availableCheckpoints(File), loadCheckpointMLN(File, int) and lostLastCheckpointMLN(File) etc methods (Link)

• MultiLayerNetwork/ComputationGraph now validate and throw an exception in certain incompatible RNN configurations, like truncated backpropagation through time combined with LastTimeStepLayer/Vertex (Link)

• Added BERT WordPiece tokenizers (Link)

• Deeplearning4j UI now has multi-user/multi-session support - use UIServer.getInstance(boolean multiSession, Function<String,StatsStorage>) to start UI in multi-session mode (Link)

• Layer/NeuralNetworkConfiguration builder method validation standardized and improved (Link)

• WordVectorSerializer now supports reading and exporting text forwat vectors via WordVectorSerializer.writeLookupTable and readLookupTable (Link]

• Updated to JavaCPP, JavaCPP presets, and JavaCV version 1.5 (Link)

• Added EvaluationBinary false alarm rate calculation (Link)

• ComputationGraph GraphBuilder now has an appendLayer method that can be used to add layers connected to the last added layer/vertex (Link)

• Added Wasserstein loss function (Link)

• Keras import: Improved errors/exceptions for lambda layer import (Link)

• Apache Lucene/Solr upgraded from 7.5.0 to 7.7.1 (Link)

• KMeans clustering strategy is now configurable (Link)

Deeplearning4J: Bug Fixes and Optimizations

• DL4J Spark training: fix for shared clusters (multiple simultaneous training jobs) - Aeron stream ID now generated randomly (Link)

• cuDNN helpers will no longer attempt to fall back on built-in layer implementations if an out-of-memory exception is thrown (Link)

• Batch normalization global variance reparameterized to avoid underflow and zero/negative variance in some cases during distributed training (Link)

• Fixed a bug where dropout instances were incorrectly shared between layers when using transfer learning with dropout (Link, Link)

• Fixed issue where tensorAlongDimension could result in an incorrect array order for edge cases and hence exceptions in LSTMs (Link)

• Fixed an edge case issue with ComputationGraph.getParam(String) where the layer name contains underscores (Link)

• Fixed an edge case with ParallelInference on CUDA where (very rarely) input array operations (such as normalization) may not be fully completed before transferring an array between threads (Link, Link)

• Fixed an edge case with KFoldIterator when the total number of examples is not a multiple of the batch size (Link, Link)

• Fixed an issue where DL4J UI could throw a NoClassDefFoundError on Java 9/10/11 (Link, Link)

• Keras import: added aliases for weight initialization (Link)

• Fixed issue where dropout instances would not be correctly cloned when network configuration was cloned (Link)

• Fixed workspace issue with ElementwiseVertex with single input (Link)

• Fixed issue with UI where detaching StatsStorage could attempt to remove storage twice, resulting in an exception (Link)

• Fixed issue where LossMultiLabel would generate NaNs when all labels in minibatch are the same class. Now 0 gradient is returned instead. (Link, Link)

• Fixed an issue where DepthwiseConv2D weight could be wrong shape on restoring network from saved format (Link)

• Fixed issue where BaseDatasetIterator.next() would not apply preprocessors, if one was set (Link)

• Improved default configuration for CenterLossOutputLayer (Link)

• Fixed an issue for UNet non-pretrained configuration (Link)

• Fixed an issue where Word2Vec VocabConstructor could deadlock under some circumstances (Link)

• SkipGram and CBOW (used in Word2Vec) were made native operations for better performance (Link)

• Fixed an issue where references to detached StatsListener instances would be maintained, potentially leading to memory issues when using InMemoryStatsListener (Link)

• Optimization: Workspaces were added to SequenceVectors and Word2Vec (Link)

• Improved validation for RecordReaderDataSetIterator (Link)

• Improved handling of unknown words in WordVectors implementation (Link)

• Yolo2OutputLayer: Added validation for incorrect labels shape. (Link)

• LastTimeStepLayer will now throw an exception when the input mask is all 0s (no data - no last time step) (Link)

• Fixed an issue where MultiLayerNetwork/ComputationGraph.setLearningRate method could lead to invalid updater state in some rare cases (Link)

• Fixed an issue where Conv1D layer would calculate output length in MultiLayerNetwork.summary() (Link)

• Async iterators are now used in EarlyStoppingTrained to improve data loading performance (Link)

• EmbeddingLayer and EmbeddingSequenceLayer performance has been improved on CUDA (Link)

• Removed outdated/legacy scala tools repository (Link, Link)

• Fixed issues in L2NormalizeVertex equals/hashcode methods (Link)

• Fixed Workspace issue in ConvolutionalListener (Link)

• Fixed EvaluationBinary falsePositiveRate calculation (Link)

• Added validation and useful exception for MultiLayerNetwork.output(DataSetIterator) methods (Link)

• Fixed minor issue where ComputationGraph.summary() would throw a NullPointerException if init() had not already been called (Link)

• Fixed a ComputationGraph issue where an input into a single layer/vertex repeated multiple times could fail during training (Link)

• Improved performance for KMeans implementation (Link)

• Fixed an issue with rnnGetPreviousState for RNNs in 'wrapper' layers such as FrozenLayer (Link)

• Keras import: Fixed an issue with order of words when importing some Keras tokenizers (Link)

• Keras import: fixed issue with possible UnsupportedOperationException in KerasTokenizer class (Link)

• Keras import: fixed an import issue with models combining embeddings, reshape and convolution layers (Link)

• Keras import: fixed an import issue with input type inference for some RNN models (Link)

• Fixed some padding issues in LocallyConnected1D/2D layers (Link)

ND4J and SameDiff