1.0.0-M2

Highlights

Adds proper support for java 9 modules: https://github.com/eclipse/deeplearning4j/pull/9631 https://github.com/eclipse/deeplearning4j/pull/9626

As part of the same work flatbuffers has been upgraded to 1.12.1. This affects the samediff file format and the user interfaces. Flatbuffers as a file format is forwards and backwards compatible but if you have any issues please do let us know. The relevant files have been updated using the flatc compiler.

Removed rl4j: in continuing to cut unmaintained modules, the 1.0 will focus the framework on a few key use cases. This invites other folks to build external modules for a tightly maintained core that focuses on deployment, framework interop and training models in java.

Added new model zoo module called omnihub for dl4j and new samediff models. These can be found here: https://github.com/KonduitAI/omnihub-zoo See more in the new omnihub section.

Migrated the snapshots to sonatype's new repository https://s01.oss.sonatype.org. More context can be found here: https://twitter.com/Brian_Fox/status/1357414532512104448 https://github.com/eclipse/deeplearning4j/pull/9618

Consolidated tests to platform-tests to allow for easy testing of behavior against different backends.

Adds proper support for jetson nano with curated binaries and an updated cuda 10.2

Adds Spark 3 support: https://github.com/eclipse/deeplearning4j/pull/9444

Reduce binary size using selective compilation: https://github.com/eclipse/deeplearning4j/pull/9443

https://github.com/eclipse/deeplearning4j/pull/9451 Remove scala 11 support. Only supporting scala 2.12: https://github.com/eclipse/deeplearning4j/pull/9440

Extensive enhancements for samediff model training: https://github.com/eclipse/deeplearning4j/pull/9501

Nd4j/Samdiff/Libnd4j

Features and Enhancements

1. Add beginnings of graph optimization framework: https://github.com/eclipse/deeplearning4j/pull/9402

2. Many onnx model import improvements (add new ops): https://github.com/eclipse/deeplearning4j/pull/9411 https://github.com/eclipse/deeplearning4j/pull/9489https://github.com/eclipse/deeplearning4j/pull/9475 https://github.com/eclipse/deeplearning4j/pull/9526 https://github.com/eclipse/deeplearning4j/pull/9502https://github.com/eclipse/deeplearning4j/pull/9587 https://github.com/eclipse/deeplearning4j/pull/9599

3. Add new op subset frameworks: allows selective inclusion of operations to enable users to reduce binary size: https://github.com/eclipse/deeplearning4j/pull/9443 https://github.com/eclipse/deeplearning4j/pull/9451 https://github.com/eclipse/deeplearning4j/pull/9569

4. Update onednn to 2.2: https://github.com/eclipse/deeplearning4j/pull/9423 https://github.com/eclipse/deeplearning4j/pull/9425

5. Add updated jetson nano support: https://github.com/eclipse/deeplearning4j/pull/9432

6. Enhance codegen exposing more functions for samediff: https://github.com/eclipse/deeplearning4j/pull/9478 https://github.com/eclipse/deeplearning4j/pull/9503 https://github.com/eclipse/deeplearning4j/pull/9500

7. Add new samediff eager mode (mainly used for model import use cases): https://github.com/eclipse/deeplearning4j/pull/9538 https://github.com/eclipse/deeplearning4j/pull/9535 https://github.com/eclipse/deeplearning4j/pull/9533

8. Add dimensions as input variables: https://github.com/eclipse/deeplearning4j/pull/9584

Bug Fixes

1. Update samediff api to allow dimensions as variables

2. Fix cuda shuffle: https://github.com/eclipse/deeplearning4j/pull/9472 https://github.com/eclipse/deeplearning4j/pull/9459

3. Fix up conditions/matching: https://github.com/eclipse/deeplearning4j/pull/9551

4. ImageResize updates to improve compatibility with onnx: https://github.com/eclipse/deeplearning4j/pull/9495

5. Rewrite compat sparse to dense op: https://github.com/eclipse/deeplearning4j/pull/9566

6. Fix creation of string scalar ndarrays: https://github.com/eclipse/deeplearning4j/pull/9556

7. Fix serialization with conv/pooling3d: https://github.com/eclipse/deeplearning4j/pull/9648

Deeplearning4j

Features and Enhancements

1. Add Spark 3 support: https://github.com/eclipse/deeplearning4j/pull/9553

2. Added Deconvolution3D for keras import https://github.com/eclipse/deeplearning4j/pull/9399

3. Add full channels last support for 3d convolutions: https://github.com/eclipse/deeplearning4j/pull/9578

Bug Fixes

1. Fix confusion matrix count increments: https://github.com/eclipse/deeplearning4j/pull/9553

2. Fix Conv3D data format serialization: https://github.com/eclipse/deeplearning4j/pull/9648

Datavec

Features and Enhancements

1. Add LabelsSource to BagOfWordsVectorizer (thanks to XAI!): https://github.com/eclipse/deeplearning4j/pull/9624

2. Performance enhancement for mnist related datasetiterators: https://github.com/eclipse/deeplearning4j/pull/9612

Bug Fixes

1. Fix memory leak in datavec-arrow: https://github.com/eclipse/deeplearning4j/pull/9441

Omnihub

Launches new Omnihub module. Allows access to models from: https://github.com/KonduitAI/omnihub-zoo

A pretrained omnihub module will provide access to pretrained samediff and dl4j modules. This will also supplant the old dl4j zoo.

Modules will be made available from a Pretrained class:https://github.com/eclipse/deeplearning4j/blob/feb8eee5eb07239c49a4d14786114dc0394aad4e/omnihub/src/main/java/org/eclipse/deeplearning4j/omnihub/models/Pretrained.java#L30

Python4j

Clean up tests/consolidate tests to platform-tests