ND4J: Added SameDiff - Java automatic differentiation library (alpha release) with Tensorflow import (technology preview) and hundreds of new operations
ND4J: Added CUDA 9.0 and 9.1 support (with cuDNN), dropped support for CUDA 7.5, continued support for CUDA 8.0
ND4J: Native binaries (nd4j-native on Maven Central) now ship with AVX/AVX2/AVX-512 support (Windows/Linux)
DL4J: Large number of new layers and API improvements
DL4J: Keras 2.0 import support
Layers (new and enhanced)
Added Yolo2OutputLayer CNN layer for object detection (Link). See also DataVec's ObjectDetectionRecordReader
Adds support for 'no bias' layers via hasBias(boolean) config (DenseLayer, EmbeddingLayer, OutputLayer, RnnOutputLayer, CenterLossOutputLayer, ConvolutionLayer, Convolution1DLayer). EmbeddingLayer now defaults to no bias (Link)
Adds support for dilated convolutions (aka 'atrous' convolutions) - ConvolutionLayer, SubsamplingLayer, and 1D versions there-of. (Link)
ElementWiseVertex now (additionally) supports Average and Max modes in addition to Add/Subtract/Product (Link)
Added SeparableConvolution2D layer (Link)
Added Deconvolution2D layer (aka transpose convolution, fractionally strided convolution layer) (Link)
Added ReverseTimeSeriesVertex (Link)
Added RnnLossLayer - no-parameter version of RnnOutputLayer, or RNN equivalent of LossLayer (Link)
Added CnnLossLayer - no-parameter CNN output layer for use cases such as segmentation, denoising, etc. (Link)
Added Bidirectional layer wrapper (converts any uni-directional RNN to a bidirectional RNN) (Link)
Added SimpleRnn layer (aka "vanilla" RNN layer) (Link)
Added LastTimeStep wrapper layer (wraps a RNN layer to get last time step, accounting for masking if present) (Link)
Added MaskLayer utility layer that simply zeros out activations on forward pass when a mask array is present (Link)
Added alpha-version (not yet stable) SameDiff layer support to DL4J (Note: forward pass, CPU only for now)(Link)
Added Cropping2D layer (Link)
Added parameter constraints API (LayerConstraint interface), and MaxNormConstraint, MinMaxNormConstraint, NonNegativeConstraint, UnitNormConstraint implementations (Link)
Significant refactoring of learning rate schedules (Link)
Added ISchedule interface; added Exponential, Inverse, Map, Poly, Sigmoid and Step schedule implementations (Link)
Added support for both iteration-based and epoch-based schedules via ISchedule. Also added support for custom (user defined) schedules
Learning rate schedules are configured on the updaters, via the .updater(IUpdater) method
Added dropout API (IDropout - previously dropout was available but not a class); added Dropout, AlphaDropout (for use with self-normalizing NNs), GaussianDropout (multiplicative), GaussianNoise (additive). Added support for custom dropout types (Link)
Added support for dropout schedules via ISchedule interface (Link)
Added weight/parameter noise API (IWeightNoise interface); added DropConnect and WeightNoise (additive/multiplicative Gaussian noise) implementations (Link); dropconnect and dropout can now be used simultaneously
Adds layer configuration alias .units(int) equivalent to .nOut(int) (Link)
Adds ComputationGraphConfiguration GraphBuilder .layer(String, Layer, String...) alias for .addLayer(String, Layer, String...)
Layer index no longer required for MultiLayerConfiguration ListBuilder (i.e., .list().layer(<layer>) can now be used for configs) (Link)
Added MultiLayerNetwork.summary(InputType) and ComputationGraph.summary(InputType...) methods (shows layer and activation size information) (Link)
MultiLayerNetwork, ComputationGraph and layerwise trainable layers now track the number of epochs (Link)
Added deeplearning4j-ui-standalone module: uber-jar for easy launching of UI server (usage: java -jar deeplearning4j-ui-standalone-1.0.0-alpha.jar -p 9124 -r true -f c:/UIStorage.bin)
Weight initializations:
Added .weightInit(Distribution) convenience/overload (previously: required .weightInit(WeightInit.DISTRIBUTION).dist(Distribution)) (Link)
WeightInit.NORMAL (for self-normalizing neural networks) (Link)
Ones, Identity weight initialization (Link)
Added new distributions (LogNormalDistribution, TruncatedNormalDistribution, OrthogonalDistribution, ConstantDistribution) which can be used for weight initialization (Link)
RNNs: Added ability to specify weight initialization for recurrent weights separately to "input" weights (Link)
Added layer alias: Convolution2D (ConvolutionLayer), Pooling1D (Subsampling1DLayer), Pooling2D (SubsamplingLayer) (Link)
Added Spark IteratorUtils - wraps a RecordReaderMultiDataSetIterator for use in Spark network training (Link)
CuDNN-supporting layers (ConvolutionLayer, etc) now warn the user if using CUDA without CuDNN (Link)
Binary cross entropy (LossBinaryXENT) now implements clipping (1e-5 to (1 - 1e-5) by default) to avoid numerical underflow/NaNs (Link)
SequenceRecordReaderDataSetIterator now supports multi-label regression (Link)
TransferLearning FineTuneConfiguration now has methods for setting training/inference workspace modes (Link)
IterationListener iterationDone method now reports both current iteration and epoch count; removed unnecessary invoke/invoked methods (Link)
Added MultiLayerNetwork.layerSize(int), ComputationGraph.layerSize(int)/layerSize(String) to easily determine size of layers (Link)
Added MultiLayerNetwork.toComputationGraph() method (Link)
Added NetworkUtils convenience methods to easily change the learning rate of an already initialized network (Link)
Added MultiLayerNetwork.save(File)/.load(File) and ComputationGraph.save(File)/.load(File) convenience methods (Link)
Added CheckpointListener to periodically save a copy of the model during training (every N iter/epochs, every T time units) (Link)
Added ComputationGraph output method overloads with mask arrays (Link)
New LossMultiLabel loss function for multi-label classification (Link)
New iterators, and iterator improvements:
Added additional score functions for early stopping (ROC metrics, full set of Evaluation/Regression metrics, etc) (Link)
Added additional ROC and ROCMultiClass evaluation overloads for MultiLayerNetwork and ComputationGraph (Link)
Clarified Evaluation.stats() output to refer to "Predictions" instead of "Examples" (former is more correct for RNNs) (Link)
EarlyStoppingConfiguration now supports Supplier<ScoreCalculator> for use with non-serializable score calculators (Link)
Improved ModelSerializer exceptions when trying to load a model via wrong method (i.e., try to load ComputationGraph via restoreMultiLayerNetwork) (Link)
Added SparkDataValidation utility methods to validate saved DataSet and MultiDataSet on HDFS or local (Link)
ModelSerializer: added restoreMultiLayerNetworkAndNormalizer and restoreComputationGraphAndNormalizer methods (Link)
ParallelInference now has output overloads with support for input mask arrays (Link)
Lombok is no longer included as a transitive dependency (Link)
Performance improvement for J7FileStatsStorage with large amount of history (Link)
Fixed UI layer sizes for variational autoencoder layers (Link)
UI Play servers switch to production (PROD) mode (Link)
Related to the above: users can now set play.crypto.secret system property to manually set the Play application secret; is randomly generated by default (Link).
SequenceRecordReaderDataSetIterator would apply preprocessor twice (Link)
Evaluation no-arg constructor could cause NaN evaluation metrics when used on Spark
CollectScoresIterationListener could recurse endlessly (Link)
Async(Multi)DataSetIterator calling reset() on underlying iterator could cause issues in some situations (Link)
In some cases, L2 regularization could be (incorrectly) applied to frozen layers (Link)
Logging fixes for NearestNeighboursServer (Link)
Memory optimization for BaseStatsListener (Link)
ModelGuesser fix for loading Keras models from streams (previously would fail) (Link)
Fix for incorrect condition in DuplicateToTimeSeriesVertex (Link)
Fix for getMemoryReport exception on some valid ComputationGraph networks (Link)
RecordReaderDataSetIterator when used with preprocessors could cause an exception under some circumstances (Link)
CnnToFeedForwardPreProcessor could silently reshape invalid input, as long as the input array length matches the expected length (Link)
ModelSerializer temporary files would not be deleted if JVM crashes; now are deleted immediately when no longer required (Link)
RecordReaderMultiDataSetIterator may not add mask arrays under some circumstances, when set to ALIGN_END mode (Link)
ConvolutionIterationListener previously produced an IndexOutOfBoundsException when all convolution layers are frozen (Link)
PrecisionRecallCurve.getPointAtRecall could return a point with a correct but sub-optimal precision when multiple points had identical recall (Link)
Setting dropout(0) on transfer learning FineTuneConfiguration did not remove dropout if present on existing layer (Link)
Under some rare circumstances, Spark evaluation could lead to a NullPointerException (Link)
ComputationGraph: disconnected vertices were not always detected in configuration validation (Link)
Activation layers would not always inherit the global activation function configuration (Link)
PerformanceListener is now serializable (Link)
ScoreIterationListener and PerformanceListener now report model iteration, not "iterations since listener creation" (Link)
Precision/recall curves cached values in ROC class may not be updated after merging ROC instances (Link)
ROC merging after evaluating a large number of examples may produce IllegalStateException (Link)
Added checks for invalid input indices to EmbeddingLayer (Link)
Fixed possible NPE when loading legacy (pre-0.9.0) model configurations from JSON (Link)
Fixed issues with EvaluationCalibration HTML export chart rendering (Link)
Fixed possible incorrect redering of UI/StatsStorage charts with J7FileStatsStorage when used with Spark training (Link)
MnistDataSetIterator would not always reliably detect and automatically fix/redownload on corrupted download data (Link)
Fixes to propagation of thread interruptions (Link)
Fixes for TSNE posting of data to UI for visualization (Link)
PerformanceListener now throws a useful exception (in constructor) on invalid frequency argument, instead of runtime ArithmeticException (Link)
RecordReader(Multi)DataSetIterator now throws more useful exceptions when Writable values are non-numerical (Link)
UI: Fixed possible character encoding issues for non-English languages when internationalization data .txt files are read from uber JARs (Link)
UI: Fixed UI incorrectly trying to parse non-DL4J UI resources when loading I18N data (Link)
Various threading fixes (Link)
Evaluation: no-arg methods (f1(), precion(), etc) now return single class value for binary case instead of macro-averaged value; clarify values in stats() method and javadoc (Link)
Early stopping training: TrainingListener opEpochStart/End (etc) methods were not being called correctly (Link)
Fixes issue where dropout was not always applied to input of RNN layers (Link)
ModelSerializer: improved validation/exceptions when reading from invalid/empty/closed streams (Link)
ParallelInference fixes:
fixes for variable size inputs (variable length time series, variable size CNN inputs) when using batch mode (Link)
fixes undelying model exceptions during output method are now properly propagated back to the user (Link)
fixes support for 'pre-batched' inputs (i.e., inputs where minibatch size is > 1) (Link)
Memory optimization for network weight initialization via in-place random ops (Link)
Fix for VariationalAutoencoder builder decoder layer size validation (Link)
Improved Kmeans throughputlink
Add RPForest to nearest neighbors link
Default training workspace mode has been switched to SEPARATE from NONE for MultiLayerNetwork and ComputationGraph (Link)
Behaviour change: fit(DataSetIterator) and similar methods no longer perform layerwise pretraining followed by backprop - only backprop is performed in these methods. For pretraining, use pretrain(DataSetIterator) and pretrain(MultiDataSetIterator) methods (Link)
Previously deprecated updater configuration methods (.learningRate(double), .momentum(double) etc) all removed
To configure learning rate: use .updater(new Adam(lr)) instead of .updater(Updater.ADAM).learningRate(lr)
To configure bias learning rate: use .biasUpdater(IUpdater) method
To configure learning rate schedules: use .updater(new Adam(ISchedule)) and similar
Updater configuration via enumeration (i.e., .updater(Updater)) has been deprecated; use .updater(IUpdater)
.regularization(boolean) config removed; functionality is now always equivalent to .regularization(true)
.useDropConnect(boolean) removed; use .weightNoise(new DropConnect(double)) instead
.iterations(int) method has been removed (was rarely used and confusing to users)
Multiple utility classes (in org.deeplearning4j.util) have been deprecated and/or moved to nd4j-common. Use same class names in nd4j-common org.nd4j.util instead.
DataSetIterators in DL4J have been moved from deeplearning4j-nn module to new deeplearning4j-datasets, deeplearning4j-datavec-iterators and deeplearning4j-utility-iterators modules. Packages/imports are unchanged; deeplearning4j-core pulls these in as transitive dependencies hence no user changes should be required in most cases (Link)
Previously deprecated .activation(String) has been removed; use .activation(Activation) or .activation(IActivation) instead
Layer API change: Custom layers may need to implement applyConstraints(int iteration, int epoch) method
Parameter initializer API change: Custom parameter initializers may need to implement isWeightParam(String) and isBiasParam(String) methods
RBM (Restricted Boltzmann Machine) layers have been removed entirely. Consider using VariationalAutoencoder layers as a replacement (Link)
GravesBidirectionalLSTM has been deprecated; use new Bidirectional(Bidirectional.Mode.ADD, new GravesLSTM.Builder()....build())) instead
Previously deprecated WordVectorSerializer methods have now been removed (Link)
Removed deeplearning4j-ui-remote-iterationlisteners module and obsolete RemoteConvolutionalIterationListener (Link)
Performance on some networks types may be reduced on CUDA compared to 0.9.1 (with workspaces configured). This will be addressed in the next release
Some issues have been noted with FP16 support on CUDA (Link)
Keras 2 support, keeping backward compatibility for keras 1
Keras 2 and 1 import use exact same API and are inferred by DL4J
Keras unit test coverage increased by 10x, many more real-world integration tests
Unit tests for importing and checking layer weights
Leaky ReLU, ELU, SELU support for model import
All Keras layers can be imported with optional bias terms
Old deeplearning4j-keras module removed, old "Model" API removed
All Keras initializations (Lecun normal, Lecun uniform, ones, zeros, Orthogonal, VarianceScaling, Constant) supported
1D convolution and pooling supported in DL4J and Keras model import
Atrous Convolution 1D and 2D layers supported in Keras model import
1D Zero padding layers supported
Keras constraints module fully supported in DL4J and model import
Upsampling 1D and 2D layers in DL4J and Keras model import (including GAN examples in tests)
Most merge modes supported in Keras model import, Keras 2 Merge layer API supported
Separable Convolution 2D layer supported in DL4J and Keras model import
Deconvolution 2D layer supported in DL4J and Keras model import
Full support of Keras noise layers on import (Alpha dropout, Gaussian dropout and noise)
Support for SimpleRNN layer in Keras model import
Support for Bidirectional layer wrapper Keras model import
Addition of LastTimestepVertex in DL4J to support return_sequences=False for Keras RNN layers.
DL4J support for recurrent weight initializations and Keras import integration.
SpaceToBatch and BatchToSpace layers in DL4J for better YOLO support, plus end-to-end YOLO Keras import test.
Cropping2D support in DL4J and Keras model import
In 0.9.1 deprecated Model and ModelConfiguration have been permanently removed. Use KerasModelImport instead, which is now the only entry point for Keras model import.
Embedding layer: In DL4J the output of an embedding layer is 2D by default, unless preprocessors are specified. In Keras the output is always 3D, but depending on specified parameters can be interpreted as 2D. This often leads to difficulties when importing Embedding layers. Many cases have been covered and issues fixed, but inconsistencies remain.
Batchnormalization layer: DL4J's batch normalization layer is much more restrictive (in a good way) than Keras' version of it. For instance, DL4J only allows to normalize spatial dimensions for 4D convolutional inputs, while in Keras any axis can be used for normalization. Depending on the dimension ordering (NCHW vs. NHWC) and the specific configuration used by a Keras user, this can lead to expected (!) and unexpected import errors.
Support for importing a Keras model for training purposes in DL4J (enforceTrainingConfig == true) is still very limited and will be tackled properly for the next release.
Keras Merge layers: seem to work fine with the Keras functional API, but have issues when used in a Sequential model.
Reshape layers: can be somewhat unreliable on import. DL4J rarely has a need to explicitly reshape input beyond (inferred) standard input preprocessors. In Keras, Reshape layers are used quite often. Mapping the two paradigms can be difficult in edge cases.
Hundreds of new operations added
New DifferentialFunction api with automatic differentiation (see samediff section) Link
Technology preview of tensorflow import added (supports 1.4.0 and up)
Apache Arrow serialization added supporting new tensor API Link
Add support for AVX/AVX2 and AVX-512 instruction sets for Windows/Linux for nd4j-native backend Link
nVidia CUDA 8/9.0/9.1 now supported
Worskpaces improvements were introduced to ensure safety: SCOPE_PANIC profiling mode is enabled by default
FlatBuffers support for INDArray serde
Support for auto-broadcastable operations was added
libnd4j, underlying c++ library, got functionality boost and now offers: NDArray class, Graph class, and can be used as standalone library or executable.
Convolution-related ops now support NHWC in addition to NCHW data format.
Accumulation ops now have option to keep reduced dimensions.
Not all op gradients implemented for automatic differentiation
Vast majority of new operations added in 1.0.0-alpha do NOT use GPU yet.
Initial tech preview Link
Control flow is supported with IF and WHILE primitives.
Alpha release of SameDiff auto-differentiation engine for ND4J.
Two execution modes available: Java-driven execution, and Native execution for serialized graphs.
SameDiff graphs can be serialized using FlatBuffers
Building and running computation graphs build from SameDiff operations.
Graphs can run forward pass on input data and compute gradients for the backward pass.
Already supports many high-level layers, like dense layers, convolutions (1D-3D) deconvolutions, separable convolutions, pooling and upsampling, batch normalization, local response normalization, LSTMs and GRUs.
In total there are about 350 SameDiff operations available, including many basic operations used in building complex graphs.
Supports rudimentary import of TensorFlow and ONNX graphs for inference.
TFOpTests is a dedicated project for creating test resources for TensorFlow import.
Vast majority of new operations added in 1.0.0-alpha do NOT use GPU yet.
While many of the widely used base operations and high-level layers used in practice are supported, op coverage is still limited. Goal is to achieve feature parity with TensorFlow and fully support import for TF graphs.
Some of the existing ops do not have a backward pass implemented (called doDiff in SameDiff).
Added LocalTransformExecutor for single machine execution (without Spark dependency) (Link)
Added ArrowRecordReader (for reading Apache Arrow format data) (Link)
Added RecordMapper class for conversion between RecordReader and RecordWriter (Link)
Added BoxImageTransform - an ImageTransform that either crops or pads without changing aspect ratio (Link)
Added CSVVariableSlidingWindowRecordReader (Link)
ImageRecordReader: supports regression use cases for labels (previously: only classification) (Link)
DataAnalysis/AnalyzeSpark now includes quantiles (via t-digest) (Link)
Added AndroidNativeImageLoader.asBitmap(), Java2DNativeImageLoader.asBufferedImage() (Link)
StringToTimeTransform will con try to guess time format if format isn't provided (Link)
Improved performance for NativeImageLoader on Android (Link)
Added BytesWritable (Writable for byte[] data) (Link)
Added TranformProcess.inferCategories methods to auto-infer categories from a RecordReader (Link)
Lombok is no longer included as a transitive dependency (Link)
MapFileRecordReader and MapFileSequenceRecordReader can handle empty partitions/splits for multi-part map files (Link)
Writables: equality semantics have been changed: for example, now DoubleWritable(1.0) is equal to IntWritable(1) (Link)
NumberedFileInputSplit now supports leading zeros (Link)
CSVSparkTransformServer and ImageSparkTransformServer Play severs changed to production mode (Link)
Fix for JSON subtype info for FloatMetaData (Link)
Serialization fixes for JacksonRecordReader, RegexSequenceRecordReader (Link)
Added RecordReader.resetSupported() method (Link)
SVMLightRecordReader now implements nextRecord() method (Link)
Fix for custom reductions when using conditions (Link)
Remove use of backported java.util.functions; use ND4J functions API instead (Link)
Fix for transforms data quality analysis for time columns (Link)
Many of the util classes (in org.datavec.api.util mainly) have been deprecated or removed; use equivalently named util clases in nd4j-common module (Link)
RecordReader.next(int) method now returns List<List<Writable>> for batches, not List<Writable>. See also NDArrayRecordBatch
RecordWriter and SequenceRecordWriter APIs have been updated with multiple new methods
As per DL4J API changes: Updater configuration options (learning rate, momentum, epsilon, rho etc) have been moved to ParameterSpace instead. Updater spaces (AdamSpace, AdaGradSpace etc) introduced (Link)
As per DL4J API changes: Dropout configuration is now via ParameterSpace<IDropout>, DropoutSpace introduced (Link)
RBM layer spaces removed (Link)
ComputationGraphSpace: added layer/vertex methods with overloads for preprocessors (Link)
Added support to specify 'fixed' layers using DL4J layers directly (instead of using LayerSpaces, even for layers without hyperparameters) (Link)
Added LogUniformDistribution (Link)
Improvements to score functions; added ROC score function (Link)
Learning rate schedule support added (Link)
Add math ops for ParameterSpace<Double> and ParameterSpace<Integer> (Link)
Improved logging for failed task execution (Link)
Fix for UI JSON serialization (Link)
Rename saved model file to model.bin (Link)
Fix threading issues with non thread-safe candidates / parameter spaces (Link)
Lombok is no longer included as a transitive dependency (Link)
As per DL4J updater API changes: old updater configuration (learningRate, momentum, etc) methods have been removed. Use .updater(IUpdater) or .updater(ParameterSpace<IUpdater>) methods instead
Add support for LSTM layer to A3C
Fix A3C to make it actually work using new ActorCriticLoss and correct use of randomness
Fix cases when QLearning would fail (non-flat input, incomplete serialization, incorrect normalization)
Fix logic of HistoryProcessor with async algorithms and failures when preprocessing images
Tidy up and correct the output of statistics, also allowing the use of IterationListener
Fix issues preventing efficient execution with CUDA
Provide access to more of the internal structures with NeuralNet.getNeuralNetworks(), Policy.getNeuralNet(), and convenience constructors for Policy
Add MDPs for ALE (Arcade Learning Environment) and MALMO to support Atari games and Minecraft
Update MDP for Doom to allow using the latest version of VizDoom
First release of ScalNet Scala API, which closely resembles Keras' API.
Can be built with sbt and maven.
Supports both Keras inspired Sequential models, corresponding to DL4J's MultiLayerNetwork, and Model, corresponding to ComputationGraph.
Project structure is closely aligned to both DL4J model-import module and Keras.
Supports the following layers: Convolution2D, Dense, EmbeddingLayer, AvgPooling2D, MaxPooling2D, GravesLSTM, LSTM, Bidirectional layer wrapper, Flatten, Reshape. Additionally, DL4J OutputLayers are supported.
Scala 2.12 support