1.0.0-alpha
- ND4J: Added SameDiff - Java automatic differentiation library (alpha release) with Tensorflow import (technology preview) and hundreds of new operations
- ND4J: Added CUDA 9.0 and 9.1 support (with cuDNN), dropped support for CUDA 7.5, continued support for CUDA 8.0
- ND4J: Native binaries (nd4j-native on Maven Central) now ship with AVX/AVX2/AVX-512 support (Windows/Linux)
- DL4J: Large number of new layers and API improvements
- DL4J: Keras 2.0 import support
- Layers (new and enhanced)
- Added Yolo2OutputLayer CNN layer for object detection (Link). See also DataVec's ObjectDetectionRecordReader
- Adds support for 'no bias' layers via
hasBias(boolean)config (DenseLayer, EmbeddingLayer, OutputLayer, RnnOutputLayer, CenterLossOutputLayer, ConvolutionLayer, Convolution1DLayer). EmbeddingLayer now defaults to no bias (Link) - Adds support for dilated convolutions (aka 'atrous' convolutions) - ConvolutionLayer, SubsamplingLayer, and 1D versions there-of. (Link)
- ElementWiseVertex now (additionally) supports
AverageandMaxmodes in addition to Add/Subtract/Product (Link) - Added Deconvolution2D layer (aka transpose convolution, fractionally strided convolution layer) (Link)
- Added CnnLossLayer - no-parameter CNN output layer for use cases such as segmentation, denoising, etc. (Link)
- Added LastTimeStep wrapper layer (wraps a RNN layer to get last time step, accounting for masking if present) (Link)
- Added MaskLayer utility layer that simply zeros out activations on forward pass when a mask array is present (Link)
- Added alpha-version (not yet stable) SameDiff layer support to DL4J (Note: forward pass, CPU only for now)(Link)
- Added parameter constraints API (LayerConstraint interface), and MaxNormConstraint, MinMaxNormConstraint, NonNegativeConstraint, UnitNormConstraint implementations (Link)
- Added ISchedule interface; added Exponential, Inverse, Map, Poly, Sigmoid and Step schedule implementations (Link)
- Added support for both iteration-based and epoch-based schedules via ISchedule. Also added support for custom (user defined) schedules
- Learning rate schedules are configured on the updaters, via the
.updater(IUpdater)method
- Added dropout API (IDropout - previously dropout was available but not a class); added Dropout, AlphaDropout (for use with self-normalizing NNs), GaussianDropout (multiplicative), GaussianNoise (additive). Added support for custom dropout types (Link)
- Added weight/parameter noise API (IWeightNoise interface); added DropConnect and WeightNoise (additive/multiplicative Gaussian noise) implementations (Link); dropconnect and dropout can now be used simultaneously
- Adds ComputationGraphConfiguration GraphBuilder
.layer(String, Layer, String...)alias for.addLayer(String, Layer, String...) - Layer index no longer required for MultiLayerConfiguration ListBuilder (i.e.,
.list().layer(<layer>)can now be used for configs) (Link) - Added
MultiLayerNetwork.summary(InputType)andComputationGraph.summary(InputType...)methods (shows layer and activation size information) (Link) - MultiLayerNetwork, ComputationGraph and layerwise trainable layers now track the number of epochs (Link)
- Added deeplearning4j-ui-standalone module: uber-jar for easy launching of UI server (usage:
java -jar deeplearning4j-ui-standalone-1.0.0-alpha.jar -p 9124 -r true -f c:/UIStorage.bin) - Weight initializations:
- Added
.weightInit(Distribution)convenience/overload (previously: required.weightInit(WeightInit.DISTRIBUTION).dist(Distribution)) (Link) - Added new distributions (LogNormalDistribution, TruncatedNormalDistribution, OrthogonalDistribution, ConstantDistribution) which can be used for weight initialization (Link)
- RNNs: Added ability to specify weight initialization for recurrent weights separately to "input" weights (Link)
- Added layer alias: Convolution2D (ConvolutionLayer), Pooling1D (Subsampling1DLayer), Pooling2D (SubsamplingLayer) (Link)
- Added Spark IteratorUtils - wraps a RecordReaderMultiDataSetIterator for use in Spark network training (Link)
- CuDNN-supporting layers (ConvolutionLayer, etc) now warn the user if using CUDA without CuDNN (Link)
- Binary cross entropy (LossBinaryXENT) now implements clipping (1e-5 to (1 - 1e-5) by default) to avoid numerical underflow/NaNs (Link)
- TransferLearning FineTuneConfiguration now has methods for setting training/inference workspace modes (Link)
- IterationListener iterationDone method now reports both current iteration and epoch count; removed unnecessary invoke/invoked methods (Link)
- Added MultiLayerNetwork.layerSize(int), ComputationGraph.layerSize(int)/layerSize(String) to easily determine size of layers (Link)
- Added NetworkUtils convenience methods to easily change the learning rate of an already initialized network (Link)
- Added MultiLayerNetwork.save(File)/.load(File) and ComputationGraph.save(File)/.load(File) convenience methods (Link)
- Added CheckpointListener to periodically save a copy of the model during training (every N iter/epochs, every T time units) (Link)
- New iterators, and iterator improvements:
- Added FileDataSetIterator, FileMultiDataSetIterator for flexibly iterating over directories of saved (Multi)DataSet objects (Link)
- Added additional score functions for early stopping (ROC metrics, full set of Evaluation/Regression metrics, etc) (Link)
- Added additional ROC and ROCMultiClass evaluation overloads for MultiLayerNetwork and ComputationGraph (Link)
- Clarified Evaluation.stats() output to refer to "Predictions" instead of "Examples" (former is more correct for RNNs) (Link)
- EarlyStoppingConfiguration now supports
Supplier<ScoreCalculator>for use with non-serializable score calculators (Link) - Improved ModelSerializer exceptions when trying to load a model via wrong method (i.e., try to load ComputationGraph via restoreMultiLayerNetwork) (Link)
- Added SparkDataValidation utility methods to validate saved DataSet and MultiDataSet on HDFS or local (Link)
- ModelSerializer: added restoreMultiLayerNetworkAndNormalizer and restoreComputationGraphAndNormalizer methods (Link)
- Related to the above: users can now set
play.crypto.secretsystem property to manually set the Play application secret; is randomly generated by default (Link). - Evaluation no-arg constructor could cause NaN evaluation metrics when used on Spark
- Async(Multi)DataSetIterator calling reset() on underlying iterator could cause issues in some situations (Link)
- RecordReaderDataSetIterator when used with preprocessors could cause an exception under some circumstances (Link)
- CnnToFeedForwardPreProcessor could silently reshape invalid input, as long as the input array length matches the expected length (Link)
- ModelSerializer temporary files would not be deleted if JVM crashes; now are deleted immediately when no longer required (Link)
- RecordReaderMultiDataSetIterator may not add mask arrays under some circumstances, when set to ALIGN_END mode (Link)
- ConvolutionIterationListener previously produced an IndexOutOfBoundsException when all convolution layers are frozen (Link)
- PrecisionRecallCurve.getPointAtRecall could return a point with a correct but sub-optimal precision when multiple points had identical recall (Link)
- Setting dropout(0) on transfer learning FineTuneConfiguration did not remove dropout if present on existing layer (Link)
- ScoreIterationListener and PerformanceListener now report model iteration, not "iterations since listener creation" (Link)
- Precision/recall curves cached values in ROC class may not be updated after merging ROC instances (Link)
- Fixed possible incorrect redering of UI/StatsStorage charts with J7FileStatsStorage when used with Spark training (Link)
- MnistDataSetIterator would not always reliably detect and automatically fix/redownload on corrupted download data (Link)
- PerformanceListener now throws a useful exception (in constructor) on invalid frequency argument, instead of runtime ArithmeticException (Link)
- RecordReader(Multi)DataSetIterator now throws more useful exceptions when Writable values are non-numerical (Link)
- UI: Fixed possible character encoding issues for non-English languages when internationalization data .txt files are read from uber JARs (Link)
- Evaluation: no-arg methods (f1(), precion(), etc) now return single class value for binary case instead of macro-averaged value; clarify values in stats() method and javadoc (Link)
- Early stopping training: TrainingListener opEpochStart/End (etc) methods were not being called correctly (Link)
- ModelSerializer: improved validation/exceptions when reading from invalid/empty/closed streams (Link)
- ParallelInference fixes:
- Default training workspace mode has been switched to SEPARATE from NONE for MultiLayerNetwork and ComputationGraph (Link)
- Behaviour change:
fit(DataSetIterator)and similar methods no longer perform layerwise pretraining followed by backprop - only backprop is performed in these methods. For pretraining, usepretrain(DataSetIterator)andpretrain(MultiDataSetIterator)methods (Link) - Previously deprecated updater configuration methods (
.learningRate(double),.momentum(double)etc) all removed- To configure learning rate: use
.updater(new Adam(lr))instead of.updater(Updater.ADAM).learningRate(lr) - To configure bias learning rate: use
.biasUpdater(IUpdater)method - To configure learning rate schedules: use
.updater(new Adam(ISchedule))and similar
- Updater configuration via enumeration (i.e.,
.updater(Updater)) has been deprecated; use.updater(IUpdater) .regularization(boolean)config removed; functionality is now always equivalent to.regularization(true).useDropConnect(boolean)removed; use.weightNoise(new DropConnect(double))instead.iterations(int)method has been removed (was rarely used and confusing to users)- Multiple utility classes (in
org.deeplearning4j.util) have been deprecated and/or moved to nd4j-common. Use same class names in nd4j-commonorg.nd4j.utilinstead. - DataSetIterators in DL4J have been moved from deeplearning4j-nn module to new deeplearning4j-datasets, deeplearning4j-datavec-iterators and deeplearning4j-utility-iterators modules. Packages/imports are unchanged; deeplearning4j-core pulls these in as transitive dependencies hence no user changes should be required in most cases (Link)
- Previously deprecated
.activation(String)has been removed; use.activation(Activation)or.activation(IActivation)instead - Layer API change: Custom layers may need to implement
applyConstraints(int iteration, int epoch)method - Parameter initializer API change: Custom parameter initializers may need to implement
isWeightParam(String)andisBiasParam(String)methods - RBM (Restricted Boltzmann Machine) layers have been removed entirely. Consider using VariationalAutoencoder layers as a replacement (Link)
- GravesBidirectionalLSTM has been deprecated; use
new Bidirectional(Bidirectional.Mode.ADD, new GravesLSTM.Builder()....build()))instead - Removed deeplearning4j-ui-remote-iterationlisteners module and obsolete RemoteConvolutionalIterationListener (Link)
- Performance on some networks types may be reduced on CUDA compared to 0.9.1 (with workspaces configured). This will be addressed in the next release
- Keras 2 support, keeping backward compatibility for keras 1
- Keras 2 and 1 import use exact same API and are inferred by DL4J
- Keras unit test coverage increased by 10x, many more real-world integration tests
- Unit tests for importing and checking layer weights
- Leaky ReLU, ELU, SELU support for model import
- All Keras layers can be imported with optional bias terms
- Old deeplearning4j-keras module removed, old "Model" API removed
- All Keras initializations (Lecun normal, Lecun uniform, ones, zeros, Orthogonal, VarianceScaling, Constant) supported
- 1D convolution and pooling supported in DL4J and Keras model import
- Atrous Convolution 1D and 2D layers supported in Keras model import
- 1D Zero padding layers supported
- Keras constraints module fully supported in DL4J and model import
- Upsampling 1D and 2D layers in DL4J and Keras model import (including GAN examples in tests)
- Most merge modes supported in Keras model import, Keras 2 Merge layer API supported
- Separable Convolution 2D layer supported in DL4J and Keras model import
- Deconvolution 2D layer supported in DL4J and Keras model import
- Full support of Keras noise layers on import (Alpha dropout, Gaussian dropout and noise)
- Support for SimpleRNN layer in Keras model import
- Support for Bidirectional layer wrapper Keras model import
- Addition of LastTimestepVertex in DL4J to support return_sequences=False for Keras RNN layers.
- DL4J support for recurrent weight initializations and Keras import integration.
- SpaceToBatch and BatchToSpace layers in DL4J for better YOLO support, plus end-to-end YOLO Keras import test.
- Cropping2D support in DL4J and Keras model import
- In 0.9.1 deprecated
ModelandModelConfigurationhave been permanently removed. Use KerasModelImport instead, which is now the only entry point for Keras model import.
- Embedding layer: In DL4J the output of an embedding layer is 2D by default, unless preprocessors are specified. In Keras the output is always 3D, but depending on specified parameters can be interpreted as 2D. This often leads to difficulties when importing Embedding layers. Many cases have been covered and issues fixed, but inconsistencies remain.
- Batchnormalization layer: DL4J's batch normalization layer is much more restrictive (in a good way) than Keras' version of it. For instance, DL4J only allows to normalize spatial dimensions for 4D convolutional inputs, while in Keras any axis can be used for normalization. Depending on the dimension ordering (NCHW vs. NHWC) and the specific configuration used by a Keras user, this can lead to expected (!) and unexpected import errors.
- Support for importing a Keras model for training purposes in DL4J (enforceTrainingConfig == true) is still very limited and will be tackled properly for the next release.
- Keras Merge layers: seem to work fine with the Keras functional API, but have issues when used in a Sequential model.
- Reshape layers: can be somewhat unreliable on import. DL4J rarely has a need to explicitly reshape input beyond (inferred) standard input preprocessors. In Keras, Reshape layers are used quite often. Mapping the two paradigms can be difficult in edge cases.
- Hundreds of new operations added
- Technology preview of tensorflow import added (supports 1.4.0 and up)
- Add support for AVX/AVX2 and AVX-512 instruction sets for Windows/Linux for nd4j-native backend Link
- nVidia CUDA 8/9.0/9.1 now supported
- Worskpaces improvements were introduced to ensure safety: SCOPE_PANIC profiling mode is enabled by default
- FlatBuffers support for INDArray serde
- Support for auto-broadcastable operations was added
- libnd4j, underlying c++ library, got functionality boost and now offers: NDArray class, Graph class, and can be used as standalone library or executable.
- Convolution-related ops now support NHWC in addition to NCHW data format.
- Accumulation ops now have option to keep reduced dimensions.
- Not all op gradients implemented for automatic differentiation
- Vast majority of new operations added in 1.0.0-alpha do NOT use GPU yet.
- Control flow is supported with IF and WHILE primitives.
- Two execution modes available: Java-driven execution, and Native execution for serialized graphs.
- SameDiff graphs can be serialized using FlatBuffers
- Building and running computation graphs build from SameDiff operations.
- Graphs can run forward pass on input data and compute gradients for the backward pass.
- Already supports many high-level layers, like dense layers, convolutions (1D-3D) deconvolutions, separable convolutions, pooling and upsampling, batch normalization, local response normalization, LSTMs and GRUs.
- In total there are about 350 SameDiff operations available, including many basic operations used in building complex graphs.
- Vast majority of new operations added in 1.0.0-alpha do NOT use GPU yet.
- While many of the widely used base operations and high-level layers used in practice are supported, op coverage is still limited. Goal is to achieve feature parity with TensorFlow and fully support import for TF graphs.
- Some of the existing ops do not have a backward pass implemented (called
doDiffin SameDiff).
- Many of the util classes (in
org.datavec.api.utilmainly) have been deprecated or removed; use equivalently named util clases in nd4j-common module (Link) - RecordReader.next(int) method now returns
List<List<Writable>>for batches, notList<Writable>. See also NDArrayRecordBatch - RecordWriter and SequenceRecordWriter APIs have been updated with multiple new methods
- As per DL4J API changes: Updater configuration options (learning rate, momentum, epsilon, rho etc) have been moved to ParameterSpace instead. Updater spaces (AdamSpace, AdaGradSpace etc) introduced (Link)
- As per DL4J API changes: Dropout configuration is now via
ParameterSpace<IDropout>, DropoutSpace introduced (Link) - Added support to specify 'fixed' layers using DL4J layers directly (instead of using LayerSpaces, even for layers without hyperparameters) (Link)
- As per DL4J updater API changes: old updater configuration (learningRate, momentum, etc) methods have been removed. Use
.updater(IUpdater)or.updater(ParameterSpace<IUpdater>)methods instead
- Add support for LSTM layer to A3C
- Fix A3C to make it actually work using new
ActorCriticLossand correct use of randomness - Fix cases when
QLearningwould fail (non-flat input, incomplete serialization, incorrect normalization) - Fix logic of
HistoryProcessorwith async algorithms and failures when preprocessing images - Tidy up and correct the output of statistics, also allowing the use of
IterationListener - Fix issues preventing efficient execution with CUDA
- Provide access to more of the internal structures with
NeuralNet.getNeuralNetworks(),Policy.getNeuralNet(), and convenience constructors forPolicy - Add MDPs for ALE (Arcade Learning Environment) and MALMO to support Atari games and Minecraft
- Update MDP for Doom to allow using the latest version of VizDoom
- Can be built with sbt and maven.
- Supports both Keras inspired Sequential models, corresponding to DL4J's
MultiLayerNetwork, and Model, corresponding toComputationGraph. - Project structure is closely aligned to both DL4J model-import module and Keras.
- Supports the following layers: Convolution2D, Dense, EmbeddingLayer, AvgPooling2D, MaxPooling2D, GravesLSTM, LSTM, Bidirectional layer wrapper, Flatten, Reshape. Additionally, DL4J OutputLayers are supported.
- Scala 2.12 support
Last modified 9mo ago