1.0.0-beta

Highlights - 1.0.0-beta Release

• Performance and memory optimizations for DL4J

Deeplearning4J

Deeplearning4J: New Features

• New or enhanced layers:

  • Added Cropping1D layer Link

  • Added Convolution3D, Cropping3D, UpSampling3D, ZeroPadding3D, Subsampling3D layers (all with Keras import support): Link Link

  • Added EmbeddingSequenceLayer (EmbeddingLayer for time series) Link

  • Added OCNNOutputLayer (one-class neural network) - implementation of this paper - Link

  • Added FrozenLayerWithBackprop layer Link

  • Added DepthwiseConvolution2D layer Link

• Added ComputationGraph.output(DataSetIterator) method Link

• Added MultiLayerNetwork/ComputationGraph.layerInputSize methods Link Link

• Added SparkComputationGraph.feedForwardWithKey overload with feature mask support Link

• Added MultiLayerNetwork.calculateGradients method (for easily getting parameter and input gradients, for example for some model interpretabilithy approaches) Link Link

• Added support to get input/activation types for each layer from configuration: ComputationGraphConfiguration.getLayerActivationTypes(InputType...), ComputationGraphConfiguration.GraphBuilder.getLayerActivationTypes(), NeuralNetConfiguration.ListBuilder.getLayerActivationTypes(), MultiLayerConfiguration.getLayerActivationTypes(InputType) methods Link

• Evaluation.stats() now prints confusion matrix in easier to read matrix format, rather than list format Link

• Added ModelSerializer.addObjectToFile, .getObjectFromFile and .listObjectsInFile for storing arbitrary Java objects in same file as saved network Link

• Added SpatialDropout support (with Keras import support) Link

• Added MultiLayerNetwork/ComputationGraph.fit((Multi)DataSetIterator, int numEpochs) overloads Link

• Added performance (hardware) listeners: SystemInfoPrintListener and SystemInfoFilePrintListener Link

Deeplearning4J: Bug Fixes and Optimizations

• Performance and memory optimizations via optimizations of internal use of workspaces Link

• Reflections library has entirely been removed from DL4J and is no longer required for custom layer serialization/deserialization Link, Link

  • Fixes issues with custom and some Keras import layers on Android

• RecordReaderMultiDataSetIterator will no longer try to convert unused columns to numerical values Link

• Added new model zoo models:

  • (to do)

• Fixes for Android compilation (removed duplicate classes, aligned versions, removed some dependencies) Link Link Link

• Fix for RecordReaderMulitDataSetIterator where output could be incorrect for some constructors Link

• Non-frozen layers before a frozen layer will no longer be skipped during backprop (useful for GANs and similar architectures) Link Link

• Fixed issue where ComputationGraph topological sort may not be consistent on all platforms; could sometimes break ComputationGraphs (with multiple valid topological orderings) trained on PC and deployed on Android Link

• Fixed issue with CuDNN batch norm using 1-decay instead of decay Link

• deeplearning4j-cuda no longer throws exceptions if present on classpath with nd4j-native backend set to higher priority Link

• Added RNG control for CifarDataSetIterator Link

• WordVectorSerializer now deletes temp files immediately once done Link

Deeplearning4J: API Changes (Transition Guide): 1.0.0-alpha to 1.0.0-beta

• WorkspaceMode.SINGLE and SEPARATE have been deprecated; use WorkspaceMode.ENABLED instead

• Internal layer API changes: custom layers will need to be updated to the new Layer API - see built-in layers or custom layer example

• Custom layers etc in pre-1.0.0-beta JSON (ModelSerializer) format need to be registered before they can be deserialized due to JSON format change. Built-in layers and models saved in 1.0.0-beta or later do not require this. Use NeuralNetConfiguration.registerLegacyCustomClassesForJSON(Class) for this purpose

• IterationListener has been deprecated in favor of TrainingListener. For existing custom listeners, switch from implements TrainingListener to extends BaseTrainingListener Link

• ExistingDataSetIterator has been deprecated; use fit(DataSetIterator, int numEpochs) method instead

Deelpearning4J: 1.0.0-beta Known Issues

• ComputationGraph TrainingListener onEpochStart and onEpochEnd methods are not being called correctly

• DL4J Zoo Model FaceNetNN4Small2 model configuration is incorrect, causing issues during forward pass

• Early stopping score calculators with values thar should be maximized (accuracy, f1 etc) are not working properly (values are minimized not maximized). Workaround: override ScoreCalculator.calculateScore(...) and return 1.0 - super.calculateScore(...).

Deeplearing4J: Keras Import

Deeplearning4J: Keras Import - API Changes (Transition Guide): 1.0.0-alpha to 1.0.0-beta

ND4J

ND4J: New Features

ND4J: Known Issues

• Not all op gradients implemented for automatic differentiation

• Vast majority of new operations added in 1.0.0-beta do NOT use GPU yet.

ND4J: API Changes (Transition Guide): 1.0.0-alpha to 1.0.0-beta

DataVec

DataVec: New Features

• ImageRecordReader now logs number of inferred label classes (to reduce risk of users missing a problem if something is misconfigured) Link

• Added AnalyzeSpark.getUnique overload for multiple columns Link

• Added performance/timing module Link

DataVec: Optimizations and Bug Fixes

• Reduced ImageRecordReader garbage generation via buffer reuse Link

• Fixes for Android compilation (aligned versions, removed some dependencies) Link Link

• Removed Reflections library use in DataVec Link

• Fix for TransformProcessRecordReader batch support Link

• Fix for TransformProcessRecordReader with filter operations Link

• Fixed issue with ImageRecordReader/ParentPathLabelGenerator incorrectly filtering directories containing . character(s) Link

• ShowImageTransform now initializes frame lazily to avoid blank windows Link

DataVec: API Changes (Transition Guide): 1.0.0-alpha to 1.0.0-beta

• DataVec ClassPathResource has been deprecated; use nd4j-common version instead Link

Arbiter

Arbiter: New Features

• Added LayerSpace for OCNN (one-class neural network)

Arbiter: Fixes

• Fixed timestamp issue that could cause incorrect rendering of first model's results in UI Link

• Execution now waits for last model(s) to complete before returning when a termination condition is hit Link

• As per DL4J etc: use of Reflections library has been removed entirely from Arbiter Link

• Remove use of Eclipse Collections library due to issues with Android compilation Link

• Improved cleanup of completed models to reduce maximum memory requirements for training Link