Deeplearning4j
Community ForumND4J JavadocDL4J Javadoc
EN 1.0.0-M2.1
EN 1.0.0-M2.1
  • Deeplearning4j Suite Overview
  • Release Notes
    • 1.0.0-M2
    • 1.0.0-M2.1
    • 1.0.0-M1.1
    • 1.0.0-M1
    • 1.0.0-beta7
    • 1.0.0-beta6
    • 1.0.0-beta5
    • 1.0.0-beta4
    • 1.0.0-beta3
    • 1.0.0-beta2
    • 1.0.0-beta
    • 1.0.0-alpha
    • 0.9.1
    • 0.9.0
    • 0.8.0
    • 0.7.2
    • 0.7.1
    • 0.7.0
    • 0.6.0
    • 0.5.0
    • 0.4.0
    • 1.00-M2.2
  • Multi-Project
    • Tutorials
      • Beginners
      • Quickstart
    • How To Guides
      • Import in to your favorite IDE
      • Contribute
        • Eclipse Contributors
      • Developer Docs
        • Github Actions/Build Infra
        • Javacpp
        • Release
        • Testing
      • Build From Source
      • Benchmark
      • Beginners
    • Reference
      • Examples Tour
      • Core Concepts
      • Trouble Shooting
    • Explanation
      • Required Dependencies
      • The core workflow
      • Configuration
        • Backends
          • Performance Issues
          • CPU
          • Cudnn
        • Memory
          • Workspaces
      • Build Tools
      • Snapshots
      • Maven
  • Deeplearning4j
    • Tutorials
      • Quick Start
      • Language Processing
        • Doc2Vec
        • Sentence Iterator
        • Tokenization
        • Vocabulary Cache
    • How To Guides
      • Custom Layers
      • Keras Import
        • Functional Models
        • Sequential Models
        • Custom Layers
        • Keras Import API Overview
          • Advanced Activations
          • Convolutional Layers
          • Core Layers
          • Embedding Layers
          • Local Layers
          • Noise Layers
          • Normalization Layers
          • Pooling Layers
          • Recurrent Layers
          • Wrapper Layers
        • Supported Features Overview
          • Activations
          • Constraints
          • Initializers
          • Losses
          • Optimizers
          • Regularizers
      • Tuning and Training
        • Visualization
        • Troubleshooting Training
        • Early Stopping
        • Evaluation
        • Transfer Learning
    • Reference
      • Model Zoo
        • Zoo Models
      • Activations
      • Auto Encoders
      • Computation Graph
      • Convolutional Layers
      • DataSet Iterators
      • Layers
      • Model Listeners
      • Saving and Loading Models
      • Multi Layer Network
      • Recurrent Layers
      • Updaters/Optimizers
      • Vertices
      • Word2vec/Glove/Doc2Vec
    • Explanation
  • datavec
    • Tutorials
      • Overview
    • How To Guides
    • Reference
      • Analysis
      • Conditions
      • Executors
      • Filters
      • Normalization
      • Operations
      • Transforms
      • Readers
      • Records
      • Reductions
      • Schemas
      • Serialization
      • Visualization
    • Explanation
  • Nd4j
    • Tutorials
      • Quickstart
    • How To Guides
      • Other Framework Interop
        • Tensorflow
        • TVM
        • Onnx
      • Matrix Manipulation
      • Element wise Operations
      • Basics
    • Reference
      • Op Descriptor Format
      • Tensor
      • Syntax
    • Explanation
  • Samediff
    • Tutorials
      • Quickstart
    • How To Guides
      • Importing Tensorflow
      • Adding Operations
        • codegen
    • Reference
      • Operation Namespaces
        • Base Operations
        • Bitwise
        • CNN
        • Image
        • LinAlg
        • Loss
        • Math
        • NN
        • Random
        • RNN
      • Variables
    • Explanation
      • Model Import Framework
  • Libnd4j
    • How To Guides
      • Building on Windows
      • Building for raspberry pi or Jetson Nano
      • Building on ios
      • How to Add Operations
      • How to Setup CLion
    • Reference
      • Understanding graph execution
      • Overview of working with libnd4j
      • Helpers Overview (CUDNN, OneDNN,Armcompute)
    • Explanation
  • Python4j
    • Tutorials
      • Quickstart
    • How To Guides
      • Write Python Script
    • Reference
      • Python Types
      • Python Path
      • Garbage Collection
      • Python Script Execution
    • Explanation
  • Spark
    • Tutorials
      • DL4J on Spark Quickstart
    • How To Guides
      • How To
      • Data How To
    • Reference
      • Parameter Server
      • Technical Reference
    • Explanation
      • Spark API Reference
  • codegen
Powered by GitBook
On this page
  • Quick Overview
  • Typical Problems and how to fix
  • Computer Vision
  • NLP
  • Spark
  • Android

Was this helpful?

Export as PDF
  1. Multi-Project
  2. Explanation

Required Dependencies

PreviousExplanationNextThe core workflow

Last updated 2 years ago

Was this helpful?

The DL4J suite has different configuration requirements for your dependencies depending on your use case. This page gives an overview of common cases to ensure people have what they need to get started.

Quick Overview

The deeplearning4j suite has a few common components to consider. Most users just need the following dependencies:

  1. Deeplearning4j NN: This contains the DSL for running the simpler neural networks.

  2. A Backends is also required. This is for running different code on cpus/gpus.

  3. The deeplearning4j suite uses Javacpp for running platform specific native code. These dependencies will have classifiers for different platforms. Read up on classifiers at baeldung: See a list here:

    1. Each jar has a classifier for different platforms. These platform specific jars only contain native code for different platforms. The classifiers come in the form of OS-architecture-platform-helper-optimization

      1. OS: This is linux,windows, mac, android

      2. Architecture: ARM(Mobile phones, rasbperry pi) or X86 (AMD/Intel)

      3. Platform: CUDA. Not present for cpu architectures

      4. Helper: Cudnn, onednn or other platform specific libraries. These provide more optimized routines for certain operations.

      5. Optimization: Only for CPU architectures. AVX2/AVX512 - these dependencies will only run on cpus that support avx2/avx512.

  4. All versions must be the same dependencies. We do not support mixing versions. There is no reason to. Each version signifies what is compatible API wise.

Typical Problems and how to fix

  1. Very large jar files: Your jar will be over sized and contain a bunch of dependencies. These are cross platform dependencies. Each platform dependency contains native c++ binaries for a specific platform. This keeps binaries smaller for different platforms but adds complexity for deployment. Simplify this by either using -platform dependencies with your specific platform in mind or specify the dependencies manually. This means including a dependency without the platform (this contains the actual java code for the dependency) and the dependency with classifier.

  2. GLIBC issues: Depending on the platform, some users may need to run older glibc versions. If you get GLIBC issues, please use the linux-x86_64-compat dependency. If you need a custom build, please contact us for support. https://konduit.ai/

  3. Quickstart confusing with deeplearning4j-core: Note that many of our quickstarts using deeplearning4j-core. This adds extra dependencies for computer vision not every user needs. Originally deeplearning4j-core was the recommended path since most people were needing computer vision tools anyways. Users should be aware of the extra dependencies before adding.

    1. -platform dependencies: Platform dependencies contain default dependency recipes for users to avoid needing to setup multiple classifiers per platform. This comes with the side effect of users being surprised by larger jar files when they go to deploy. This should be fixed when a user wants to go to production. If you are using maven ensure you specify -Djavacpp.platform=your-platform - you can also set a property in your pom.xml as well. The alternative is to specify platform specific dependencies yourself. The following considerations should apply for different backends:

      1. Nd4j-minimizer: this should not need any extra dependencies. This is just nd4j-native without the openblas dependency for embedded use cases where binary size matters.

  4. Linker errors: If you run in to issues please run your program with -Dorg.bytedeco.javacpp.logger.debug=true and submit an issue: https://github.com/deeplearning4j/deeplearning4j/issues/new

If you are looking for more advanced neural networks, we recommend Samediff this just needs the nd4j-api and Backends

Computer Vision

NLP

If you are looking to run NLP workloads, you just need deeplearning4j-nlp and a Backends

Spark

Use dl4j-spark_2.12 and a Backends - note also depending on your spark job you may run in to jar file size limits. Ensure you minimize your dependencies as much as possible. In this case the dependencies should be restricted to the specific platform the jar will be running on.

Android

Users running android should be heavily aware of different ABIs as mentioned above. We recommend using the nd4j-minimizer backend to avoid dependencies on openblas if they are not needed. Note only cpu based android calculations are supported.

Cuda version clashes: Users may run in to clashing dependencies with cuda. Ensure you only have 1 cuda install. On linux you can also use javacpp's cuda redist artifacts for different versions of cuda. Note these dependencies are very large. See an overview here for the various redist artifacts: This covers linux-x86_64 and jetson ORIN based platforms.

Android oversized APKs: Ensure your ABI filters are setup for different dependencies. Suggested reading is also the gradle-javacpp plugin:

Nd4j-native: If you using nd4j-native specify openblas as a dependency as well:

Nd4j-cuda: This needs the cuda dependency.

EAR files: One user reported an issue with EAR files: if you run in to this please let us know. EAR files and WAR files as well as general jakarta application servers due to having specialized classloaders can have accidental dependency clashes.

Running on Apple mac M1 systems See the bottom of this page:

Computer vision workloads typically need deeplearning4j-nn and Backends

https://search.maven.org/artifact/org.deeplearning4j/deeplearning4j-nn/1.0.0-M2.1/jar
https://www.baeldung.com/maven-artifact-classifiers
https://repo1.maven.org/maven2/org/nd4j/nd4j-native/1.0.0-M2.1/
https://repo1.maven.org/maven2/org/bytedeco/cuda/11.8-8.6-1.5.8/
https://github.com/bytedeco/gradle-javacpp
https://search.maven.org/artifact/org.bytedeco/openblas/0.3.21-1.5.8/jar
https://search.maven.org/artifact/org.bytedeco/cuda/11.8-8.6-1.5.8/jar
https://github.com/deeplearning4j/deeplearning4j/issues/9906
https://deeplearning4j.konduit.ai/multi-project/explanation/maven
https://search.maven.org/artifact/org.datavec/datavec-data-image/1.0.0-M2.1/jar
https://search.maven.org/artifact/org.deeplearning4j/deeplearning4j-nn/1.0.0-M2.1/jar