For the complete documentation index, see llms.txt. This page is also available as Markdown.

Spark Data Pipelines

Loading and preprocessing data on Apache Spark for distributed DL4J training

This page covers how to build data pipelines for DL4J distributed training on Apache Spark. It assumes familiarity with basic Spark concepts (RDDs, partitions, executors) and DL4J's DataSet/MultiDataSet classes.

The final output of any data pipeline must be one of:

  • JavaRDD<DataSet> — for single input/output networks

  • JavaRDD<MultiDataSet> — for multi-input/output networks

  • A directory of serialized DataSet/MultiDataSet files on network storage (HDFS, S3, Azure Blob)

  • A directory of minibatches in a custom format

Best practice: Preprocess your data once and save it to HDFS. Then train by pointing SparkDl4jMultiLayer.fit(String path) at that directory. This avoids recomputing the pipeline on every training run and reduces memory pressure during training.

Contents:


CSV Data for Classification or Regression

To load a CSV file from HDFS and produce a JavaRDD<DataSet>:

Classification (6 columns total: columns 0–4 are features, column 5 is an integer class index, 10 classes):

Regression (6 columns total: columns 0–2 are features, columns 3–5 are labels):

DataVecDataSetFunction is the Spark equivalent of RecordReaderDataSetIterator for single-machine pipelines.


Image Classification Pipelines

Image pipelines on Spark use a two-step process: first batch images into FileBatch objects (which preserve efficient compression like JPEG), then load those batches during training. This avoids per-image remote reads during training.

Step 1a: Preprocess locally

Then copy destDir to HDFS: hadoop fs -put /home/user/preprocessed hdfs:///data/preprocessed

Step 1b: Preprocess using Spark (if images are already on HDFS)

Step 2: Training from preprocessed batches

For labels from filenames rather than parent directories (e.g., cat_img1234.jpg):


MultiDataSet from Multiple RDDs

Use RecordReaderMultiDataSetIterator (RRMDSI) with SparkSourceDummyReader to bridge between Spark RDDs and the multi-dataset pipeline API.

Case 1: Single RDD<List<Writable>> to RDD<MultiDataSet>

Single-machine equivalent:

Spark equivalent:

Case 2: Multiple RDDs joined into RDD<MultiDataSet>

For sequence data (RDD<List<List<Writable>>>), use SparkSourceDummySeqReader in place of SparkSourceDummyReader.


Save and Load RDD DataSet to Network Storage

Saving preprocessed data to HDFS and loading it for training avoids recomputing the pipeline and reduces memory use during training. This is the recommended workflow for multi-run experiments.

Save a JavaRDD<DataSet> to HDFS:

For JavaRDD<MultiDataSet>, use BatchAndExportMultiDataSetsFunction instead. It takes the same arguments.

Load and train directly from the saved directory:

Or for MultiDataSet:

Manual path loading:


Prepare Data on a Single Machine, Use on a Cluster

If you have an existing single-machine data pipeline, you can export DataSet objects locally and copy them to HDFS.

Step 1: Save DataSets locally

To save directly to HDFS (if the machine has HDFS client access):

Step 2: Copy to HDFS and train

Note: you can also use FileDataSetIterator to read locally saved DataSets on a single machine without Spark.


Hadoop MapFile/SequenceFile Format

An alternative to serialized DataSet files is Hadoop's MapFile/SequenceFile binary format. This can convert any RecordReader or SequenceRecordReader output into a Spark-compatible format.

Dependencies:

Step 1: Create a MapFile locally

MapFileRecordWriter supports splitting into multiple smaller files, which is recommended for Spark parallelism.

Step 2: Copy to HDFS

Step 3: Load into RDD<DataSet> for training


RNN Data from Multiple CSV Files

For RNN/sequence datasets where each CSV file is one sequence:

  • Each row of the CSV is one time step.

  • Files may have different numbers of rows (variable-length sequences are supported).

  • All files must have the same number of columns.


Custom Minibatch Format

For data stored in a custom binary format (one minibatch per file), implement the DataSetLoader or MultiDataSetLoader interface:

Then use it at training time:

This approach is typically not needed unless you have pre-existing data in a format DL4J does not natively support (e.g., a proprietary binary protocol or a Parquet-based format).

Last updated

Was this helpful?