Ctrlk
Community ForumND4J JavadocDL4J Javadoc
For the complete documentation index, see llms.txt. This page is also available as Markdown.

LLM Fine-Tuning Overview

Practical guide to the SameDiff LLM fine-tuning paths: supervised instruction tuning, LoRA and QLoRA adapters, distillation, preference alignment, and quantized training.

Deeplearning4j 1.0.0-rewrite adds several ways to adapt large language models in Java. The new SameDiff training APIs cover full supervised fine-tuning, parameter-efficient adapters, teacher-student distillation, preference/RL alignment, and memory-efficient quantized training.

The examples repository is the source of truth for runnable code. Start in ../deeplearning4j-examples/samediff-examples/src/main/java/org/nd4j/examples/samediff/quickstart/training/ and use the pages below as the decision guide and production checklist.

Choose a Method

Goal
Recommended path
Main APIs
Example

Teach a model a new instruction format or domain

Supervised fine-tuning (SFT)

SFTConfig, SFTTrainingPipeline, TrainingConfig

LLMInstructionFineTuningExample.java

Adapt a pretrained model on limited GPU memory

LoRA or QLoRA

LoraConfig, QLoraConfig, PeftModel

LoRAFineTuningExample.java, QLoRAAndAdvancedAdaptersExample.java

Serve many task-specific variants of one base model

Adapter hot-swap

PeftModel, LoraAdapterCache

QLoRAAndAdvancedAdaptersExample.java

Compress a large teacher into a smaller student

Distillation

DistillationConfig, DistillationTrainer, SameDiff.fit

QwenToStudentDistillationExample.java

Improve preference behavior after SFT

DPO, GRPO, PPO, ORPO, KTO, reward model training

DPOTrainer, GRPOTrainer, RLAlignmentPipeline

RLAlignmentTrainingExample.java

Reduce training memory and optimizer state

Quantized training

Adam8bit, FP8TrainingConfig, GradientCheckpointConfig, GraphOptimizer

GraphOptimizerQuantizedTrainingExample.java

The New Training Stack

The training stack is built around SameDiff graphs. A trainable LLM graph normally exposes these placeholders:

Placeholder
Shape
Purpose

input_ids

[batch, seq]

Token IDs for teacher forcing

labels

[batch, seq]

Next-token IDs

loss_mask

[batch, seq]

1.0 for assistant or target tokens, 0.0 for ignored prompt tokens

teacher_logits

[batch, seq, vocab]

Optional soft targets for distillation

The graph marks a scalar loss variable as the training loss. SameDiff then drives the update through TrainingConfig and fit():

TrainingConfig config = new TrainingConfig.Builder()
    .updater(adamW)
    .dataSetFeatureMapping("input_ids")
    .dataSetLabelMapping("labels")
    .dataSetFeatureMaskMapping("loss_mask")
    .build();

model.setTrainingConfig(config);
model.fit(trainIterator, epochs);

The higher-level pipelines build that same machinery for common LLM workflows. Use the low-level TrainingConfig route when you already own tokenized MultiDataSet batches and want full control. Use SFTTrainingPipeline or RLAlignmentPipeline when you want the framework to own chat formatting, schedule construction, PEFT wrapping, and the outer train loop.

Data Flow

A production fine-tuning job usually follows this path:

  1. Load a tokenizer from the same model directory used for generation.

  2. Render conversations with a chat template such as ChatML or Llama 3.

  3. Build exact token-level loss masks so the model trains on assistant tokens only.

  4. Pack or bucket examples by length to keep batches efficient.

  5. Choose full fine-tuning, adapters, distillation, preference optimization, or a combined recipe.

  6. Train with SameDiff.fit() or a training pipeline.

  7. Evaluate held-out masked NLL, perplexity, or preference metrics.

  8. Save a SameDiff checkpoint, save adapter weights, or merge adapters into a deployment graph.

The important distinction is the loss mask. Continued pretraining trains on every next token. Instruction SFT trains only on response tokens. Preference alignment trains on chosen/rejected responses or reward-scored samples. Distillation trains against teacher logits, optionally mixed with hard labels.

Practical Starting Points

Starting point
Use this page

You have instruction/response pairs

Instruction SFT, LoRA, and QLoRA

You have a large teacher and want a smaller student

Knowledge Distillation

You have chosen/rejected preference data or reward functions

Preference and RL Alignment

You are blocked by memory or optimizer state

Quantized and Memory-Efficient Training

You need the complete class reference

PEFT and RL Alignment Training

Maven Modules

The examples use the SameDiff examples module, which pulls in the LLM, tokenizer, GGUF, and training APIs together. For an application, include only the pieces you need:

For CUDA, set the backend artifact in your project to the CUDA backend matching your installed CUDA version. The examples default to nd4j-native so they run on CPU.

Running the Examples

From the examples repository:

The more resource-intensive examples expose system properties for shorter dry runs. For example:

What to Validate

Before treating a fine-tuned model as usable, check all of the following:

Check
Why it matters

Mask audit

Confirms prompt/system tokens are ignored and assistant tokens are trained

Loss curve

Catches dead masks, frozen trainable variables, and exploding updates

Held-out masked NLL/perplexity

Detects memorization and domain overfitting

Base-weight drift for PEFT

Confirms only adapter weights changed

Adapter on/off delta

Confirms the trained adapter contributes non-zero behavior

Teacher/student agreement

Confirms distillation is learning the teacher distribution

Preference loss by method

Confirms DPO/GRPO/PPO is optimizing the intended objective

Memory profile

Confirms quantization, gradient accumulation, and checkpointing are actually reducing peak memory

Related Pages

PreviousLLM Inference and Serving
NextInstruction SFT, LoRA, and QLoRA

Last updated

Was this helpful?