> For the complete documentation index, see [llms.txt](https://deeplearning4j.konduit.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://deeplearning4j.konduit.ai/en-1.0.0-rewrite/deeplearning4j/overview-4/fine-tuning.md).

# LLM Fine-Tuning Overview

Practical guide to the SameDiff LLM fine-tuning paths: supervised instruction tuning, LoRA and QLoRA adapters, distillation, preference alignment, and quantized training.

Deeplearning4j 1.0.0-rewrite adds several ways to adapt large language models in Java. The new SameDiff training APIs cover full supervised fine-tuning, parameter-efficient adapters, teacher-student distillation, preference/RL alignment, and memory-efficient quantized training.

The examples repository is the source of truth for runnable code. Start in `../deeplearning4j-examples/samediff-examples/src/main/java/org/nd4j/examples/samediff/quickstart/training/` and use the pages below as the decision guide and production checklist.

## Choose a Method

| Goal                                                | Recommended path                                 | Main APIs                                                                     | Example                                                              |
| --------------------------------------------------- | ------------------------------------------------ | ----------------------------------------------------------------------------- | -------------------------------------------------------------------- |
| Teach a model a new instruction format or domain    | Supervised fine-tuning (SFT)                     | `SFTConfig`, `SFTTrainingPipeline`, `TrainingConfig`                          | `LLMInstructionFineTuningExample.java`                               |
| Adapt a pretrained model on limited GPU memory      | LoRA or QLoRA                                    | `LoraConfig`, `QLoraConfig`, `PeftModel`                                      | `LoRAFineTuningExample.java`, `QLoRAAndAdvancedAdaptersExample.java` |
| Serve many task-specific variants of one base model | Adapter hot-swap                                 | `PeftModel`, `LoraAdapterCache`                                               | `QLoRAAndAdvancedAdaptersExample.java`                               |
| Compress a large teacher into a smaller student     | Distillation                                     | `DistillationConfig`, `DistillationTrainer`, `SameDiff.fit`                   | `QwenToStudentDistillationExample.java`                              |
| Improve preference behavior after SFT               | DPO, GRPO, PPO, ORPO, KTO, reward model training | `DPOTrainer`, `GRPOTrainer`, `RLAlignmentPipeline`                            | `RLAlignmentTrainingExample.java`                                    |
| Reduce training memory and optimizer state          | Quantized training                               | `Adam8bit`, `FP8TrainingConfig`, `GradientCheckpointConfig`, `GraphOptimizer` | `GraphOptimizerQuantizedTrainingExample.java`                        |

## The New Training Stack

The training stack is built around SameDiff graphs. A trainable LLM graph normally exposes these placeholders:

| Placeholder      | Shape                 | Purpose                                                           |
| ---------------- | --------------------- | ----------------------------------------------------------------- |
| `input_ids`      | `[batch, seq]`        | Token IDs for teacher forcing                                     |
| `labels`         | `[batch, seq]`        | Next-token IDs                                                    |
| `loss_mask`      | `[batch, seq]`        | 1.0 for assistant or target tokens, 0.0 for ignored prompt tokens |
| `teacher_logits` | `[batch, seq, vocab]` | Optional soft targets for distillation                            |

The graph marks a scalar loss variable as the training loss. SameDiff then drives the update through `TrainingConfig` and `fit()`:

```java
TrainingConfig config = new TrainingConfig.Builder()
    .updater(adamW)
    .dataSetFeatureMapping("input_ids")
    .dataSetLabelMapping("labels")
    .dataSetFeatureMaskMapping("loss_mask")
    .build();

model.setTrainingConfig(config);
model.fit(trainIterator, epochs);
```

The higher-level pipelines build that same machinery for common LLM workflows. Use the low-level `TrainingConfig` route when you already own tokenized `MultiDataSet` batches and want full control. Use `SFTTrainingPipeline` or `RLAlignmentPipeline` when you want the framework to own chat formatting, schedule construction, PEFT wrapping, and the outer train loop.

## Data Flow

A production fine-tuning job usually follows this path:

1. Load a tokenizer from the same model directory used for generation.
2. Render conversations with a chat template such as ChatML or Llama 3.
3. Build exact token-level loss masks so the model trains on assistant tokens only.
4. Pack or bucket examples by length to keep batches efficient.
5. Choose full fine-tuning, adapters, distillation, preference optimization, or a combined recipe.
6. Train with `SameDiff.fit()` or a training pipeline.
7. Evaluate held-out masked NLL, perplexity, or preference metrics.
8. Save a SameDiff checkpoint, save adapter weights, or merge adapters into a deployment graph.

The important distinction is the loss mask. Continued pretraining trains on every next token. Instruction SFT trains only on response tokens. Preference alignment trains on chosen/rejected responses or reward-scored samples. Distillation trains against teacher logits, optionally mixed with hard labels.

## Practical Starting Points

| Starting point                                               | Use this page                                                                                                |
| ------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------ |
| You have instruction/response pairs                          | [Instruction SFT, LoRA, and QLoRA](/en-1.0.0-rewrite/deeplearning4j/overview-4/instruction-sft-lora.md)      |
| You have a large teacher and want a smaller student          | [Knowledge Distillation](/en-1.0.0-rewrite/deeplearning4j/overview-4/knowledge-distillation.md)              |
| You have chosen/rejected preference data or reward functions | [Preference and RL Alignment](/en-1.0.0-rewrite/deeplearning4j/overview-4/preference-rl-alignment.md)        |
| You are blocked by memory or optimizer state                 | [Quantized and Memory-Efficient Training](/en-1.0.0-rewrite/deeplearning4j/overview-4/quantized-training.md) |
| You need the complete class reference                        | [PEFT and RL Alignment Training](/en-1.0.0-rewrite/deeplearning4j/peft-and-rl.md)                            |

## Maven Modules

The examples use the SameDiff examples module, which pulls in the LLM, tokenizer, GGUF, and training APIs together. For an application, include only the pieces you need:

```xml
<dependency>
    <groupId>org.eclipse.deeplearning4j</groupId>
    <artifactId>samediff-llm</artifactId>
    <version>${dl4j.version}</version>
</dependency>
<dependency>
    <groupId>org.eclipse.deeplearning4j</groupId>
    <artifactId>nd4j-ggml</artifactId>
    <version>${dl4j.version}</version>
</dependency>
<dependency>
    <groupId>org.eclipse.deeplearning4j</groupId>
    <artifactId>tokenizers-native</artifactId>
    <version>${dl4j.version}</version>
</dependency>
<dependency>
    <groupId>org.eclipse.deeplearning4j</groupId>
    <artifactId>deeplearning4j-core</artifactId>
    <version>${dl4j.version}</version>
</dependency>
```

For CUDA, set the backend artifact in your project to the CUDA backend matching your installed CUDA version. The examples default to `nd4j-native` so they run on CPU.

## Running the Examples

From the examples repository:

```bash
cd ../deeplearning4j-examples/samediff-examples
mvn exec:java -Dexec.mainClass="org.nd4j.examples.samediff.quickstart.training.LLMInstructionFineTuningExample"
```

The more resource-intensive examples expose system properties for shorter dry runs. For example:

```bash
mvn exec:java \
  -Dexec.mainClass="org.nd4j.examples.samediff.quickstart.training.QwenToStudentDistillationExample" \
  -Dexample.distill.batches=2 \
  -Dexample.distill.epochs=1
```

## What to Validate

Before treating a fine-tuned model as usable, check all of the following:

| Check                          | Why it matters                                                                                    |
| ------------------------------ | ------------------------------------------------------------------------------------------------- |
| Mask audit                     | Confirms prompt/system tokens are ignored and assistant tokens are trained                        |
| Loss curve                     | Catches dead masks, frozen trainable variables, and exploding updates                             |
| Held-out masked NLL/perplexity | Detects memorization and domain overfitting                                                       |
| Base-weight drift for PEFT     | Confirms only adapter weights changed                                                             |
| Adapter on/off delta           | Confirms the trained adapter contributes non-zero behavior                                        |
| Teacher/student agreement      | Confirms distillation is learning the teacher distribution                                        |
| Preference loss by method      | Confirms DPO/GRPO/PPO is optimizing the intended objective                                        |
| Memory profile                 | Confirms quantization, gradient accumulation, and checkpointing are actually reducing peak memory |

## Related Pages

* [LLM and VLM Application Stack](/en-1.0.0-rewrite/deeplearning4j/overview-4.md)
* [Instruction SFT, LoRA, and QLoRA](/en-1.0.0-rewrite/deeplearning4j/overview-4/instruction-sft-lora.md)
* [Knowledge Distillation](/en-1.0.0-rewrite/deeplearning4j/overview-4/knowledge-distillation.md)
* [Preference and RL Alignment](/en-1.0.0-rewrite/deeplearning4j/overview-4/preference-rl-alignment.md)
* [Quantized and Memory-Efficient Training](/en-1.0.0-rewrite/deeplearning4j/overview-4/quantized-training.md)
* [PEFT and RL Alignment Training](/en-1.0.0-rewrite/deeplearning4j/peft-and-rl.md)