> For the complete documentation index, see [llms.txt](https://deeplearning4j.konduit.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://deeplearning4j.konduit.ai/en-1.0.0-rewrite/deeplearning4j/overview-4/instruction-sft-lora.md). # Instruction SFT, LoRA, and QLoRA How to fine-tune LLMs with SameDiff using response-masked supervised fine-tuning, LoRA adapters, QLoRA, and the SFTTrainingPipeline API. Use supervised fine-tuning when you have examples of the behavior you want: instruction/response pairs, multi-turn chat transcripts, tool-use traces, or domain-specific answer formats. SameDiff supports both full model fine-tuning and parameter-efficient fine-tuning (PEFT) adapters. The main runnable examples are: * `LLMInstructionFineTuningExample.java` for real tokenizer, ChatML formatting, exact response masking, full SFT, LoRA adaptation, adapter save/merge, and GGUF export checks. * `SFTTrainingPipelineExample.java` for `SFTConfig`, curation utilities, bucketing, data mixing, and the high-level pipeline API. * `LoRAFineTuningExample.java` for direct `PeftModel` use. * `QLoRAAndAdvancedAdaptersExample.java` for QLoRA, DoRA, AdaLoRA, IA3, LoHa, LoKr, VeRA, DyLoRA, LoftQ, prompt tuning, prefix tuning, bottleneck adapters, and adapter caching. ## Full Fine-Tuning vs Adapters | Method | Trainable weights | Best for | Tradeoff | | ------------- | ------------------------------------------------------------------------- | ------------------------------------------------------------------- | ------------------------------------------------------------- | | Full SFT | All model weights | Small models, from-scratch compact LMs, final domain specialization | Highest memory and optimizer-state cost | | LoRA | Low-rank adapter matrices on selected projections | Most single-domain instruction tuning | Small checkpoints; base model remains frozen | | QLoRA | LoRA adapters over a quantized base | Consumer GPUs or very large base models | Quantization-aware constraints and slightly more moving parts | | Advanced PEFT | DoRA, AdaLoRA, IA3, LoHa, LoKr, VeRA, DyLoRA, LoftQ, prompt/prefix tuning | Specific quality, memory, or serving requirements | Choose method based on target deployment and data size | For transformer LMs, LoRA targets are usually attention projections first: ```java LoraConfig loraConfig = LoraConfig.builder() .r(8) .loraAlpha(16) .loraDropout(0.0) .targetModules(Arrays.asList("q_proj", "v_proj")) .build(); ``` Use wider target lists when the data is large enough to justify more capacity: ```java .targetModules(Arrays.asList("q_proj", "k_proj", "v_proj", "o_proj")) ``` ## Tokenization and Response Masks Instruction SFT is not just next-token training on the whole prompt. The loss must ignore system and user tokens and train only on assistant response tokens. `LLMInstructionFineTuningExample` uses the same tokenizer path as generation: ```java File gguf = LLMModelDownloader .download(LLMModel.QWEN35_0_8B, QuantType.Q4_K_M) .getModelFile(); Tokenizer tokenizer = HuggingFaceTokenizer.fromDirectory(gguf.getParentFile()); int vocabSize = tokenizer.getVocabSize(); ``` The example renders each conversation twice: once through the assistant header and once with the full response. The longest common token prefix is the prompt span. Tokens after that prefix get `loss_mask=1`, and earlier tokens get `loss_mask=0`. The resulting batches map into the graph as: | Array | SameDiff mapping | Meaning | | ----------- | ---------------- | ---------------------------------- | | `input_ids` | feature | Prompt and shifted response tokens | | `labels` | label | Next token IDs | | `loss_mask` | feature mask | 1.0 only where loss should train | ## Full SFT with `SameDiff.fit()` When you already have tokenized `MultiDataSet` batches, use standard SameDiff training: ```java int totalSteps = trainBatches.size() * trainEpochs; Adam adamW = Adam.builder() .learningRateSchedule(CosineWarmupSchedule.fromRatio( 3e-4, 3e-5, 0.1, totalSteps)) .beta1(0.9) .beta2(0.999) .epsilon(1e-8) .build(); TrainingConfig trainingConfig = new TrainingConfig.Builder() .updater(adamW) .dataSetFeatureMapping("input_ids") .dataSetLabelMapping("labels") .dataSetFeatureMaskMapping("loss_mask") .build(); model.setTrainingConfig(trainingConfig); History history = model.fit(new SimpleListMultiDataSetIterator(trainBatches), trainEpochs); ``` This is the most direct route and is useful for custom graph losses, custom schedules, and experiments where you want exact control over the iterator. ## LoRA with `PeftModel` `PeftModel` wraps a `SameDiff` base model, freezes base weights, injects adapter variables, and exposes training, output, adapter toggling, saving, and merging methods. ```java PeftModel peft = PeftModel.fromPretrained(model, loraConfig); peft.setTrainingConfig(new TrainingConfig.Builder() .updater(Adam.builder().learningRate(1e-3).build()) .dataSetFeatureMapping("input_ids") .dataSetLabelMapping("labels") .dataSetFeatureMaskMapping("loss_mask") .build()); peft.fit(new SimpleListMultiDataSetIterator(adaptBatches), adaptEpochs); ``` After training, verify that the base model stayed frozen and the adapter contributes non-zero output: ```java INDArray baseBefore = peft.getModel() .getVariable("model.layers.0.self_attn.q_proj.weight") .getArr() .dup(); // train the adapter INDArray baseAfter = peft.getModel() .getVariable("model.layers.0.self_attn.q_proj.weight") .getArr(); double baseDrift = baseBefore.sub(baseAfter).norm2Number().doubleValue(); ``` For deployment, either save the adapter or merge it into the base weights: ```java peft.saveAdapter(adapterDir); SameDiff merged = peft.mergeAndUnload(); ``` Save adapters when you will load multiple task variants over the same base model. Merge when you want a standalone graph with no adapter indirection. ## SFTTrainingPipeline `SFTTrainingPipeline` owns the common setup: optimizer schedule, `TrainingConfig`, optional PEFT wrapping, epoch loop, and merge/export for adapter runs. ```java SFTConfig sftConfig = SFTConfig.builder() .chatTemplate(ChatTemplate.CHATML) .systemMessage("You are a helpful assistant.") .maxSeqLength(512) .learningRate(2e-5) .minLearningRate(0.0) .warmupRatio(0.03) .numEpochs(1) .gradientAccumulationSteps(2) .computeDataType(DataType.FLOAT) .peftConfig(LoraConfig.builder() .r(4) .loraAlpha(8) .targetModules(Arrays.asList("q_proj", "v_proj")) .build()) .build(); SFTTrainingPipeline pipeline = new SFTTrainingPipeline(baseModel, sftConfig); pipeline.train(new SimpleListMultiDataSetIterator(trainBatches), trainBatches.size()); SameDiff merged = pipeline.mergeAndExport(); ``` The pipeline also accepts raw instruction/response pairs and conversation lists when you want it to handle chat formatting: ```java pipeline.trainFromPairs(pairs); pipeline.train(conversations); ``` For production tokenization, prefer exact token masks from your tokenizer, especially when templates include special tokens or model-specific assistant headers. ## QLoRA and Advanced Adapters QLoRA combines a quantized base with trainable LoRA adapters: ```java QLoraConfig qloraConfig = QLoraConfig.builder() .r(16) .loraAlpha(32) .loraDropout(0.05) .bits(4) .quantType("nf4") .doubleQuant(true) .computeDataType(DataType.BFLOAT16) .loraDataType(DataType.BFLOAT16) .blockSize(64) .targetModules(targetModules) .taskType(TaskType.CAUSAL_LM) .build(); PeftModel qloraModel = PeftModel.fromPretrained(baseModel, qloraConfig); ``` Use the advanced adapter types when LoRA is not the best fit: | Adapter | Use when | | ------------------- | -------------------------------------------------------------------------- | | DoRA | You want better quality at low rank by learning magnitude and direction | | AdaLoRA | You want rank budget allocated adaptively across layers | | IA3 | You need the fewest trainable parameters | | LoHa / LoKr | You want higher effective rank with structured low-rank updates | | VeRA | You want shared frozen matrices and very small per-layer trainable vectors | | DyLoRA | You want one adapter that can deploy at multiple ranks | | LoftQ | You want LoRA initialization that compensates for quantization residuals | | Prompt tuning | You want soft prompt tokens without changing model weights | | Prefix tuning | You want per-layer attention key/value prefixes | | Bottleneck adapters | You want classic down/up projection adapters | ## Multi-Adapter Serving `LoraAdapterCache` supports serving many task adapters over one base model with hot and warm tiers: ```java try (LoraAdapterCache cache = new LoraAdapterCache(3, 10)) { cache.registerAdapter("summarize", weights, false); cache.registerAdapter("translate-de", weights, false); cache.applyAdapter("summarize", peftModel.getModel()); } ``` Use this pattern when the base model is large, adapters are small, and requests switch between tasks or tenants. ## Evaluation Checklist * Log the number of prompt tokens and response tokens for a sample batch. * Confirm `loss_mask` is zero for system/user turns and one for assistant tokens. * Track held-out masked NLL and perplexity before and after training. * For PEFT, verify base-weight drift is zero. * Toggle the adapter off and on and confirm output changes after training. * Save adapter weights separately before merging. * If exporting, validate that the graph is compatible with `GGMLModelExport` before writing GGUF. ## Run the Examples ```bash cd ../deeplearning4j-examples/samediff-examples mvn exec:java -Dexec.mainClass="org.nd4j.examples.samediff.quickstart.training.LLMInstructionFineTuningExample" mvn exec:java -Dexec.mainClass="org.nd4j.examples.samediff.quickstart.training.SFTTrainingPipelineExample" mvn exec:java -Dexec.mainClass="org.nd4j.examples.samediff.quickstart.training.QLoRAAndAdvancedAdaptersExample" ``` ## Related Pages * [LLM Fine-Tuning Overview](/en-1.0.0-rewrite/deeplearning4j/overview-4/fine-tuning.md) * [Knowledge Distillation](/en-1.0.0-rewrite/deeplearning4j/overview-4/knowledge-distillation.md) * [Preference and RL Alignment](/en-1.0.0-rewrite/deeplearning4j/overview-4/preference-rl-alignment.md) * [Quantized and Memory-Efficient Training](/en-1.0.0-rewrite/deeplearning4j/overview-4/quantized-training.md) * [PEFT and RL Alignment Training](/en-1.0.0-rewrite/deeplearning4j/peft-and-rl.md)