Ctrlk
Community ForumND4J JavadocDL4J Javadoc
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Instruction SFT, LoRA, and QLoRA

How to fine-tune LLMs with SameDiff using response-masked supervised fine-tuning, LoRA adapters, QLoRA, and the SFTTrainingPipeline API.

Use supervised fine-tuning when you have examples of the behavior you want: instruction/response pairs, multi-turn chat transcripts, tool-use traces, or domain-specific answer formats. SameDiff supports both full model fine-tuning and parameter-efficient fine-tuning (PEFT) adapters.

The main runnable examples are:

  • LLMInstructionFineTuningExample.java for real tokenizer, ChatML formatting, exact response masking, full SFT, LoRA adaptation, adapter save/merge, and GGUF export checks.

  • SFTTrainingPipelineExample.java for SFTConfig, curation utilities, bucketing, data mixing, and the high-level pipeline API.

  • LoRAFineTuningExample.java for direct PeftModel use.

  • QLoRAAndAdvancedAdaptersExample.java for QLoRA, DoRA, AdaLoRA, IA3, LoHa, LoKr, VeRA, DyLoRA, LoftQ, prompt tuning, prefix tuning, bottleneck adapters, and adapter caching.

Full Fine-Tuning vs Adapters

Method
Trainable weights
Best for
Tradeoff

Full SFT

All model weights

Small models, from-scratch compact LMs, final domain specialization

Highest memory and optimizer-state cost

LoRA

Low-rank adapter matrices on selected projections

Most single-domain instruction tuning

Small checkpoints; base model remains frozen

QLoRA

LoRA adapters over a quantized base

Consumer GPUs or very large base models

Quantization-aware constraints and slightly more moving parts

Advanced PEFT

DoRA, AdaLoRA, IA3, LoHa, LoKr, VeRA, DyLoRA, LoftQ, prompt/prefix tuning

Specific quality, memory, or serving requirements

Choose method based on target deployment and data size

For transformer LMs, LoRA targets are usually attention projections first:

LoraConfig loraConfig = LoraConfig.builder()
    .r(8)
    .loraAlpha(16)
    .loraDropout(0.0)
    .targetModules(Arrays.asList("q_proj", "v_proj"))
    .build();

Use wider target lists when the data is large enough to justify more capacity:

Tokenization and Response Masks

Instruction SFT is not just next-token training on the whole prompt. The loss must ignore system and user tokens and train only on assistant response tokens.

LLMInstructionFineTuningExample uses the same tokenizer path as generation:

The example renders each conversation twice: once through the assistant header and once with the full response. The longest common token prefix is the prompt span. Tokens after that prefix get loss_mask=1, and earlier tokens get loss_mask=0.

The resulting batches map into the graph as:

Array
SameDiff mapping
Meaning

input_ids

feature

Prompt and shifted response tokens

labels

label

Next token IDs

loss_mask

feature mask

1.0 only where loss should train

Full SFT with SameDiff.fit()

When you already have tokenized MultiDataSet batches, use standard SameDiff training:

This is the most direct route and is useful for custom graph losses, custom schedules, and experiments where you want exact control over the iterator.

LoRA with PeftModel

PeftModel wraps a SameDiff base model, freezes base weights, injects adapter variables, and exposes training, output, adapter toggling, saving, and merging methods.

After training, verify that the base model stayed frozen and the adapter contributes non-zero output:

For deployment, either save the adapter or merge it into the base weights:

Save adapters when you will load multiple task variants over the same base model. Merge when you want a standalone graph with no adapter indirection.

SFTTrainingPipeline

SFTTrainingPipeline owns the common setup: optimizer schedule, TrainingConfig, optional PEFT wrapping, epoch loop, and merge/export for adapter runs.

The pipeline also accepts raw instruction/response pairs and conversation lists when you want it to handle chat formatting:

For production tokenization, prefer exact token masks from your tokenizer, especially when templates include special tokens or model-specific assistant headers.

QLoRA and Advanced Adapters

QLoRA combines a quantized base with trainable LoRA adapters:

Use the advanced adapter types when LoRA is not the best fit:

Adapter
Use when

DoRA

You want better quality at low rank by learning magnitude and direction

AdaLoRA

You want rank budget allocated adaptively across layers

IA3

You need the fewest trainable parameters

LoHa / LoKr

You want higher effective rank with structured low-rank updates

VeRA

You want shared frozen matrices and very small per-layer trainable vectors

DyLoRA

You want one adapter that can deploy at multiple ranks

LoftQ

You want LoRA initialization that compensates for quantization residuals

Prompt tuning

You want soft prompt tokens without changing model weights

Prefix tuning

You want per-layer attention key/value prefixes

Bottleneck adapters

You want classic down/up projection adapters

Multi-Adapter Serving

LoraAdapterCache supports serving many task adapters over one base model with hot and warm tiers:

Use this pattern when the base model is large, adapters are small, and requests switch between tasks or tenants.

Evaluation Checklist

  • Log the number of prompt tokens and response tokens for a sample batch.

  • Confirm loss_mask is zero for system/user turns and one for assistant tokens.

  • Track held-out masked NLL and perplexity before and after training.

  • For PEFT, verify base-weight drift is zero.

  • Toggle the adapter off and on and confirm output changes after training.

  • Save adapter weights separately before merging.

  • If exporting, validate that the graph is compatible with GGMLModelExport before writing GGUF.

Run the Examples

Related Pages

PreviousLLM Fine-Tuning Overview
NextKnowledge Distillation

Last updated

Was this helpful?