> For the complete documentation index, see [llms.txt](https://deeplearning4j.konduit.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://deeplearning4j.konduit.ai/en-1.0.0-rewrite/deeplearning4j/overview-4.md). # LLM & VLM Stack Source-verified overview of the 1.0.0-rewrite SameDiff generation, tokenizer, evaluation, VLM, audio, TorchScript, and web modules. Deeplearning4j 1.0.0-rewrite adds an application layer for modern language, vision-language, and audio models on top of SameDiff. SameDiff owns graph execution and DSP compilation; these modules add tokenization, generation, continuation, constrained decoding, evaluation, model editing, and multimodal orchestration. This page describes the public module boundaries and their principal entry points. For runnable `GenerationPipeline` configuration, fixed-buffer reuse, constrained decoding, prefix caching, and cache metrics, use [LLM Inference and Serving](/en-1.0.0-rewrite/deeplearning4j/overview-4/inference-and-serving.md). ## Module Map | Maven artifact | Principal packages and entry points | Purpose | | ------------------- | ----------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `samediff-llm` | `org.eclipse.deeplearning4j.llm.generation.GenerationPipeline`, `GenerationPipelineConfig`, `TextGenerator` | Text generation, sampling, continuation, cache integration, evaluation, benchmarking, editing, and fine-tuning utilities | | `samediff-vlm` | `org.eclipse.deeplearning4j.vlm.model.VisionLanguageModel`, `VLMPipelineExecutor` | Image/video preprocessing and vision-language model execution | | `samediff-audio` | `org.eclipse.deeplearning4j.audio.whisper.WhisperModel` | Whisper loading, feature extraction, transcription, and audio utilities | | `tokenizers-native` | `org.eclipse.deeplearning4j.tokenizers.NativeTokenizer` | JavaCPP facade over the Rust tokenizers runtime; the Java facade is transitive, while applications using `HuggingFaceTokenizer` must add the matching native classifier | | `nd4j-torchscript` | `org.nd4j.torchscript.TorchScriptModelImport` | TorchScript and SafeTensors reading and conversion to SameDiff | | `nd4j-webjar` | generated `@eclipse/nd4j-webjar` TypeScript bindings | FlatBuffers graph/request/result bindings for browser or JavaScript tooling | `nd4j-tokenizers` and `nd4j-web` are Maven aggregator projects. Applications depend on `tokenizers-native` and `nd4j-webjar`, respectively, rather than on those aggregators. ## Maven Setup Add the application module you need and one ND4J backend. `samediff-llm` already depends on the unclassified `tokenizers-native` Java facade. `HuggingFaceTokenizer` also requires the native JAR whose classifier matches the deployment platform; set `javacpp.platform` accordingly (for example, `linux-x86_64`). ```xml 1.0.0-rewrite linux-x86_64 org.eclipse.deeplearning4j samediff-llm ${dl4j.version} org.eclipse.deeplearning4j tokenizers-native ${dl4j.version} ${javacpp.platform} org.eclipse.deeplearning4j nd4j-native-platform ${dl4j.version} ``` Replace `nd4j-native-platform` with `nd4j-cuda-12.9-platform` for the CUDA backend. See [Maven Setup](/en-1.0.0-rewrite/configuration/maven.md) and [GPU and CPU Setup](/en-1.0.0-rewrite/configuration/gpu-cpu.md) for backend and classifier details. Optional application artifacts use the same release version: ```xml org.eclipse.deeplearning4j samediff-vlm ${dl4j.version} org.eclipse.deeplearning4j samediff-audio ${dl4j.version} org.eclipse.deeplearning4j nd4j-torchscript ${dl4j.version} ``` ## Text Generation `GenerationPipeline.create(GenerationPipelineConfig)` is the supported high-level entry point. A tokenizer is required, and the decoder can be supplied as a loaded `SameDiff` graph or as a supported model path. ```java import org.eclipse.deeplearning4j.llm.generation.GenerationPipeline; import org.eclipse.deeplearning4j.llm.generation.GenerationPipelineConfig; import org.eclipse.deeplearning4j.llm.generation.GenerationResult; import org.eclipse.deeplearning4j.llm.generation.kvcache.KvCacheStrategy; import org.eclipse.deeplearning4j.llm.generation.sampling.SamplingConfig; import org.eclipse.deeplearning4j.llm.tokenizer.HuggingFaceTokenizer; import org.eclipse.deeplearning4j.llm.tokenizer.Tokenizer; Tokenizer tokenizer = HuggingFaceTokenizer.fromFile("/models/tokenizer.json"); try { GenerationPipelineConfig config = GenerationPipelineConfig.builder() .decoderPath("/models/decoder.sdz") .tokenizer(tokenizer) .samplingConfig(SamplingConfig.greedy()) .maxNewTokens(256) .kvCacheStrategy(KvCacheStrategy.STATIC) .dspEnabled(true) .build(); try (GenerationPipeline pipeline = GenerationPipeline.create(config)) { GenerationResult result = pipeline.generate("Explain SameDiff in one paragraph."); System.out.println(result.getText()); } } finally { tokenizer.close(); } ``` The pipeline closes models that it loaded from paths. A decoder or tokenizer supplied by the caller remains caller-owned. ### Supported Pipeline Capabilities | Capability | Public surface | Notes | | -------------------------- | -------------------------------------------------- | ---------------------------------------------------------------------------------- | | Sampling | `SamplingConfig` | Greedy and configurable temperature/top-k/top-p sampling | | Request-specific sampling | `GenerationPipeline.generate(..., SamplingConfig)` | Changes token selection without reloading the model | | Continuation | `GenerationPipeline.GenerationSession` | Retains KV and recurrent state; one thread-confined session per pipeline | | Fixed-buffer reuse | `maxPrefillLength`, `maxKvCacheLength` | Keeps DSP-visible addresses stable across compatible calls | | Constrained output | `ConstraintConfig` | JSON and tool-call constraints use a Java token-selection loop | | Cross-request prefix reuse | `prefixCacheEnabled`, `KvPrefixBlockPool` | Opt-in, block-aligned reuse for supported non-recurrent, in-graph static-KV models | | Streaming | `TextGenerator.generateStreaming(...)` | Separate callback-oriented generation API | | Continuous batching | `ContinuousBatchScheduler` | Lower-level scheduler; not enabled merely by constructing `GenerationPipeline` | | Speculative components | classes under `generation.speculative` | Explicit integration is required; class presence is not a pipeline default | See [LLM Inference and Serving](/en-1.0.0-rewrite/deeplearning4j/overview-4/inference-and-serving.md) for configuration restrictions, ownership, and prefix-cache metric semantics. See the [DSP Framework Overview](/en-1.0.0-rewrite/nd4j/overview-2/dsp-framework.md) for execution modes, capture/replay, and diagnostics. ## Tokenizers Most generation applications use the `samediff-llm` tokenizer abstraction: * `org.eclipse.deeplearning4j.llm.tokenizer.Tokenizer` * `HuggingFaceTokenizer.fromFile(...)` or `fromDirectory(...)` * `TokenizerFactory` for format-based loading * `ChatTemplate` for model-specific conversation formatting The lower-level `tokenizers-native` artifact exposes `org.eclipse.deeplearning4j.tokenizers.NativeTokenizer.fromFile(...)` and `fromJson(...)`. Use it directly only when an application needs the native tokenizer facade without the generation layer. Tokenizer assets belong to a specific model family and revision. Do not silently substitute a tokenizer with a similar vocabulary size; token IDs, special tokens, and chat templates are part of the model contract. ## Evaluation and Benchmarking LLM quality evaluation is under `org.eclipse.deeplearning4j.llm.eval`: * `EvalRunner`, `EvalConfig`, and `EvalResult` * `PerplexityEvaluator` and `GenerationQualityValidator` * benchmark tasks under `org.eclipse.deeplearning4j.llm.eval.benchmark`: `MMLUBenchmark`, `ArcBenchmark`, `Gsm8kBenchmark`, `HellaSwagBenchmark`, `TruthfulQABenchmark`, and `WinograndeBenchmark` * datasets under `org.eclipse.deeplearning4j.llm.eval.dataset` * metrics under `org.eclipse.deeplearning4j.llm.eval.metrics` Decode performance and numerical validation are separate from task-quality evaluation. They live under `org.eclipse.deeplearning4j.model.benchmark` and include `BenchmarkRunner`, `BenchmarkConfig`, `BenchmarkConfigApplier`, `DecodeValidationFramework`, `DspAccuracyValidator`, and `MultiLevelComparator`. Always identify which measurement is being reported: task accuracy, perplexity, whole-request throughput, decode throughput, or steady-state replay throughput are not interchangeable. ## Model Editing Model-editing utilities live under `org.eclipse.deeplearning4j.llm.editing`: * `AbliterationWorkflow`, `AbliterationConfig`, and `AbliterationResult` * `RefusalDirectionFinder` and `RefusalDirection` * `WeightOrthogonalizer` * `DefaultPromptSets` These utilities operate on model weights and should be run against a copy or a versioned checkpoint. Validate edited models with held-out prompts and task-level evaluation before replacing a production artifact. ## Fine-Tuning The rewrite includes dataset preparation and student-workflow utilities under `org.eclipse.deeplearning4j.llm.finetune`, alongside SameDiff PEFT and alignment infrastructure. Start with: * [LLM Fine-Tuning Overview](/en-1.0.0-rewrite/deeplearning4j/overview-4/fine-tuning.md) * [Instruction SFT, LoRA, and QLoRA](/en-1.0.0-rewrite/deeplearning4j/overview-4/instruction-sft-lora.md) * [Knowledge Distillation](/en-1.0.0-rewrite/deeplearning4j/overview-4/knowledge-distillation.md) * [Preference and RL Alignment](/en-1.0.0-rewrite/deeplearning4j/overview-4/preference-rl-alignment.md) * [Quantized and Memory-Efficient Training](/en-1.0.0-rewrite/deeplearning4j/overview-4/quantized-training.md) ## Vision-Language Models The `samediff-vlm` artifact builds on `samediff-llm` and the SameDiff ONNX importer. Principal entry points include: * `VisionLanguageModel.fromDirectory(File)` for a prepared multi-part model directory * `VideoVisionLanguageModel` for video-aware orchestration * `VLMImagePreprocessor` for image tensor preparation * `VLMPipelineExecutor` for composed pipeline execution * `OCREngineFactory` for registered OCR engines * `VlmEvalRunner` for VLM evaluation JavaCV and FFmpeg support is optional at runtime and is needed only for video frame extraction paths that use it. ## Audio and Whisper The `samediff-audio` artifact provides: * `WhisperModel.fromOnnx(...)` and public GGML loading overloads * `WhisperTokenizer` * `AudioLoader` * `WhisperMelSpectrogram` * waveform-generation utilities such as `AudioFileGenerator` and `SameDiffWaveformGenerator` The artifact depends on `samediff-llm`, the SameDiff ONNX importer, and `nd4j-ggml` so that its supported model-loading paths converge on SameDiff execution. ## TorchScript and SafeTensors `nd4j-torchscript` reads TorchScript archives and SafeTensors metadata and converts supported models to SameDiff: ```java import org.nd4j.autodiff.samediff.SameDiff; import org.nd4j.torchscript.TorchScriptModelImport; try (SameDiff model = TorchScriptModelImport.importModel("/models/model.pt")) { // Inspect or execute the imported SameDiff graph. } ``` Related lower-level classes include `TorchScriptReader`, `SafetensorsReader`, `TorchScriptToSameDiffConverter`, and `org.nd4j.torchscript.convert.ConversionOptions`. Conversion support depends on the operators present in the archive; validate the imported graph before deployment. ## Web Bindings `nd4j-webjar` packages generated TypeScript FlatBuffers bindings and publishes the npm identity `@eclipse/nd4j-webjar`. It exposes graph and transport records such as `FlatArray`, `FlatGraph`, `FlatInferenceRequest`, `FlatResult`, and `FlatResponse`. This artifact is a serialization binding, not a standalone Java inference server. Applications provide their own transport and execution service around the generated records. ## Operational Boundaries * Add exactly one ND4J runtime backend to an application unless a documented multi-backend deployment explicitly requires otherwise. * Keep long-lived `GenerationPipeline` instances alive across related requests; recreating a pipeline per prompt discards opportunities for DSP-plan or prefix-cache reuse. * Close pipelines, tokenizers, imported `SameDiff` graphs, and multimodal models according to their ownership contracts. * Treat lower-level cache, batching, and offload classes as explicit integration components rather than automatically active features. * Use DSP diagnostics for plan/capture/transfer failures and LLM evaluation APIs for generation-quality measurements; one does not replace the other. ## Next Steps * [LLM Inference and Serving](/en-1.0.0-rewrite/deeplearning4j/overview-4/inference-and-serving.md) * [DSP Framework Overview](/en-1.0.0-rewrite/nd4j/overview-2/dsp-framework.md) * [GGML/GGUF Import](/en-1.0.0-rewrite/model-import/overview-3.md) * [Hardware Backends](/en-1.0.0-rewrite/nd4j/overview-1/hardware-backends.md) * [1.0.0-rewrite Release Notes](/en-1.0.0-rewrite/release-notes/1.0.0-rewrite.md)