LLM & VLM Stack

Source-verified overview of the 1.0.0-rewrite SameDiff generation, tokenizer, evaluation, VLM, audio, TorchScript, and web modules.

Deeplearning4j 1.0.0-rewrite adds an application layer for modern language, vision-language, and audio models on top of SameDiff. SameDiff owns graph execution and DSP compilation; these modules add tokenization, generation, continuation, constrained decoding, evaluation, model editing, and multimodal orchestration.

This page describes the public module boundaries and their principal entry points. For runnable GenerationPipeline configuration, fixed-buffer reuse, constrained decoding, prefix caching, and cache metrics, use LLM Inference and Serving.

Module Map

Maven artifact	Principal packages and entry points	Purpose
samediff-llm	org.eclipse.deeplearning4j.llm.generation.GenerationPipeline, GenerationPipelineConfig, TextGenerator	Text generation, sampling, continuation, cache integration, evaluation, benchmarking, editing, and fine-tuning utilities
samediff-vlm	org.eclipse.deeplearning4j.vlm.model.VisionLanguageModel, VLMPipelineExecutor	Image/video preprocessing and vision-language model execution
samediff-audio	org.eclipse.deeplearning4j.audio.whisper.WhisperModel	Whisper loading, feature extraction, transcription, and audio utilities
tokenizers-native	org.eclipse.deeplearning4j.tokenizers.NativeTokenizer	JavaCPP facade over the Rust tokenizers runtime; the Java facade is transitive, while applications using HuggingFaceTokenizer must add the matching native classifier
nd4j-torchscript	org.nd4j.torchscript.TorchScriptModelImport	TorchScript and SafeTensors reading and conversion to SameDiff
nd4j-webjar	generated @eclipse/nd4j-webjar TypeScript bindings	FlatBuffers graph/request/result bindings for browser or JavaScript tooling

nd4j-tokenizers and nd4j-web are Maven aggregator projects. Applications depend on tokenizers-native and nd4j-webjar, respectively, rather than on those aggregators.

Maven Setup

Add the application module you need and one ND4J backend. samediff-llm already depends on the unclassified tokenizers-native Java facade. HuggingFaceTokenizer also requires the native JAR whose classifier matches the deployment platform; set javacpp.platform accordingly (for example, linux-x86_64).

<properties>
    <dl4j.version>1.0.0-rewrite</dl4j.version>
    <javacpp.platform>linux-x86_64</javacpp.platform>
</properties>

<dependencies>
    <dependency>
        <groupId>org.eclipse.deeplearning4j</groupId>
        <artifactId>samediff-llm</artifactId>
        <version>${dl4j.version}</version>
    </dependency>

    <dependency>
        <groupId>org.eclipse.deeplearning4j</groupId>
        <artifactId>tokenizers-native</artifactId>
        <version>${dl4j.version}</version>
        <classifier>${javacpp.platform}</classifier>
    </dependency>

    <dependency>
        <groupId>org.eclipse.deeplearning4j</groupId>
        <artifactId>nd4j-native-platform</artifactId>
        <version>${dl4j.version}</version>
    </dependency>
</dependencies>

Replace nd4j-native-platform with nd4j-cuda-12.9-platform for the CUDA backend. See Maven Setup and GPU and CPU Setup for backend and classifier details.

Optional application artifacts use the same release version:

<dependency>
    <groupId>org.eclipse.deeplearning4j</groupId>
    <artifactId>samediff-vlm</artifactId>
    <version>${dl4j.version}</version>
</dependency>

<dependency>
    <groupId>org.eclipse.deeplearning4j</groupId>
    <artifactId>samediff-audio</artifactId>
    <version>${dl4j.version}</version>
</dependency>

<dependency>
    <groupId>org.eclipse.deeplearning4j</groupId>
    <artifactId>nd4j-torchscript</artifactId>
    <version>${dl4j.version}</version>
</dependency>

Text Generation

GenerationPipeline.create(GenerationPipelineConfig) is the supported high-level entry point. A tokenizer is required, and the decoder can be supplied as a loaded SameDiff graph or as a supported model path.

import org.eclipse.deeplearning4j.llm.generation.GenerationPipeline;
import org.eclipse.deeplearning4j.llm.generation.GenerationPipelineConfig;
import org.eclipse.deeplearning4j.llm.generation.GenerationResult;
import org.eclipse.deeplearning4j.llm.generation.kvcache.KvCacheStrategy;
import org.eclipse.deeplearning4j.llm.generation.sampling.SamplingConfig;
import org.eclipse.deeplearning4j.llm.tokenizer.HuggingFaceTokenizer;
import org.eclipse.deeplearning4j.llm.tokenizer.Tokenizer;

Tokenizer tokenizer = HuggingFaceTokenizer.fromFile("/models/tokenizer.json");
try {
    GenerationPipelineConfig config = GenerationPipelineConfig.builder()
            .decoderPath("/models/decoder.sdz")
            .tokenizer(tokenizer)
            .samplingConfig(SamplingConfig.greedy())
            .maxNewTokens(256)
            .kvCacheStrategy(KvCacheStrategy.STATIC)
            .dspEnabled(true)
            .build();

    try (GenerationPipeline pipeline = GenerationPipeline.create(config)) {
        GenerationResult result = pipeline.generate("Explain SameDiff in one paragraph.");
        System.out.println(result.getText());
    }
} finally {
    tokenizer.close();
}

The pipeline closes models that it loaded from paths. A decoder or tokenizer supplied by the caller remains caller-owned.

Supported Pipeline Capabilities

Capability	Public surface	Notes
Sampling	SamplingConfig	Greedy and configurable temperature/top-k/top-p sampling
Request-specific sampling	GenerationPipeline.generate(..., SamplingConfig)	Changes token selection without reloading the model
Continuation	GenerationPipeline.GenerationSession	Retains KV and recurrent state; one thread-confined session per pipeline
Fixed-buffer reuse	maxPrefillLength, maxKvCacheLength	Keeps DSP-visible addresses stable across compatible calls
Constrained output	ConstraintConfig	JSON and tool-call constraints use a Java token-selection loop
Cross-request prefix reuse	prefixCacheEnabled, KvPrefixBlockPool	Opt-in, block-aligned reuse for supported non-recurrent, in-graph static-KV models
Streaming	TextGenerator.generateStreaming(...)	Separate callback-oriented generation API
Continuous batching	ContinuousBatchScheduler	Lower-level scheduler; not enabled merely by constructing GenerationPipeline
Speculative components	classes under generation.speculative	Explicit integration is required; class presence is not a pipeline default

See LLM Inference and Serving for configuration restrictions, ownership, and prefix-cache metric semantics. See the DSP Framework Overview for execution modes, capture/replay, and diagnostics.

Tokenizers

Most generation applications use the samediff-llm tokenizer abstraction:

• org.eclipse.deeplearning4j.llm.tokenizer.Tokenizer

• HuggingFaceTokenizer.fromFile(...) or fromDirectory(...)

• TokenizerFactory for format-based loading

• ChatTemplate for model-specific conversation formatting

The lower-level tokenizers-native artifact exposes org.eclipse.deeplearning4j.tokenizers.NativeTokenizer.fromFile(...) and fromJson(...). Use it directly only when an application needs the native tokenizer facade without the generation layer.

Tokenizer assets belong to a specific model family and revision. Do not silently substitute a tokenizer with a similar vocabulary size; token IDs, special tokens, and chat templates are part of the model contract.

Evaluation and Benchmarking

LLM quality evaluation is under org.eclipse.deeplearning4j.llm.eval:

• EvalRunner, EvalConfig, and EvalResult

• PerplexityEvaluator and GenerationQualityValidator

• benchmark tasks under org.eclipse.deeplearning4j.llm.eval.benchmark: MMLUBenchmark, ArcBenchmark, Gsm8kBenchmark, HellaSwagBenchmark, TruthfulQABenchmark, and WinograndeBenchmark

• datasets under org.eclipse.deeplearning4j.llm.eval.dataset

• metrics under org.eclipse.deeplearning4j.llm.eval.metrics

Decode performance and numerical validation are separate from task-quality evaluation. They live under org.eclipse.deeplearning4j.model.benchmark and include BenchmarkRunner, BenchmarkConfig, BenchmarkConfigApplier, DecodeValidationFramework, DspAccuracyValidator, and MultiLevelComparator.

Always identify which measurement is being reported: task accuracy, perplexity, whole-request throughput, decode throughput, or steady-state replay throughput are not interchangeable.

Model Editing

Model-editing utilities live under org.eclipse.deeplearning4j.llm.editing:

• AbliterationWorkflow, AbliterationConfig, and AbliterationResult

• RefusalDirectionFinder and RefusalDirection

• WeightOrthogonalizer

• DefaultPromptSets

These utilities operate on model weights and should be run against a copy or a versioned checkpoint. Validate edited models with held-out prompts and task-level evaluation before replacing a production artifact.

Fine-Tuning

The rewrite includes dataset preparation and student-workflow utilities under org.eclipse.deeplearning4j.llm.finetune, alongside SameDiff PEFT and alignment infrastructure. Start with:

• LLM Fine-Tuning Overview

• Instruction SFT, LoRA, and QLoRA

• Knowledge Distillation

• Preference and RL Alignment

• Quantized and Memory-Efficient Training

Vision-Language Models

The samediff-vlm artifact builds on samediff-llm and the SameDiff ONNX importer. Principal entry points include:

• VisionLanguageModel.fromDirectory(File) for a prepared multi-part model directory

• VideoVisionLanguageModel for video-aware orchestration

• VLMImagePreprocessor for image tensor preparation

• VLMPipelineExecutor for composed pipeline execution

• OCREngineFactory for registered OCR engines

• VlmEvalRunner for VLM evaluation

JavaCV and FFmpeg support is optional at runtime and is needed only for video frame extraction paths that use it.

Audio and Whisper

The samediff-audio artifact provides:

• WhisperModel.fromOnnx(...) and public GGML loading overloads

• WhisperTokenizer

• AudioLoader

• WhisperMelSpectrogram

• waveform-generation utilities such as AudioFileGenerator and SameDiffWaveformGenerator

The artifact depends on samediff-llm, the SameDiff ONNX importer, and nd4j-ggml so that its supported model-loading paths converge on SameDiff execution.

TorchScript and SafeTensors

nd4j-torchscript reads TorchScript archives and SafeTensors metadata and converts supported models to SameDiff:

import org.nd4j.autodiff.samediff.SameDiff;
import org.nd4j.torchscript.TorchScriptModelImport;

try (SameDiff model = TorchScriptModelImport.importModel("/models/model.pt")) {
    // Inspect or execute the imported SameDiff graph.
}

Related lower-level classes include TorchScriptReader, SafetensorsReader, TorchScriptToSameDiffConverter, and org.nd4j.torchscript.convert.ConversionOptions. Conversion support depends on the operators present in the archive; validate the imported graph before deployment.

Web Bindings

nd4j-webjar packages generated TypeScript FlatBuffers bindings and publishes the npm identity @eclipse/nd4j-webjar. It exposes graph and transport records such as FlatArray, FlatGraph, FlatInferenceRequest, FlatResult, and FlatResponse.

This artifact is a serialization binding, not a standalone Java inference server. Applications provide their own transport and execution service around the generated records.

Operational Boundaries

• Add exactly one ND4J runtime backend to an application unless a documented multi-backend deployment explicitly requires otherwise.

• Keep long-lived GenerationPipeline instances alive across related requests; recreating a pipeline per prompt discards opportunities for DSP-plan or prefix-cache reuse.

• Close pipelines, tokenizers, imported SameDiff graphs, and multimodal models according to their ownership contracts.

• Treat lower-level cache, batching, and offload classes as explicit integration components rather than automatically active features.

• Use DSP diagnostics for plan/capture/transfer failures and LLM evaluation APIs for generation-quality measurements; one does not replace the other.

Next Steps

• LLM Inference and Serving

• DSP Framework Overview

• GGML/GGUF Import

• Hardware Backends

• 1.0.0-rewrite Release Notes