Ctrlk
Community ForumND4J JavadocDL4J Javadoc
For the complete documentation index, see llms.txt. This page is also available as Markdown.

GGML/GGUF Import

Import quantized LLMs from GGUF files — architecture handlers, quantization codecs, adaptive quantization, round-trip export, and pipeline modules

GGML/GGUF Model Import

Eclipse Deeplearning4j 1.0.0-rewrite introduces native support for loading GGUF and GGML model files directly into SameDiff. This enables the JVM ecosystem to consume the enormous library of community-quantized models distributed through Hugging Face and other repositories — LLaMA, Gemma, Mistral, Phi, Qwen, Whisper, and many others — without any Python tooling or intermediate conversion step.

The implementation is split across the nd4j-ggml Maven module (87 files) and three pipeline SPI modules (34 files) that provide a pluggable format layer on top of SameDiff.

When to Use GGML Import

Scenario
Recommended approach

Run a community quantized LLM (.gguf) on the JVM

GGMLModelImport.importModel(File)

Inspect metadata and tensor layout before loading

GGMLModelImport.inspectModel(File)

Convert to DL4J native format for repeated use

GGMLModelImport.convertToSDZ(src, dst)

Export a SameDiff model back to GGUF

GGMLModelExport.exportModel(SameDiff, File, ExportOptions)

Load split multimodal GGUF bundles (e.g., Qwen3-VL)

MultimodalGGUFLoader

Maven Setup

The core import capability lives in nd4j-ggml. Add it alongside the ND4J backend for your platform.

<!-- GGML/GGUF model import -->
<dependency>
    <groupId>org.nd4j</groupId>
    <artifactId>nd4j-ggml</artifactId>
    <version>${dl4j.version}</version>
</dependency>

<!-- ND4J CPU backend (choose one) -->
<dependency>
    <groupId>org.nd4j</groupId>
    <artifactId>nd4j-native-platform</artifactId>
    <version>${dl4j.version}</version>
</dependency>

<!-- ND4J CUDA backend (alternative) -->
<!--
<dependency>
    <groupId>org.nd4j</groupId>
    <artifactId>nd4j-cuda-12.3-platform</artifactId>
    <version>${dl4j.version}</version>
</dependency>
-->

For pipeline integration (format-agnostic loading across GGUF, SafeTensors, and ONNX), add the relevant SPI modules:

Replace ${dl4j.version} with your project version, for example 1.0.0-rewrite.

Quick Start

Import a GGUF model into SameDiff

Inspect a model without loading all weights

Convert to DL4J native format (SDZ)

Run a forward pass

GGUF Format Support

File Format Versions

GGUFReader supports all three released versions of the GGUF binary format:

Version
Notes

GGUF v1

Original release format

GGUF v2

Adds alignment padding for tensors

GGUF v3

Extended metadata KV type set

All versions share the same outer structure:

  1. Magic bytes — 0x46554747 (GGUF in ASCII, little-endian)

  2. Version — uint32

  3. Tensor count — uint64

  4. Metadata KV count — uint64

  5. Metadata KV pairs — typed key-value entries (strings, scalars, arrays)

  6. Tensor descriptors — name, shape, quantization type, offset

  7. Tensor data — raw quantized bytes, padded to alignment boundary

GGMLFormatDetector reads the first four bytes of any file and selects either GGUFReader (magic 0x46554747) or the legacy GGMLReader (older magic 0x67676d6c / 0x67676d66). You never need to choose the reader manually; GGMLModelImport calls the detector automatically.

Legacy GGML Format

For pre-GGUF models (GGML format v1–v3), GGMLReader and GGMLWriter provide compatible reading and writing. These files lack the structured metadata KV section; architecture detection falls back to heuristics based on tensor name patterns.

Supported Architectures

Architecture detection is handled by ArchitectureRegistry, which uses ServiceLoader auto-discovery and a priority ordering. Each handler implements the ModelArchitecture interface:

The registry iterates handlers in priority order and delegates to the first compatible one. GenericArchitecture is always last and accepts any model as a fallback.

Architecture Handler Reference

Architecture class
Model families
Notes

LLaMAArchitecture

LLaMA 1, LLaMA 2, LLaMA 3

Standard dense transformer; RoPE positional encoding

Llama4Architecture

LLaMA 4

Interleaved mixture-of-experts (MoE) layers

GemmaArchitecture

Gemma 2, Gemma 3

Google's open models; grouped-query attention

MistralArchitecture

Mistral 7B, Mixtral

Sliding-window attention; optional MoE

PhiArchitecture

Phi-3, Phi-3.5

Microsoft; ROPE + QKV fused projection

GLMArchitecture

ChatGLM, CodeGLM

Zhipu AI; bidirectional prefix attention

GraniteArchitecture

IBM Granite

IBM Research code/language models

LFM2Architecture

Liquid LFM-2

State-space model (SSM) hybrid

NemotronArchitecture

NVIDIA Nemotron

NVIDIA instruction-tuned models

OLMoArchitecture

OLMo

Allen AI; no bias in attention

OpenELMArchitecture

OpenELM

Apple; layer-wise head count variation

Qwen3VLArchitecture

Qwen3-VL

Alibaba multimodal; loads split GGUF shards

SmolVLM2Architecture

SmolVLM2

HuggingFace compact vision-language model

MiniCPMVArchitecture

MiniCPM-V

ModelBest multimodal

WhisperArchitecture

Whisper (tiny/base/small/medium/large)

OpenAI ASR; encoder-decoder

GenericArchitecture

Any

Fallback; maps tensors by name without graph rewiring

LayerTensorDiscovery resolves GGUF tensor naming conventions (e.g., blk.0.attn_q.weight, blk.0.ffn_gate.weight) to the canonical SameDiff variable names used by each architecture.

Quantization Formats

Quantization Type Reference

GGMLQuantType enumerates every supported dtype with its bits-per-weight value. GGMLDataType provides the corresponding GGML integer dtype codes used in the binary format.

Standard quantization types

Type
Bits/weight
Block size
Notes
When to use

F32

32.0

—

Full precision float32

Accuracy-critical research; very large GPU

F16

16.0

—

Half precision float16

Good balance; standard GPU inference

Q8_0

8.5

32

8-bit, zero-point offset

Near-lossless; reference quality

Q8_K

8.5

256

8-bit, K-quant block

Used as intermediate for K-quant dequant

Q6_K

6.5625

256

6-bit K-quant

Excellent quality; fits larger models in RAM

Q5_K_M

5.6875

256

5-bit K-quant, mixed precision

Recommended for quality-size balance

Q5_K_S

5.5

256

5-bit K-quant, small

Slightly smaller than Q5_K_M

Q5_1

5.5

32

5-bit with non-zero min

Legacy; prefer Q5_K_M

Q5_0

5.5

32

5-bit, zero min

Legacy; prefer Q5_K_S

Q4_K_M

4.8

256

4-bit K-quant, mixed precision

Most popular community choice

Q4_K_S

4.375

256

4-bit K-quant, small

Compact; slightly lower quality

Q4_1

4.5

32

4-bit with non-zero min

Legacy; prefer Q4_K_M

Q4_0

4.0

32

4-bit, zero min

Smallest standard quant; legacy use

Q3_K_L

3.4375

256

3-bit K-quant, large

Very compressed; some quality loss

Q3_K_M

3.28125

256

3-bit K-quant, medium

Aggressive compression

Q3_K_S

3.0

256

3-bit K-quant, small

Extreme compression

Q2_K

2.5625

256

2-bit K-quant

Maximum compression; significant degradation

I-quant types (importance-matrix quantization)

I-quants use a calibration dataset to assign higher precision to the weights that matter most. They require an importance matrix (imatrix) generated during quantization and generally outperform equivalent-BPW standard quants.

Type
Bits/weight
Notes

IQ4_XS

4.25

4-bit imatrix; best 4-bit quality

IQ4_NL

4.0

4-bit non-linear imatrix

IQ3_XXS

3.0625

3-bit ultra-small imatrix

IQ3_S

3.4375

3-bit imatrix, standard

IQ2_XXS

2.0625

2-bit ultra-small imatrix

IQ2_XS

2.3125

2-bit imatrix, extra small

IQ2_S

2.5

2-bit imatrix, standard

IQ1_S

1.5625

1-bit imatrix; extreme compression

IQ1_M

1.75

1-bit imatrix, mixed

Ternary types

Type
Bits/weight
Notes

TQ1_0

~1.69

Ternary quant v1

TQ2_0

~2.06

Ternary quant v2; better accuracy than TQ1_0

Dequantization at Import Time

During GGMLModelImport.importModel(), each tensor is dequantized to float32 (or float16 depending on ConversionOptions) before being stored as a SameDiff variable. A dedicated dequantizer class handles each format:

  • Standard: Q4_0Dequantizer, Q4_1Dequantizer, Q5_0Dequantizer, Q5_1Dequantizer, Q8_0Dequantizer, Q8_KDequantizer, Q4_KDequantizer, Q5_KDequantizer, Q6_KDequantizer, Q2_KDequantizer, Q3_KDequantizer

  • I-quant: IQ1_MDequantizer, IQ1_SDequantizer, IQ2_SDequantizer, IQ2_XSDequantizer, IQ2_XXSDequantizer, IQ3_SDequantizer, IQ3_XXSDequantizer, IQ4_NLDequantizer, IQ4_XSDequantizer

  • Ternary: TQ1_0Dequantizer, TQ2_0Dequantizer

GGMLToSameDiffConverter coordinates this process: it iterates the tensor descriptors from GGUFReader, dispatches to the appropriate dequantizer, and creates the resulting SDVariable in the target SameDiff graph.

Adaptive Quantization

When exporting a SameDiff model back to GGUF, you rarely want uniform quantization across all layers. The adaptive quantization subsystem assigns per-layer quantization types to meet a target model size budget while preserving quality in the most sensitive weight matrices.

How it works

AdaptiveLayerQuantizer uses a two-pass algorithm:

  1. Analysis pass — DynamicQuantizationAnalyzer inspects each weight tensor's value distribution (min, max, kurtosis, outlier ratio) and computes a recommended precision level.

  2. Budget pass — AdaptiveLayerQuantizer maps the recommendations to concrete GGMLQuantType values, then adjusts to meet the total size budget. Attention projection matrices (attn_q, attn_k, attn_v, attn_output) are assigned a higher-quality quantization type than feed-forward matrices, since attention weights are more sensitive to precision loss.

Quantizer classes (for export)

The following quantizer classes are available for re-quantizing float32 tensors during export:

Class
Output type

Q4_0Quantizer

Q4_0

Q4_1Quantizer

Q4_1

Q4_KQuantizer

Q4_K

Q5_0Quantizer

Q5_0

Q5_1Quantizer

Q5_1

Q5_KQuantizer

Q5_K

Q6_KQuantizer

Q6_K

Q8_0Quantizer

Q8_0

Configuring adaptive quantization

Round-Trip Export

GGMLModelExport writes a SameDiff graph back to GGUF format, enabling workflows that:

  • Fine-tune a model inside SameDiff, then re-export for use with llama.cpp or other GGUF-native runtimes.

  • Quantize a float32 SameDiff model to GGUF for distribution.

  • Convert between quantization levels (e.g., Q8_0 -> Q4_K_M) without leaving the JVM.

Export counterpart architecture classes (e.g., LLaMAExportArchitecture) handle the reverse tensor name mapping from SameDiff variable names back to the GGUF blk.N.* naming convention. ExportArchitectureRegistry mirrors ArchitectureRegistry and uses the same ServiceLoader discovery mechanism.

Basic export

GGUFWriter handles alignment padding so that the output file is compatible with any standard GGUF reader.

Pipeline SPI Modules

The pipeline SPI provides a format-agnostic loading interface. When multiple format adapters are on the classpath, code written against samediff-pipeline-core works with any supported format without change.

Module overview

Maven artifact
Purpose

samediff-pipeline-core

Shared SPI interfaces (ModelPipelineLoader, PipelineFormat, etc.)

samediff-pipeline-ggml

Adapts nd4j-ggml behind the SPI; auto-registers via ServiceLoader

samediff-pipeline-safetensors

Loads Hugging Face SafeTensors (.safetensors) files

samediff-pipeline-onnx

Loads ONNX models through the SameDiff ONNX importer

Using the pipeline API

Explicit format selection:

Multimodal Model Support

Several vision-language and audio models distribute as multiple GGUF shards — a language model shard and one or more projection/vision encoder shards. MultimodalGGUFLoader assembles these into a unified SameDiff graph.

Supported multimodal families

Model family
Architecture handler
Shards

Qwen3-VL

Qwen3VLArchitecture

Language + vision encoder

SmolVLM2

SmolVLM2Architecture

Language + vision encoder

MiniCPM-V

MiniCPMVArchitecture

Language + vision encoder

Loading a multimodal GGUF

MultimodalGGUFLoader reads the metadata from each shard to identify its role, delegates to the appropriate Qwen3VLArchitecture (or equivalent), and merges the resulting SameDiff sub-graphs with shared variable namespaces.

API Reference

GGMLModelImport

Primary entry point for all import operations.

Method
Signature
Description

importModel

static SameDiff importModel(File file)

Reads GGUF or GGML file, dequantizes all tensors, returns populated SameDiff graph

importModel

static SameDiff importModel(File file, ConversionOptions options)

Import with custom conversion options (target dtype, layer filter, etc.)

inspectModel

static GGMLMetadata inspectModel(File file)

Reads header and metadata only; does not load tensor data

convertToSDZ

static void convertToSDZ(File src, File dst)

Converts GGUF to DL4J native SDZ format for fast subsequent loads

GGMLModelExport

Round-trip export from SameDiff back to GGUF.

Method
Signature
Description

exportModel

static void exportModel(SameDiff sd, File dst, ExportOptions options)

Writes SameDiff graph to GGUF with the given quantization options

GGUFReader

Low-level reader for the GGUF binary format.

Method
Description

readHeader()

Parses magic, version, tensor count, KV count

readMetadata()

Returns the full metadata KV map

readTensorDescriptors()

Returns list of TensorDescriptor (name, shape, quant type, data offset)

readTensorData(TensorDescriptor)

Returns raw quantized bytes for a single tensor

GGMLMetadata

Structured view of GGUF metadata KV entries.

Method
Description

getArchitecture()

Returns the general.architecture string (e.g., "llama", "gemma2")

getTensorCount()

Total number of tensors in the file

getContextLength()

llm.context_length KV value

getQuantizationType()

Most common quant type across all tensors

get(String key)

Returns raw KV value by key

ArchitectureRegistry

Method
Description

findCompatible(GGMLMetadata)

Returns first compatible ModelArchitecture in priority order

register(ModelArchitecture, int priority)

Registers a custom architecture handler

listAll()

Returns all registered handlers

GGMLQuantType

Enum of quantization types with bits-per-weight metadata.

AdaptiveLayerQuantizer

DynamicQuantizationAnalyzer

MultimodalGGUFLoader

Architecture Auto-Detection (ADR 0054)

When GGMLModelImport.importModel() is called, the first thing the importer does is read the GGUF metadata header and pass it to ArchitectureRegistry.detectArchitecture(). The registry resolves the correct handler without any user input.

How detection works

Every GGUF file written by a conforming tool sets the general.architecture key in its metadata. The value is a short ASCII string such as "llama", "mistral", "bert", or "gpt2". The detection sequence is:

  1. Read general.architecture from the GGUF KV metadata.

  2. Look up the string directly in the registry's name/variant map.

  3. If found, call canHandle(metadata) on the candidate — some handlers check secondary fields (e.g., a general.architecture of "llama" with a num_experts field triggers the MoE handler for Llama 4).

  4. If no direct match, iterate all registered handlers and call canHandle() on each.

  5. If still unresolved, fall back to GenericArchitecture, which maps every tensor by its raw GGUF name.

The handler that wins provides two things: an ArchitectureConfig (derived from metadata fields like llama.embedding_length, llama.block_count, llama.attention.head_count_kv) and a buildGraph() implementation that constructs the SameDiff computational graph.

Inspecting the detected architecture before loading

Architecture-specific features extracted from metadata

Metadata key
Description
Used by

general.architecture

Primary architecture identifier

All handlers

{arch}.embedding_length

Hidden dimension size

LLaMA, Mistral, Gemma, …

{arch}.block_count

Number of transformer layers

All transformer handlers

{arch}.attention.head_count

Number of query heads

All transformer handlers

{arch}.attention.head_count_kv

Number of KV heads (GQA)

LLaMA 3, Mistral, Gemma

{arch}.context_length

Maximum sequence length

All handlers

{arch}.rope.freq_base

RoPE frequency base

LLaMA, Mistral, Phi

{arch}.attention.layer_norm_rms_epsilon

RMSNorm epsilon

LLaMA, Mistral

GGUF tensor naming to SameDiff variable mapping

Each architecture handler declares a getTensorNamePatterns() map that LayerTensorDiscovery uses to translate GGUF block-indexed names (e.g., blk.0.attn_q.weight) to canonical SameDiff names (e.g., model.layers.0.self_attn.q_proj.weight). The LLaMA handler's mapping is representative:

GGUF tensor name
SameDiff variable name

token_embd.weight

model.embed_tokens.weight

blk.0.attn_q.weight

model.layers.0.self_attn.q_proj.weight

blk.15.ffn_gate.weight

model.layers.15.mlp.gate_proj.weight

output_norm.weight

model.norm.weight

output.weight

lm_head.weight

Quantization Handling During Import (ADR 0053)

The dequantization decision

ND4J does not support native quantized tensor operations, so the importer dequantizes each tensor to floating-point before creating the corresponding SDVariable. The target precision is controlled by ConversionOptions:

The available QuantizationMode values (set via ConversionOptions) are:

Mode
Output dtype
When to use

DEQUANTIZE_TO_FLOAT32

FLOAT32

Maximum accuracy; fine-tuning

DEQUANTIZE_TO_FLOAT16

FLOAT16

Halved memory; good for inference

DEQUANTIZE_TO_BFLOAT16

BFLOAT16

Hardware-specific (e.g., Ampere GPUs)

How dequantization works inside the importer

GGMLToSameDiffConverter iterates the tensor descriptors from GGUFReader and for each one:

  1. Reads the raw quantized bytes with GGUFReader.readTensorData(TensorDescriptor).

  2. Looks up the matching Dequantizer in DequantizerFactory by GGMLDataType.

  3. Calls dequantizer.dequantizeToArray(bytes, shape, targetDtype), which decodes the block structure and reconstructs floating-point values.

  4. Wraps the result in an INDArray and stores it as an SDVariable constant.

Block-level dequantization mechanics

Each quantization format packs values into fixed-size blocks with a shared scale (and sometimes a minimum). Two representative examples:

Q4_0 (legacy, 18 bytes per 32 values):

  • 2 bytes: FP16 scale

  • 16 bytes: 32 four-bit unsigned values (centered at 8, so value = (nibble - 8) * scale)

Q4_K (K-quant, 144 bytes per 256 values):

  • 4 bytes: super-block FP16 scale and min

  • 12 bytes: 8 sub-block 6-bit scales

  • 12 bytes: 8 sub-block 6-bit minimums

  • 128 bytes: 256 four-bit values (value = nibble * sub_scale + sub_min)

Lazy vs eager dequantization

By default, importModel dequantizes all tensors eagerly — every tensor is decoded and loaded into memory before the method returns. For very large models, you can work at the reader level to dequantize one tensor at a time:

Typical reconstruction error by format

Format
Mean absolute error
Max error

Q8_0

~0.001

~0.01

Q6_K

~0.002

~0.02

Q5_K

~0.003

~0.03

Q4_K

~0.005

~0.05

Q4_0

~0.008

~0.08

Q3_K

~0.010

~0.10

Q2_K

~0.020

~0.20

These values match the llama.cpp reference implementation; they are well within acceptable range for inference and can be corrected during fine-tuning.

GGUF Export Depth

The export path mirrors the import path with a dedicated set of classes:

  • GGMLModelExport — public entry point; calls ExportArchitectureRegistry and GGUFWriter.

  • SameDiffToGGMLConverter — coordinates the export: iterates SameDiff variables, dispatches to per-layer quantizers, writes tensor descriptors and data blocks.

  • GGUFWriter — low-level binary writer; handles alignment padding (default 32 bytes) so output files are compatible with llama.cpp and other GGUF readers.

  • ExportArchitectureRegistry — counterpart to ArchitectureRegistry; holds ExportArchitecture handlers (e.g., LLaMAExportArchitecture) that reverse-map SameDiff variable names back to the GGUF blk.N.* naming convention.

Round-trip workflow

The exported file includes re-generated GGUF metadata KV pairs derived from the SameDiff graph's variable shapes and the ExportOptions you supply.

Adaptive Quantization: AdaptiveLayerQuantizer

When you export a SameDiff model to GGUF, uniform quantization across all layers is rarely optimal. AdaptiveLayerQuantizer implements a budget-aware assignment algorithm:

Budget-aware Q2_K-to-F32 walk

Starting from the most aggressive compression (Q2_K), the quantizer walks up the precision ladder one step at a time — Q2_K → Q3_K_S → Q3_K_M → Q4_K_S → Q4_K_M → Q5_K_S → Q5_K_M → Q6_K → Q8_0 → F16 → F32 — assigning higher precision to layers that have the highest sensitivity score until the cumulative model size would exceed the target budget.

Sensitivity is computed by DynamicQuantizationAnalyzer, which examines each weight tensor's value distribution (min, max, kurtosis, outlier ratio).

Protected layers

Two categories of layers are always assigned higher precision, regardless of budget:

  • Embedding matrices (token_embd.weight, output.weight): these span the entire vocabulary and errors compound across all token positions.

  • First and last transformer blocks: errors in the initial and final blocks have the largest impact on output quality.

All other layers compete in the budget walk.

Building a quantization plan without exporting

Exporting with adaptive quantization

Unified Model Loading: AutoModel and the Pipeline SPI

The samediff-pipeline-core module provides a format-agnostic loading API modeled on HuggingFace's from_pretrained pattern. AutoModel is the single entry point; the actual loading is delegated to whichever PipelineLoader implementation handles the detected format.

AutoModel.fromPretrained

AutoModel.fromPretrained accepts a path to either a single model file or a directory containing a model with a manifest. It detects the format from the file extension and magic bytes and dispatches to the registered loader.

Supported format values (ModelFormat enum)

Enum constant
File extension(s)
Description

GGUF

.gguf, .ggml

GGML Universal Format

SAFETENSORS

.safetensors

HuggingFace SafeTensors

ONNX

.onnx

Open Neural Network Exchange

SDZ

.sdz

DL4J native SameDiff ZIP

PYTORCH

.pt, .pth, .bin

PyTorch pickle/TorchScript

ModelFormat.fromFilename(String) and ModelFormat.fromExtension(String) resolve the enum from a file name or bare extension string.

Customizing load behavior

When cacheConvertedModel is true, AutoModel saves the resulting SameDiff graph as a .sdz file next to the cache directory after the first conversion. Subsequent calls load from that cache, skipping dequantization entirely.

PipelineLoader SPI

Each format adapter implements PipelineLoader and registers itself via Java ServiceLoader. To add a custom format:

  1. Implement PipelineLoader.

  2. Add a META-INF/services/org.eclipse.deeplearning4j.pipeline.PipelineLoader file listing your class.

  3. Put the jar on the classpath. PipelineLoaderRegistry discovers it automatically.

Loading a multi-component pipeline

For multimodal models that expose several sub-graphs (language model + vision encoder), use pipelineFromPretrained:

SafeTensors Import

HuggingFace SafeTensors (.safetensors) files can be loaded through the unified AutoModel API or directly using SafeTensorsReader. Add the pipeline adapter to your Maven dependencies:

Via AutoModel (recommended)

AutoModel detects the .safetensors extension, resolves SafeTensorsPipelineLoader, and returns a populated SameDiff graph.

Via SafeTensorsReader directly

SafeTensorsReader gives access to the raw tensor map without constructing a SameDiff graph, which is useful for inspection or custom assembly:

SafeTensorsReader reads the 8-byte little-endian header length, parses the JSON tensor index, then seeks directly to each tensor's data region using a RandomAccessFile + FileChannel — no intermediate copy of the full file into RAM.

Supported SafeTensors dtypes

SafeTensors dtype
ND4J DataType

F32

FLOAT

F16

HALF

BF16

BFLOAT16

F64

DOUBLE

I64

LONG

I32

INT

I16

SHORT

I8

BYTE

U8

UBYTE

BOOL

BOOL

TorchScript Import

PyTorch .pt ZIP archives and .safetensors weight files can be imported via the nd4j-torchscript module. Add it to your Maven dependencies:

Importing a .pt file

TorchScript .pt files are ZIP archives containing a data.pkl (Python pickle) and numbered tensor data files. PickleParser reads the pickle stream to reconstruct the model graph, and TorchScriptReader maps each tensor to its data file.

Inspecting a file before import

Supported architectures (TorchScript)

Architecture detection in TorchScriptToSameDiffConverter is pattern-based — it looks for characteristic weight names in the tensor map:

Architecture
Detection signals

ResNet

layer1.0.conv1.weight, layer1.0.downsample.0, fc.weight

VGG

features.0.weight, classifier.0.weight

EfficientNet

_conv_stem.weight, _blocks.0._expand_conv.weight, _blocks.0._se_reduce.weight

Generic CNN

Fallback for unrecognized patterns

Convert a .pt model to SDZ for faster repeated loading

PyTorch-to-ND4J weight layout transformations

PyTorch and ND4J use different memory layouts for convolution and linear weights. TorchScriptToSameDiffConverter applies these automatically:

Layer type
PyTorch shape
ND4J shape
Transformation

Conv2D weights

[out, in, kH, kW]

[kH, kW, in, out]

permute(2, 3, 1, 0)

Linear weights

[out, in]

[in, out]

transpose()

Format Detection and Custom Architectures

Adding a custom architecture

Implement ModelArchitecture and register it either programmatically or via ServiceLoader:

Register programmatically (highest priority wins):

Or register via ServiceLoader by adding a file:

containing the fully qualified class name of your implementation. The registry picks it up automatically at startup.

Troubleshooting

UnsupportedFormatException: Not a GGUF file: the file does not start with magic bytes 0x46554747. Verify the file is a valid GGUF and was not corrupted during download. Use GGMLFormatDetector.detect(file) to check before importing.

UnknownArchitectureException: none of the registered architecture handlers returned true from isCompatible(). The model's general.architecture metadata key may be absent or set to an unrecognized value. GenericArchitecture should handle this as a fallback; if it does not, inspect the metadata with GGMLModelImport.inspectModel() and register a custom handler.

Out of memory during dequantization: dequantizing to float32 expands each tensor significantly. A 4-bit quantized 8B model is ~4 GB on disk but expands to ~16 GB in float32. Use ConversionOptions.targetDtype(DataType.FLOAT16) to halve the memory footprint, or process tensors one at a time using GGUFReader directly.

Slow first load: importModel dequantizes every tensor synchronously. For repeated use, call convertToSDZ once to produce a native .sdz file, which loads approximately 3-5x faster on subsequent runs.

Split shard ordering: MultimodalGGUFLoader expects shards in a specific order (language model first). Pass the files in the order listed in the model card. Incorrect ordering results in mismatched tensor namespaces.

ServiceLoader not finding pipeline adapters: ensure the samediff-pipeline-ggml jar is on the runtime classpath (not just compile scope). The SPI registration in META-INF/services/ must be present in the deployed artifact.

Further Reading

PreviousONNX Import & Export (Expanded)NextONNX Runtime

Last updated

Was this helpful?