Ctrlk
Community ForumND4J JavadocDL4J Javadoc
For the complete documentation index, see llms.txt. This page is also available as Markdown.

LLM Inference and Serving

Production guide to GenerationPipeline, fixed-buffer DSP reuse, constrained decoding, cross-request KV prefix caching, and cache observability.

GenerationPipeline is the supported high-level entry point for SameDiff text generation. It coordinates the decoder, tokenizer integration, prefill and decode lifecycle, fixed-size KV buffers, DSP plans, optional cross-request prefix cache, and native resources it creates. A caller-supplied model remains caller-owned; models loaded from paths by the pipeline are released with the pipeline.

This page describes the path that is wired into GenerationPipeline. The samediff-llm module also contains lower-level cache managers and policy classes for paged, quantized, tiered, beam, and speculative use cases. Those classes are building blocks; their presence does not mean that every policy is active in the production GenerationPipeline path.

Create and Close a Pipeline

Build pipelines with GenerationPipeline.create(...). The constructor is not public. A tokenizer instance and either a loaded decoder or a decoder path are required by the selected loading mode.

import org.eclipse.deeplearning4j.llm.generation.GenerationPipeline;
import org.eclipse.deeplearning4j.llm.generation.GenerationPipelineConfig;
import org.eclipse.deeplearning4j.llm.generation.GenerationResult;
import org.eclipse.deeplearning4j.llm.generation.kvcache.KvCacheStrategy;
import org.eclipse.deeplearning4j.llm.generation.sampling.SamplingConfig;
import org.eclipse.deeplearning4j.llm.tokenizer.HuggingFaceTokenizer;
import org.eclipse.deeplearning4j.llm.tokenizer.Tokenizer;

Tokenizer tokenizer = HuggingFaceTokenizer.fromFile("/models/tokenizer.json");

GenerationPipelineConfig config = GenerationPipelineConfig.builder()
        .decoderPath("/models/decoder.sdz")
        .tokenizer(tokenizer)
        .samplingConfig(SamplingConfig.greedy())
        .maxNewTokens(256)
        .maxPrefillLength(2048)
        .maxKvCacheLength(4096)
        .kvCacheStrategy(KvCacheStrategy.STATIC)
        .dspEnabled(true)
        .build();

try (GenerationPipeline pipeline = GenerationPipeline.create(config)) {
    GenerationResult result = pipeline.generate("Explain KV caching.");
    System.out.println(result.getText());
} finally {
    tokenizer.close();
}

Closing the pipeline releases retained fixed-buffer state, prefix-cache blocks, SameDiff plans owned by the pipeline, and any models the pipeline loaded itself. A tokenizer supplied by the caller remains caller-owned and must also be closed. Keep a pipeline alive across related requests to benefit from the applicable reuse mode; do not create one per prompt.

Native and Constrained Decode Paths

For a compatible single-model, in-graph-KV decoder, ordinary generation performs prefill and warmup token selection in Java, then hands the remaining steady-state decode to the native autoregressive_decode loop. This avoids a Java-to-native transition for each steady-state token. Rotating KV is an exception: physical write positions wrap, so that mode advances through a Java-controlled one-step loop.

A constrained generation is also different. When SamplingConfig.constraintConfig is set, each token must be checked by a Java ConstraintMasker. The pipeline therefore uses a Java step-by-step loop for that request while continuing to execute the decoder graph through SameDiff/DSP.

Request
Token-selection path
Reason

Ordinary static in-graph KV

Java prefill/warmup, then native steady-state loop

Lowest steady-state dispatch overhead

Rotating KV

Java-controlled one-step loop

Maps logical positions to wrapping physical slots

JSON/tool constrained

Java decode loop

Applies the constraint automaton before each token selection

Ordinary request after constrained

Native steady-state loop again when compatible

Constraints are per request; the native path remains available

The same pipeline may safely process constrained and unconstrained requests in sequence. When fixed-buffer mode is enabled, a constrained request may return its retained buffers to the pipeline and the next compatible unconstrained request may reuse those stable addresses and its frozen DSP plan.

Constrained JSON Output

Available factories include:

  • ConstraintConfig.jsonObject() for one syntactically valid JSON object.

  • ConstraintConfig.toolCall(...) for the canonical JSON tool-call envelope.

  • ConstraintConfig.nativeToolCall(...) for a model's native sentinel/function-call protocol.

  • ConstraintConfig.xmlToolCall(...) for schema-aware XML tool calls.

Constrained generation trades throughput for structural validity because token masking currently runs in Java. A null constraint preserves the native path with no constraint-processing overhead.

Fixed-Buffer Reuse

The continuation and prefix-reuse paths described below require a single-model decoder with an in-graph KV cache, the layout used by supported GGUF decoders. startSession(...) rejects two-model/external-embedding and non-in-graph-KV decoders.

Set maxPrefillLength to a positive value to use retained, stable-address prefill and decode buffers across independent compatible calls. It is also a hard padded-input bound: a longer prompt is left-truncated to this length, which discards the oldest tokens and may remove system instructions. Size it for the longest accepted prompt and reject oversized input in the application when truncation is not acceptable.

Set maxKvCacheLength to the model's context ceiling when requests with different generation budgets should share the same physical KV envelope. When both values are positive, maxPrefillLength must be strictly smaller than maxKvCacheLength so at least one decode position remains; the actual prompt must also leave room below the KV ceiling.

Static KV buffers follow the decoder input placeholders' data types, not a possibly widened prefill-output wrapper. Reuse fails closed if a retained buffer's dtype no longer matches the decoder contract rather than substituting a wider buffer into a captured external input.

With prefix caching disabled, a retained state is reused only when its physical KV length is compatible with the next request. An incompatible state is closed and rebuilt rather than being rebound to the wrong captured addresses. This fixed-buffer reuse contract applies across:

  • one-shot generation followed by another one-shot generation;

  • a one-shot request followed by startSession(...);

  • a closed continuation session followed by another compatible request;

  • constrained generation followed by ordinary native generation.

GenerationSession remains thread-confined, and callers should not execute concurrent generations through the same pipeline unless an API explicitly documents that use.

Fixed-Plan Reuse Versus Prefix Reuse

Cross-request prefix caching takes precedence over retained one-shot plan reuse. Before probing the prefix pool, the pipeline closes the previous cached one-shot state so a captured plan cannot alias the preceding prompt. A prefix hit reuses KV contents, but independent one-shot requests do not also retain the same frozen fixed-buffer plan while prefix caching is enabled.

Choose based on the workload: leave prefix caching disabled when repeated compatible calls should prioritize frozen-plan reuse; enable it when avoiding prefill of a long shared prompt is more valuable than retaining that one-shot plan.

Cross-Request KV Prefix Cache

For a single-model in-graph-KV decoder, the prefix cache avoids recomputing the block-aligned part of a prompt shared with an earlier request, such as a common system prompt. It is opt-in and disabled by default.

Supported Contract

For the first prefix-cache implementation:

  • Use the documented STATIC KV strategy. Pipeline creation currently also accepts the PAGED enum, but prefix restoration still targets dense static buffers; it does not activate page-table prefix sharing.

  • Do not combine prefix caching with rotating KV buffers.

  • QUANTIZED and TURBOQUANT strategies are rejected because their scale/layout metadata cannot be safely reused as independent dense blocks.

  • Do not enable cross-request prefix caching for gated-delta or other recurrent-state models in this release. Recurrent snapshots are not yet keyed strongly enough by the matched prompt identity.

  • Prefixes are matched at block boundaries. The default block size is 16 tokens.

  • Only complete blocks are stored. A prefill shorter than one block creates no reusable entry.

  • Restoring a prefix copies cached backend-resident blocks into a newly prepared static KV buffer. On CUDA this is device-to-device copy reuse; it is not zero-copy page-table sharing.

  • The pool is shared by generate(...) and startSession(...) calls on one pipeline and is released by GenerationPipeline.close().

For non-recurrent transformer models, a block-aligned match that leaves a non-empty suffix restores the matched blocks and prefills only that suffix. An exact full-prompt match currently falls back to a full prefill so warmup produces the required logits correctly. Although the implementation carries experimental recurrent snapshots and can fall back when a snapshot is missing, equal-length prefixes are not yet isolated by full prompt identity. Treat recurrent/GDN prefix reuse as unsupported until that identity contract is fixed.

Memory Budget

prefixCacheMaxBytes is the LRU eviction target for resident block data. Existing blocks are evicted before an insertion that would exceed it; one block larger than the configured target can still be retained after the pool is emptied. A value of 0 selects the runtime heuristic:

  1. Start with 10% of total device memory.

  2. Clamp the result to 64–512 MiB.

  3. Use a 256 MiB fallback when device capacity cannot be determined.

A larger block size reduces trie and metadata overhead but requires a longer shared prefix. A smaller block size can find shorter partial matches at the cost of more blocks and trie entries.

Prefix-Cache Observability

When prefix caching is enabled, access its pool through GenerationPipeline.getPrefixBlockPool(). The getter returns null when the feature is disabled.

The snapshot contains:

Field
Kind
Meaning

blockSize

Configuration

Tokens represented by one reusable block

maxByteBudget

Configuration

Resolved LRU eviction target

residentBlocks

Gauge

Distinct blocks currently in the pool

currentBytes

Gauge

Current bytes held by resident block arrays

totalLookups

Lifetime counter

Radix-trie lookup operations, including misses

totalHits

Lifetime counter

Lookup operations with a positive matched-token count

hitRate

Ratio

totalHits / totalLookups, or zero before any lookup

totalHitTokens

Lifetime counter

Matched tokens accumulated across successful lookup operations

totalStoredTokens

Lifetime counter

Block-aligned prefill tokens registered over the pool's lifetime

These are lookup-operation counters, not request counters. A successful pipeline request may perform a probe lookup followed by a restore lookup, so do not interpret totalHits as “requests served from cache.” clear() removes resident blocks and zeros the resident gauges; lifetime counters remain cumulative for the pool instance.

For live diagnosis, pair the snapshot with the INFO messages:

Operational Checklist

  • Reuse one pipeline instead of rebuilding it per request.

  • Size maxPrefillLength for the longest accepted prompt, account for its left-truncation behavior, and keep it below a context-sized maxKvCacheLength.

  • Choose retained frozen-plan reuse or cross-request prefix reuse with the precedence above; do not assume both apply to the same one-shot transition.

  • Enable prefix caching only for non-recurrent in-graph-KV models with repeated prompts; otherwise its device memory has no benefit.

  • Monitor both resident bytes and matched-token counters rather than hit rate alone.

  • Close the pipeline during model replacement or application shutdown.

Related Documentation

PreviousLLM & VLM Stack
NextLLM Fine-Tuning Overview

Last updated

Was this helpful?