> For the complete documentation index, see [llms.txt](https://deeplearning4j.konduit.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://deeplearning4j.konduit.ai/en-1.0.0-rewrite/deeplearning4j/overview-4/inference-and-serving.md).

# LLM Inference and Serving

Production guide to GenerationPipeline, fixed-buffer DSP reuse, constrained decoding, cross-request KV prefix caching, and cache observability.

`GenerationPipeline` is the supported high-level entry point for SameDiff text generation. It coordinates the decoder, tokenizer integration, prefill and decode lifecycle, fixed-size KV buffers, DSP plans, optional cross-request prefix cache, and native resources it creates. A caller-supplied model remains caller-owned; models loaded from paths by the pipeline are released with the pipeline.

This page describes the path that is wired into `GenerationPipeline`. The `samediff-llm` module also contains lower-level cache managers and policy classes for paged, quantized, tiered, beam, and speculative use cases. Those classes are building blocks; their presence does not mean that every policy is active in the production `GenerationPipeline` path.

## Create and Close a Pipeline

Build pipelines with `GenerationPipeline.create(...)`. The constructor is not public. A tokenizer instance and either a loaded decoder or a decoder path are required by the selected loading mode.

```java
import org.eclipse.deeplearning4j.llm.generation.GenerationPipeline;
import org.eclipse.deeplearning4j.llm.generation.GenerationPipelineConfig;
import org.eclipse.deeplearning4j.llm.generation.GenerationResult;
import org.eclipse.deeplearning4j.llm.generation.kvcache.KvCacheStrategy;
import org.eclipse.deeplearning4j.llm.generation.sampling.SamplingConfig;
import org.eclipse.deeplearning4j.llm.tokenizer.HuggingFaceTokenizer;
import org.eclipse.deeplearning4j.llm.tokenizer.Tokenizer;

Tokenizer tokenizer = HuggingFaceTokenizer.fromFile("/models/tokenizer.json");

GenerationPipelineConfig config = GenerationPipelineConfig.builder()
        .decoderPath("/models/decoder.sdz")
        .tokenizer(tokenizer)
        .samplingConfig(SamplingConfig.greedy())
        .maxNewTokens(256)
        .maxPrefillLength(2048)
        .maxKvCacheLength(4096)
        .kvCacheStrategy(KvCacheStrategy.STATIC)
        .dspEnabled(true)
        .build();

try (GenerationPipeline pipeline = GenerationPipeline.create(config)) {
    GenerationResult result = pipeline.generate("Explain KV caching.");
    System.out.println(result.getText());
} finally {
    tokenizer.close();
}
```

Closing the pipeline releases retained fixed-buffer state, prefix-cache blocks, SameDiff plans owned by the pipeline, and any models the pipeline loaded itself. A tokenizer supplied by the caller remains caller-owned and must also be closed. Keep a pipeline alive across related requests to benefit from the applicable reuse mode; do not create one per prompt.

## Native and Constrained Decode Paths

For a compatible single-model, in-graph-KV decoder, ordinary generation performs prefill and warmup token selection in Java, then hands the remaining steady-state decode to the native `autoregressive_decode` loop. This avoids a Java-to-native transition for each steady-state token. Rotating KV is an exception: physical write positions wrap, so that mode advances through a Java-controlled one-step loop.

A constrained generation is also different. When `SamplingConfig.constraintConfig` is set, each token must be checked by a Java `ConstraintMasker`. The pipeline therefore uses a Java step-by-step loop for that request while continuing to execute the decoder graph through SameDiff/DSP.

| Request                            | Token-selection path                               | Reason                                                         |
| ---------------------------------- | -------------------------------------------------- | -------------------------------------------------------------- |
| Ordinary static in-graph KV        | Java prefill/warmup, then native steady-state loop | Lowest steady-state dispatch overhead                          |
| Rotating KV                        | Java-controlled one-step loop                      | Maps logical positions to wrapping physical slots              |
| JSON/tool constrained              | Java decode loop                                   | Applies the constraint automaton before each token selection   |
| Ordinary request after constrained | Native steady-state loop again when compatible     | Constraints are per request; the native path remains available |

The same pipeline may safely process constrained and unconstrained requests in sequence. When fixed-buffer mode is enabled, a constrained request may return its retained buffers to the pipeline and the next compatible unconstrained request may reuse those stable addresses and its frozen DSP plan.

### Constrained JSON Output

```java
import org.eclipse.deeplearning4j.llm.generation.constraint.ConstraintConfig;
import org.eclipse.deeplearning4j.llm.generation.sampling.SamplingConfig;

SamplingConfig jsonSampling = SamplingConfig.greedy().toBuilder()
        .constraintConfig(ConstraintConfig.jsonObject())
        .build();

GenerationResult json = pipeline.generate(
        "Return one JSON object describing the model.",
        128,
        jsonSampling);
```

Available factories include:

* `ConstraintConfig.jsonObject()` for one syntactically valid JSON object.
* `ConstraintConfig.toolCall(...)` for the canonical JSON tool-call envelope.
* `ConstraintConfig.nativeToolCall(...)` for a model's native sentinel/function-call protocol.
* `ConstraintConfig.xmlToolCall(...)` for schema-aware XML tool calls.

Constrained generation trades throughput for structural validity because token masking currently runs in Java. A null constraint preserves the native path with no constraint-processing overhead.

## Fixed-Buffer Reuse

The continuation and prefix-reuse paths described below require a single-model decoder with an in-graph KV cache, the layout used by supported GGUF decoders. `startSession(...)` rejects two-model/external-embedding and non-in-graph-KV decoders.

Set `maxPrefillLength` to a positive value to use retained, stable-address prefill and decode buffers across independent compatible calls. It is also a hard padded-input bound: a longer prompt is left-truncated to this length, which discards the oldest tokens and may remove system instructions. Size it for the longest accepted prompt and reject oversized input in the application when truncation is not acceptable.

Set `maxKvCacheLength` to the model's context ceiling when requests with different generation budgets should share the same physical KV envelope. When both values are positive, `maxPrefillLength` must be strictly smaller than `maxKvCacheLength` so at least one decode position remains; the actual prompt must also leave room below the KV ceiling.

Static KV buffers follow the decoder input placeholders' data types, not a possibly widened prefill-output wrapper. Reuse fails closed if a retained buffer's dtype no longer matches the decoder contract rather than substituting a wider buffer into a captured external input.

With prefix caching disabled, a retained state is reused only when its physical KV length is compatible with the next request. An incompatible state is closed and rebuilt rather than being rebound to the wrong captured addresses. This fixed-buffer reuse contract applies across:

* one-shot generation followed by another one-shot generation;
* a one-shot request followed by `startSession(...)`;
* a closed continuation session followed by another compatible request;
* constrained generation followed by ordinary native generation.

`GenerationSession` remains thread-confined, and callers should not execute concurrent generations through the same pipeline unless an API explicitly documents that use.

### Fixed-Plan Reuse Versus Prefix Reuse

Cross-request prefix caching takes precedence over retained one-shot plan reuse. Before probing the prefix pool, the pipeline closes the previous cached one-shot state so a captured plan cannot alias the preceding prompt. A prefix hit reuses KV contents, but independent one-shot requests do not also retain the same frozen fixed-buffer plan while prefix caching is enabled.

Choose based on the workload: leave prefix caching disabled when repeated compatible calls should prioritize frozen-plan reuse; enable it when avoiding prefill of a long shared prompt is more valuable than retaining that one-shot plan.

## Cross-Request KV Prefix Cache

For a single-model in-graph-KV decoder, the prefix cache avoids recomputing the block-aligned part of a prompt shared with an earlier request, such as a common system prompt. It is opt-in and disabled by default.

```java
GenerationPipelineConfig config = GenerationPipelineConfig.builder()
        .decoderPath("/models/decoder.sdz")
        .tokenizer(tokenizer)
        .kvCacheStrategy(KvCacheStrategy.STATIC)
        .prefixCacheEnabled(true)
        .prefixCacheBlockSize(16)
        .prefixCacheMaxBytes(512L * 1024 * 1024)
        .maxPrefillLength(2048)
        .maxKvCacheLength(4096)
        .build();
```

### Supported Contract

For the first prefix-cache implementation:

* Use the documented `STATIC` KV strategy. Pipeline creation currently also accepts the `PAGED` enum, but prefix restoration still targets dense static buffers; it does not activate page-table prefix sharing.
* Do not combine prefix caching with rotating KV buffers.
* `QUANTIZED` and `TURBOQUANT` strategies are rejected because their scale/layout metadata cannot be safely reused as independent dense blocks.
* Do not enable cross-request prefix caching for gated-delta or other recurrent-state models in this release. Recurrent snapshots are not yet keyed strongly enough by the matched prompt identity.
* Prefixes are matched at block boundaries. The default block size is 16 tokens.
* Only complete blocks are stored. A prefill shorter than one block creates no reusable entry.
* Restoring a prefix copies cached backend-resident blocks into a newly prepared static KV buffer. On CUDA this is device-to-device copy reuse; it is not zero-copy page-table sharing.
* The pool is shared by `generate(...)` and `startSession(...)` calls on one pipeline and is released by `GenerationPipeline.close()`.

For non-recurrent transformer models, a block-aligned match that leaves a non-empty suffix restores the matched blocks and prefills only that suffix. An exact full-prompt match currently falls back to a full prefill so warmup produces the required logits correctly. Although the implementation carries experimental recurrent snapshots and can fall back when a snapshot is missing, equal-length prefixes are not yet isolated by full prompt identity. Treat recurrent/GDN prefix reuse as unsupported until that identity contract is fixed.

### Memory Budget

`prefixCacheMaxBytes` is the LRU eviction target for resident block data. Existing blocks are evicted before an insertion that would exceed it; one block larger than the configured target can still be retained after the pool is emptied. A value of `0` selects the runtime heuristic:

1. Start with 10% of total device memory.
2. Clamp the result to 64–512 MiB.
3. Use a 256 MiB fallback when device capacity cannot be determined.

A larger block size reduces trie and metadata overhead but requires a longer shared prefix. A smaller block size can find shorter partial matches at the cost of more blocks and trie entries.

## Prefix-Cache Observability

When prefix caching is enabled, access its pool through `GenerationPipeline.getPrefixBlockPool()`. The getter returns `null` when the feature is disabled.

```java
import org.eclipse.deeplearning4j.llm.generation.kvcache.KvPrefixBlockPool;
import java.util.Map;

KvPrefixBlockPool pool = pipeline.getPrefixBlockPool();
if (pool != null) {
    Map<String, Object> stats = pool.toStatsMap();
    System.out.println(stats);
}
```

The snapshot contains:

| Field               | Kind             | Meaning                                                          |
| ------------------- | ---------------- | ---------------------------------------------------------------- |
| `blockSize`         | Configuration    | Tokens represented by one reusable block                         |
| `maxByteBudget`     | Configuration    | Resolved LRU eviction target                                     |
| `residentBlocks`    | Gauge            | Distinct blocks currently in the pool                            |
| `currentBytes`      | Gauge            | Current bytes held by resident block arrays                      |
| `totalLookups`      | Lifetime counter | Radix-trie lookup operations, including misses                   |
| `totalHits`         | Lifetime counter | Lookup operations with a positive matched-token count            |
| `hitRate`           | Ratio            | `totalHits / totalLookups`, or zero before any lookup            |
| `totalHitTokens`    | Lifetime counter | Matched tokens accumulated across successful lookup operations   |
| `totalStoredTokens` | Lifetime counter | Block-aligned prefill tokens registered over the pool's lifetime |

These are lookup-operation counters, not request counters. A successful pipeline request may perform a probe lookup followed by a restore lookup, so do not interpret `totalHits` as “requests served from cache.” `clear()` removes resident blocks and zeros the resident gauges; lifetime counters remain cumulative for the pool instance.

For live diagnosis, pair the snapshot with the INFO messages:

```
KvPrefixBlockPool created: blockSize=... maxBytes=...MB
[PrefixCache] HIT: ... tokens matched (... blocks saved)
[PrefixCache] restored ... tokens into ... KV buffers
```

## Operational Checklist

* Reuse one pipeline instead of rebuilding it per request.
* Size `maxPrefillLength` for the longest accepted prompt, account for its left-truncation behavior, and keep it below a context-sized `maxKvCacheLength`.
* Choose retained frozen-plan reuse or cross-request prefix reuse with the precedence above; do not assume both apply to the same one-shot transition.
* Enable prefix caching only for non-recurrent in-graph-KV models with repeated prompts; otherwise its device memory has no benefit.
* Monitor both resident bytes and matched-token counters rather than hit rate alone.
* Close the pipeline during model replacement or application shutdown.

## Related Documentation

* [LLM & VLM Stack](/en-1.0.0-rewrite/deeplearning4j/overview-4.md)
* [DSP Framework Overview](/en-1.0.0-rewrite/nd4j/overview-2/dsp-framework.md)
* [DSP Execution Engine](/en-1.0.0-rewrite/nd4j/overview-2/dsp.md)
* [Memory and Workspaces](/en-1.0.0-rewrite/core-concepts/memory-and-workspaces.md)