> For the complete documentation index, see [llms.txt](https://deeplearning4j.konduit.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://deeplearning4j.konduit.ai/en-1.0.0-beta7/arbiter/parameter-spaces.md). # Parameter Spaces ### Overview A `ParameterSpace` defines the set of valid values for a single hyperparameter. Arbiter uses parameter spaces to sample candidate configurations during random search, or to enumerate all combinations during grid search. All parameter spaces are in the `arbiter-core` module and live under `org.deeplearning4j.arbiter.optimize.parameter`. *** ### Primitive Parameter Spaces #### ContinuousParameterSpace [\[source\]](https://github.com/eclipse/deeplearning4j/tree/master/arbiter/arbiter-core/src/main/java/org/deeplearning4j/arbiter/optimize/parameter/continuous/ContinuousParameterSpace.java) Defines a continuous range of `double` values with a uniform distribution between a minimum and maximum. ```java // Uniform in [0.0001, 0.01] ParameterSpace learningRate = new ContinuousParameterSpace(0.0001, 0.01); ``` **Constructor:** ```java ContinuousParameterSpace(double min, double max) ``` **Usage in layer builder:** ```java new DenseLayerSpace.Builder() .l2(new ContinuousParameterSpace(1e-6, 1e-3)) .build(); ``` For scale-sensitive hyperparameters like learning rate, consider `LogUniformDistribution` which is more likely to sample values across orders of magnitude evenly. `ContinuousParameterSpace` samples uniformly in linear space, which over-samples the high end. *** #### IntegerParameterSpace [\[source\]](https://github.com/eclipse/deeplearning4j/tree/master/arbiter/arbiter-core/src/main/java/org/deeplearning4j/arbiter/optimize/parameter/integer/IntegerParameterSpace.java) Defines an integer range with a uniform distribution. ```java // Uniform integer in [64, 512], inclusive ParameterSpace layerSize = new IntegerParameterSpace(64, 512); ``` **Constructor:** ```java IntegerParameterSpace(int min, int max) ``` Min and max are both inclusive. ```java // Number of layers: 1, 2, 3, or 4 ParameterSpace numLayers = new IntegerParameterSpace(1, 4); ``` **Accessor methods:** ```java int min = space.getMin(); // 64 int max = space.getMax(); // 512 ``` *** #### DiscreteParameterSpace [\[source\]](https://github.com/eclipse/deeplearning4j/tree/master/arbiter/arbiter-core/src/main/java/org/deeplearning4j/arbiter/optimize/parameter/discrete/DiscreteParameterSpace.java) Defines an unordered set of discrete values. The search strategy samples uniformly from the set. ```java // Discrete set of activation functions ParameterSpace activation = new DiscreteParameterSpace<>( Activation.RELU, Activation.TANH, Activation.ELU, Activation.LEAKYRELU ); ``` **Constructor:** ```java DiscreteParameterSpace(T... values) DiscreteParameterSpace(List values) ``` The type parameter `T` can be any Java object, including arrays: ```java // Kernel sizes: 3x3 or 5x5 ParameterSpace kernelSize = new DiscreteParameterSpace<>( new int[]{3, 3}, new int[]{5, 5} ); ``` With custom loss functions: ```java ILossFunction[] lossFunctions = { new LossMCXENT(Nd4j.create(new double[]{1.0, 2.0})), new LossMCXENT(Nd4j.create(new double[]{1.0, 5.0})), new LossMCXENT(Nd4j.create(new double[]{1.0, 10.0})) }; ParameterSpace lossSpace = new DiscreteParameterSpace<>(lossFunctions); ``` *** #### BooleanSpace [\[source\]](https://github.com/eclipse/deeplearning4j/tree/master/arbiter/arbiter-core/src/main/java/org/deeplearning4j/arbiter/optimize/parameter/BooleanSpace.java) Samples `true` or `false` with equal probability. Internally, the space maps from a continuous value: values <= 0.5 map to `true`, values > 0.5 map to `false`. ```java ParameterSpace hasBias = new BooleanSpace(); ``` **Usage:** ```java new MultiLayerSpace.Builder() .hasBias(new BooleanSpace()) // ... ``` *** #### FixedValue [\[source\]](https://github.com/eclipse/deeplearning4j/tree/master/arbiter/arbiter-core/src/main/java/org/deeplearning4j/arbiter/optimize/parameter/FixedValue.java) A `ParameterSpace` that always returns the same value. Useful for mixing fixed and variable hyperparameters in the same builder, or for locking a parameter that you want to exclude from the search: ```java // Fix the learning rate to exactly 0.001 across all candidates ParameterSpace fixedLR = new FixedValue<>(0.001); // Fix the number of added layers to 3 ParameterSpace fixedLayers = new FixedValue<>(3); ``` **Constructor:** ```java FixedValue(T value) ``` *** ### Composite Parameter Spaces #### MathOp [\[source\]](https://github.com/eclipse/deeplearning4j/tree/master/arbiter/arbiter-core/src/main/java/org/deeplearning4j/arbiter/optimize/parameter/math/MathOp.java) Applies a scalar mathematical operation to another parameter space. This allows you to derive one hyperparameter as a function of another. ```java ParameterSpace firstLayerSize = new IntegerParameterSpace(64, 256); // Second layer is 2x the first layer's size ParameterSpace secondLayerSize = new MathOp<>( firstLayerSize, Op.MUL, 2 ); ``` Available operations (`MathOp.Op`): | Op | Description | | ----- | ------------------------- | | `ADD` | `result = space + scalar` | | `SUB` | `result = space - scalar` | | `MUL` | `result = space * scalar` | | `DIV` | `result = space / scalar` | | `MOD` | `result = space % scalar` | The scalar is always on the right-hand side. The space value and the scalar must have compatible types. **Full example:** ```java ParameterSpace baseLR = new ContinuousParameterSpace(1e-4, 1e-2); // Use 10x smaller LR for fine-tuning layers ParameterSpace fineTuneLR = new MathOp<>(baseLR, Op.DIV, 10.0); MultiLayerSpace mls = new MultiLayerSpace.Builder() .addLayer(new DenseLayerSpace.Builder() .updater(new AdamSpace(baseLR)) .nOut(256) .build()) .addLayer(new OutputLayerSpace.Builder() .updater(new AdamSpace(fineTuneLR)) // correlated with baseLR .nOut(10) .build()) .numEpochs(20) .build(); ``` When a candidate is generated, `firstLayerSize` is sampled once and `secondLayerSize` is derived from it deterministically. All candidates will satisfy the relationship `secondLayerSize = 2 * firstLayerSize`. *** #### PairMathOp [\[source\]](https://github.com/eclipse/deeplearning4j/tree/master/arbiter/arbiter-core/src/main/java/org/deeplearning4j/arbiter/optimize/parameter/math/PairMathOp.java) Applies a mathematical operation to two independent parameter spaces. This allows you to create a derived parameter from two sampled values. ```java ParameterSpace l1Space = new ContinuousParameterSpace(1e-6, 1e-4); ParameterSpace l2Space = new ContinuousParameterSpace(1e-6, 1e-4); // Total regularization = l1 + l2 ParameterSpace totalReg = new PairMathOp<>(l1Space, Op.ADD, l2Space); ``` Available operations (`PairMathOp.Op`): `ADD`, `SUB`, `MUL`, `DIV`, `MOD`. *** ### Updater Spaces Arbiter provides updater-specific spaces in `org.deeplearning4j.arbiter.optimize.parameter.updater` (and `org.deeplearning4j.arbiter.conf.updater`). These allow the optimizer itself to be part of the hyperparameter search. #### AdamSpace ```java ParameterSpace updaterSpace = new AdamSpace( new ContinuousParameterSpace(1e-4, 1e-2) // learning rate ); ``` #### SgdSpace ```java ParameterSpace updaterSpace = new SgdSpace( new ContinuousParameterSpace(1e-3, 0.1) ); ``` #### NesterovSpace ```java ParameterSpace updaterSpace = new NesterovSpace( new ContinuousParameterSpace(1e-3, 0.1) ); ``` #### RmsPropSpace ```java ParameterSpace updaterSpace = new RmsPropSpace( new ContinuousParameterSpace(1e-4, 1e-2) ); ``` Use `DiscreteParameterSpace` to include the updater choice itself in the search: ```java ParameterSpace updaterSpace = new DiscreteParameterSpace<>( new Adam(0.001), new Nesterovs(0.01, 0.9), new RmsProp(0.001) ); ``` *** ### How Arbiter Samples Parameter Spaces All parameter spaces ultimately map a `double[]` input (sampled uniformly from `[0, 1]`) to a concrete value. The mapping varies by space type: | Space | Mapping | | -------------------------- | ----------------------------------------------------------------- | | `ContinuousParameterSpace` | `min + input[0] * (max - min)` | | `IntegerParameterSpace` | `(int)(min + input[0] * (max - min + 1))` clamped to `[min, max]` | | `DiscreteParameterSpace` | `values[(int)(input[0] * values.length)]` | | `BooleanSpace` | `input[0] <= 0.5` | | `FixedValue` | always returns the stored value | | `MathOp` | samples the inner space, then applies the operation | For random search, the `double[]` inputs are drawn uniformly. For grid search, they are enumerated at the specified `discretizationCount` steps. *** ### Implementing a Custom ParameterSpace You can implement `ParameterSpace` to support domain-specific sampling logic: ```java public class PowerOfTwoSpace implements ParameterSpace { private final int minExp; private final int maxExp; public PowerOfTwoSpace(int minExp, int maxExp) { this.minExp = minExp; this.maxExp = maxExp; } @Override public Integer getValue(double[] parameterValues) { int exp = minExp + (int)(parameterValues[0] * (maxExp - minExp + 1)); return 1 << Math.min(exp, maxExp); } @Override public int numParameters() { return 1; } @Override public List collectLeaves() { return Collections.singletonList(this); } @Override public boolean isLeaf() { return true; } @Override public void setIndices(int... indices) { /* store index */ } } ``` Usage: ```java // Layer size sampled from {32, 64, 128, 256, 512} ParameterSpace powerOfTwoSize = new PowerOfTwoSpace(5, 9); ``` *** ### Related Pages * [Arbiter Overview](/en-1.0.0-beta7/arbiter/overview.md) — optimization configuration and runner * [Layer Spaces](/en-1.0.0-beta7/arbiter/layer-spaces.md) — per-layer hyperparameter spaces * [Visualization](/en-1.0.0-beta7/arbiter/visualization.md) — monitoring the optimization run --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://deeplearning4j.konduit.ai/en-1.0.0-beta7/arbiter/parameter-spaces.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.