Class RMSProp

java.lang.Object

org.tensorflow.framework.optimizers.Optimizer

org.tensorflow.framework.optimizers.RMSProp

public class RMSProp
extends Optimizer

Optimizer that implements the RMSProp algorithm.

The gist of RMSprop is to:

• Maintain a moving (discounted) average of the square of gradients
• Divide the gradient by the root of this average

This implementation of RMSprop uses plain momentum, not Nesterov momentum.

The centered version additionally maintains a moving average of the gradients, and uses that average to estimate the variance.

See Also:

• Hinton G, et al. 2012, lecture notes, that is inexplicably the canonical reference.

Nested Class Summary

Nested classes/interfaces inherited from class Optimizer

Optimizer.GradAndVar<T>, Optimizer.Options

Field Summary

Fields

Modifier and Type	Field	Description
static final boolean	CENTERED_DEFAULT
static final float	DECAY_DEFAULT
static final float	EPSILON_DEFAULT
static final float	LEARNING_RATE_DEFAULT
static final String	MG
static final String	MOMENTUM
static final float	MOMENTUM_DEFAULT
static final String	RMS

Fields inherited from class Optimizer

globals, graph, tf, VARIABLE_V2

Constructor Summary

Constructors

Constructor	Description
RMSProp(Graph graph)	Creates an RMSPRrop Optimizer
RMSProp(Graph graph, float learningRate)	Creates an RMSPRrop Optimizer
RMSProp(Graph graph, float learningRate, float decay, float momentum, float epsilon, boolean centered)	Creates an RMSPRrop Optimizer
RMSProp(Graph graph, String name, float learningRate)	Creates an RMSPRrop Optimizer
RMSProp(Graph graph, String name, float learningRate, float decay, float momentum, float epsilon, boolean centered)	Creates an RMSPRrop Optimizer

Method Summary

Modifier and Type	Method	Description
protected <T extends TType> Op	applyDense(Ops deps, Output<T> gradient, Output<T> variable)	Generates the gradient update operations for the specific variable and gradient.
protected void	createSlots(List<Output<? extends TType>> variables)	Performs a No-op slot creation method.
String	getOptimizerName()	Get the Name of the optimizer.
String	toString()

Methods inherited from class Optimizer

applyGradients, computeGradients, createName, createSlot, finish, getSlot, getTF, minimize, minimize, prepare

Methods inherited from class Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Details

LEARNING_RATE_DEFAULT

public static final float LEARNING_RATE_DEFAULT

See Also:

• Constant Field Values

DECAY_DEFAULT

public static final float DECAY_DEFAULT

See Also:

• Constant Field Values

MOMENTUM_DEFAULT

public static final float MOMENTUM_DEFAULT

See Also:

• Constant Field Values

EPSILON_DEFAULT

public static final float EPSILON_DEFAULT

See Also:

• Constant Field Values

CENTERED_DEFAULT

public static final boolean CENTERED_DEFAULT

See Also:

• Constant Field Values

RMS

public static final String RMS

See Also:

• Constant Field Values

MG

public static final String MG

See Also:

• Constant Field Values

MOMENTUM

public static final String MOMENTUM

See Also:

• Constant Field Values

Constructor Details

RMSProp

public RMSProp(Graph graph)

Creates an RMSPRrop Optimizer

Parameters:

graph - the TensorFlow Graph

RMSProp

public RMSProp(Graph graph,
 float learningRate)

Creates an RMSPRrop Optimizer

Parameters:

graph - the TensorFlow Graph

learningRate - the learning rate

RMSProp

public RMSProp(Graph graph,
 float learningRate,
 float decay,
 float momentum,
 float epsilon,
 boolean centered)

Creates an RMSPRrop Optimizer

Parameters:

graph - the TensorFlow Graph

learningRate - the learning rate

decay - Discounting factor for the history/coming gradient. Defaults to 0.9.

momentum - the acceleration factor, default is 0.

epsilon - A small constant for numerical stability

centered - If true, gradients are normalized by the estimated variance of the gradient; if false, by the uncentered second moment. Setting this to true may help with training, but is slightly more expensive in terms of computation and memory. Defaults to false.

RMSProp

public RMSProp(Graph graph,
 String name,
 float learningRate)

Creates an RMSPRrop Optimizer

Parameters:

graph - the TensorFlow Graph

name - the name of this Optimizer. Defaults to "RMSProp".

learningRate - the learning rate

RMSProp

public RMSProp(Graph graph,
 String name,
 float learningRate,
 float decay,
 float momentum,
 float epsilon,
 boolean centered)

Creates an RMSPRrop Optimizer

Parameters:

graph - the TensorFlow Graph

name - the name of this Optimizer. Defaults to "RMSProp".

learningRate - the learning rate

decay - Discounting factor for the history/coming gradient. Defaults to 0.9.

momentum - The acceleration factor, default is 0.

epsilon - A small constant for numerical stability

centered - If true, gradients are normalized by the estimated variance of the gradient; if false, by the uncentered second moment. Setting this to true may help with training, but is slightly more expensive in terms of computation and memory. Defaults to false.

Method Details

createSlots

protected void createSlots(List<Output<? extends TType>> variables)

Performs a No-op slot creation method.

Overrides:

createSlots in class Optimizer

Parameters:

variables - The variables to create slots for.

applyDense

protected <T extends TType> Op applyDense(Ops deps,
 Output<T> gradient,
 Output<T> variable)

Generates the gradient update operations for the specific variable and gradient.

Specified by:

applyDense in class Optimizer

Type Parameters:

T - The type of the variable.

Parameters:

gradient - The gradient to use.

variable - The variable to update.

Returns:

An operand which applies the desired optimizer update to the variable.

toString

public String toString()

Overrides:

toString in class Object

getOptimizerName

public String getOptimizerName()

Get the Name of the optimizer.

Specified by:

getOptimizerName in class Optimizer

Returns:

The optimizer name.