lingvo.core.model_pruning.hparam module

Hyperparameter values.

lingvo.core.model_pruning.hparam._parse_fail(name, var_type, value, values)

Helper function for raising a value error for bad assignment.

lingvo.core.model_pruning.hparam._reuse_fail(name, values)

Helper function for raising a value error for reuse of name.

lingvo.core.model_pruning.hparam._process_scalar_value(name, parse_fn, var_type, m_dict, values, results_dictionary)

Update results_dictionary with a scalar value.

Used to update the results_dictionary to be returned by parse_values when encountering a clause with a scalar RHS (e.g. “s=5” or “arr[0]=5”.)

Mutates results_dictionary.

Parameters

• name – Name of variable in assignment (“s” or “arr”).

• parse_fn – Function for parsing the actual value.

• var_type – Type of named variable.

• m_dict – Dictionary constructed from regex parsing. m_dict[‘val’]: RHS value (scalar) m_dict[‘index’]: List index value (or None)

• values – Full expression being parsed

• results_dictionary – The dictionary being updated for return by the parsing function.

Raises

ValueError – If the name has already been used.

lingvo.core.model_pruning.hparam._process_list_value(name, parse_fn, var_type, m_dict, values, results_dictionary)

Update results_dictionary from a list of values.

Used to update results_dictionary to be returned by parse_values when encountering a clause with a list RHS (e.g. “arr=[1,2,3]”.)

Mutates results_dictionary.

Parameters

• name – Name of variable in assignment (“arr”).

• parse_fn – Function for parsing individual values.

• var_type – Type of named variable.

• m_dict – Dictionary constructed from regex parsing. m_dict[‘val’]: RHS value (scalar)

• values – Full expression being parsed

• results_dictionary – The dictionary being updated for return by the parsing function.

Raises

ValueError – If the name has an index or the values cannot be parsed.

lingvo.core.model_pruning.hparam._cast_to_type_if_compatible(name, param_type, value)

Cast hparam to the provided type, if compatible.

Parameters

• name – Name of the hparam to be cast.

• param_type – The type of the hparam.

• value – The value to be cast, if compatible.

Returns

The result of casting value to param_type.

Raises

ValueError – If the type of value is not compatible with param_type. * If param_type is a string type, but value is not. * If param_type is a boolean, but value is not, or vice versa. * If param_type is an integer type, but value is not. * If param_type is a float type, but value is not a numeric type.

lingvo.core.model_pruning.hparam.parse_values(values, type_map, ignore_unknown=False)

Parses hyperparameter values from a string into a python map.

values is a string containing comma-separated name=value pairs. For each pair, the value of the hyperparameter named name is set to value.

If a hyperparameter name appears multiple times in values, a ValueError is raised (e.g. ‘a=1,a=2’, ‘a[1]=1,a[1]=2’).

If a hyperparameter name in both an index assignment and scalar assignment, a ValueError is raised. (e.g. ‘a=[1,2,3],a[0] = 1’).

The hyperparameter name may contain ‘.’ symbols, which will result in an attribute name that is only accessible through the getattr and setattr functions. (And must be first explicit added through add_hparam.)

WARNING: Use of ‘.’ in your variable names is allowed, but is not well supported and not recommended.

The value in name=value must follows the syntax according to the type of the parameter:

• Scalar integer: A Python-parsable integer point value. E.g.: 1, 100, -12.

• Scalar float: A Python-parsable floating point value. E.g.: 1.0, -.54e89.

• Boolean: Either true or false.

• Scalar string: A non-empty sequence of characters, excluding comma, spaces, and square brackets. E.g.: foo, bar_1.

• List: A comma separated list of scalar values of the parameter type enclosed in square brackets. E.g.: [1,2,3], [1.0,1e-12], [high,low].

When index assignment is used, the corresponding type_map key should be the list name. E.g. for “arr[1]=0” the type_map must have the key “arr” (not “arr[1]”).

Parameters

• values – String. Comma separated list of name=value pairs where ‘value’ must follow the syntax described above.

• type_map – A dictionary mapping hyperparameter names to types. Note every parameter name in values must be a key in type_map. The values must conform to the types indicated, where a value V is said to conform to a type T if either V has type T, or V is a list of elements of type T. Hence, for a multidimensional parameter ‘x’ taking float values, ‘x=[0.1,0.2]’ will parse successfully if type_map[‘x’] = float.

• ignore_unknown – Bool. Whether values that are missing a type in type_map should be ignored. If set to True, a ValueError will not be raised for unknown hyperparameter type.

Returns

• A scalar value.

• A list of scalar values.

• A dictionary mapping index numbers to scalar values. (e.g. "x=5,L=[1,2],arr[1]=3" results in {'x':5,'L':[1,2],'arr':{1:3}})

Return type

A python map mapping each name to either

Raises

• ValueError – If there is a problem with input.

• * If values cannot be parsed. –

• * If a list is assigned to a list index (e.g. 'a[1] = [1,2,3]') –

• * If the same rvalue is assigned two different values (e.g. 'a=1,a=2', – ‘a[1]=1,a[1]=2’, or ‘a=1,a=[1]’)

class lingvo.core.model_pruning.hparam.HParams(model_structure=None, **kwargs)

Bases: object

Class to hold a set of hyperparameters as name-value pairs.

A HParams object holds hyperparameters used to build and train a model, such as the number of hidden units in a neural net layer or the learning rate to use when training.

You first create a HParams object by specifying the names and values of the hyperparameters.

To make them easily accessible the parameter names are added as direct attributes of the class. A typical usage is as follows:

# Create a HParams object specifying names and values of the model
# hyperparameters:
hparams = HParams(learning_rate=0.1, num_hidden_units=100)

# The hyperparameter are available as attributes of the HParams object:
hparams.learning_rate ==> 0.1
hparams.num_hidden_units ==> 100

Hyperparameters have type, which is inferred from the type of their value passed at construction type. The currently supported types are: integer, float, boolean, string, and list of integer, float, boolean, or string.

You can override hyperparameter values by calling the [parse()](#HParams.parse) method, passing a string of comma separated name=value pairs. This is intended to make it possible to override any hyperparameter values from a single command-line flag to which the user passes ‘hyper-param=value’ pairs. It avoids having to define one flag for each hyperparameter.

The syntax expected for each value depends on the type of the parameter. See parse() for a description of the syntax.

Example:

# Define a command line flag to pass name=value pairs.
# For example using argparse:
import argparse
parser = argparse.ArgumentParser(description='Train my model.')
parser.add_argument('--hparams', type=str,
                    help='Comma separated list of "name=value" pairs.')
args = parser.parse_args()
...
def my_program():
  # Create a HParams object specifying the names and values of the
  # model hyperparameters:
  hparams = tf.HParams(learning_rate=0.1, num_hidden_units=100,
                       activations=['relu', 'tanh'])

  # Override hyperparameters values by parsing the command line
  hparams.parse(args.hparams)

  # If the user passed `--hparams=learning_rate=0.3` on the command line
  # then 'hparams' has the following attributes:
  hparams.learning_rate ==> 0.3
  hparams.num_hidden_units ==> 100
  hparams.activations ==> ['relu', 'tanh']

  # If the hyperparameters are in json format use parse_json:
  hparams.parse_json('{"learning_rate": 0.3, "activations": "relu"}')

_HAS_DYNAMIC_ATTRIBUTES = True

add_hparam(name, value)

Adds {name, value} pair to hyperparameters.

Parameters

• name – Name of the hyperparameter.

• value – Value of the hyperparameter. Can be one of the following types: int, float, string, int list, float list, or string list.

Raises

ValueError – if one of the arguments is invalid.

set_hparam(name, value)

Set the value of an existing hyperparameter.

This function verifies that the type of the value matches the type of the existing hyperparameter.

Parameters

• name – Name of the hyperparameter.

• value – New value of the hyperparameter.

Raises

• KeyError – If the hyperparameter doesn’t exist.

• ValueError – If there is a type mismatch.

del_hparam(name)

Removes the hyperparameter with key ‘name’.

Does nothing if it isn’t present.

Parameters

name – Name of the hyperparameter.

parse(values)

Override existing hyperparameter values, parsing new values from a string.

See parse_values for more detail on the allowed format for values.

Parameters

values – String. Comma separated list of name=value pairs where ‘value’ must follow the syntax described above.

Returns

The HParams instance.

Raises

• ValueError – If values cannot be parsed or a hyperparameter in values

• doesn't exist. –

override_from_dict(values_dict)

Override existing hyperparameter values, parsing new values from a dictionary.

Parameters

values_dict – Dictionary of name:value pairs.

Returns

The HParams instance.

Raises

• KeyError – If a hyperparameter in values_dict doesn’t exist.

• ValueError – If values_dict cannot be parsed.

set_model_structure(model_structure)

get_model_structure()

to_json(indent=None, separators=None, sort_keys=False)

Serializes the hyperparameters into JSON.

Parameters

• indent – If a non-negative integer, JSON array elements and object members will be pretty-printed with that indent level. An indent level of 0, or negative, will only insert newlines. None (the default) selects the most compact representation.

• separators – Optional (item_separator, key_separator) tuple. Default is (', ', ': ').

• sort_keys – If True, the output dictionaries will be sorted by key.

Returns

A JSON string.

parse_json(values_json)

Override existing hyperparameter values, parsing new values from a json object.

Parameters

values_json – String containing a json object of name:value pairs.

Returns

The HParams instance.

Raises

• KeyError – If a hyperparameter in values_json doesn’t exist.

• ValueError – If values_json cannot be parsed.

values()

Return the hyperparameter values as a Python dictionary.

Returns

A dictionary with hyperparameter names as keys. The values are the hyperparameter values.

get(key, default=None)

Returns the value of key if it exists, else default.

static _get_kind_name(param_type, is_list)

Returns the field name given parameter type and is_list.

Parameters

• param_type – Data type of the hparam.

• is_list – Whether this is a list.

Returns

A string representation of the field name.

Raises

ValueError – If parameter type is not recognized.