lingvo.tasks.car.waymo.waymo_open_input_generator module

Input generator for waymo open dataset (WaymoOD).

lingvo.tasks.car.waymo.waymo_open_input_generator._Dense(sparse, default_value=0)

lingvo.tasks.car.waymo.waymo_open_input_generator._NestedMapToParams(nmap)

class lingvo.tasks.car.waymo.waymo_open_input_generator.WaymoFrameMetadataExtractor(*args, **kwargs)

Bases: lingvo.tasks.car.input_extractor.FieldsExtractor

Extracts per frame metadata from a WaymoOD tf.Example.

Emits:

pose: [4, 4] - A float Tensor with the 4x4 transformation matrix for converting from “world” coordinates SDC coordinates.

run_segment: string scalar - The run segment identifier.

run_start_offset: int64 scalar - Offset of this scene from the start of the run segment (in microseconds).

time_of_day: string scalar - Categorical description of time of day, e.g., “Day”.

location: string scalar - Categorical description of geographical location, e.g., “location_sf”.

weather: string scalar - Categorical description of weather of scene, e.g., “sunny”.

VALIDATED_FILTER_OPTIONS = {'location': ['location_sf', 'location_phx', 'location_kir', 'location_other'], 'time_of_day': ['Day', 'Dawn/Dusk', 'Night'], 'weather': ['rain', 'rainy', 'sunny', 'unknown']}

classmethod Params()

Defaults params.

_ValidateFilterValues()

Check the filter against several blessed values.

FeatureMap()

Return a dictionary from tf.Example feature names to Features.

_Extract(features)

Extract data into Tensor format.

Shape()

The expected shape of each field.

DType()

The Dtype of each field.

Filter(outputs)

Optionally filters the data based on context info.

class lingvo.tasks.car.waymo.waymo_open_input_generator.WaymoImageExtractor(*args, **kwargs)

Bases: lingvo.tasks.car.input_extractor.FieldsExtractor

Extracts the camera image data from a WaymoOD tf.Example.

The cameras are [FRONT, FRONT_LEFT, FRONT_RIGHT, SIDE_LEFT, SIDE_RIGHT].

Emits dictionary, where each camera is a key (camera name) and the value is a NestedMap containing:

image: [height, width, 3] - Images from the corresponding cameras.

intrinsics: [9] - Instrinsics of the camera.

extrinsics: [4, 4] - Extrinsics of the camera

pose: [4, 4] - Pose of the camera when the corresponding image is taken.

velocity: [6] - Velocity of the camera when the corresponding image is taken. The first three numbers (vx, vy, vz) are velocities in world frame, in m/s. The last three numbers (roll, pitch, yaw) are the rotation rates in vehicle frame, in rad/s.

classmethod Params()

Defaults params.

FeatureMap()

Return a dictionary from tf.Example feature names to Features.

_Extract(features)

Returns the image Tensor.

Shape()

Shape of images.

DType()

Dtypes of images.

class lingvo.tasks.car.waymo.waymo_open_input_generator.WaymoLaserExtractor(*args, **kwargs)

Bases: lingvo.tasks.car.input_extractor.LaserExtractor

Extracts the raw laser data from a WaymoOD tf.Example.

classmethod Params()

Defaults params.

FeatureMap()

Return a dictionary from tf.Example feature names to Features.

_Extract(features)

Returns the laser Tensor.

class lingvo.tasks.car.waymo.waymo_open_input_generator.WaymoLaserSceneflowExtractor(*args, **kwargs)

Bases: lingvo.tasks.car.waymo.waymo_open_input_generator.WaymoLaserExtractor

Extracts the raw laser and sceneflow data from a WaymoOD tf.Example.

FeatureMap()

Return a dictionary from tf.Example feature names to Features.

_Extract(features)

Returns the laser Tensor.

Shape()

Return a NestedMap of un-batched fully-specified tf.TensorShapes.

class lingvo.tasks.car.waymo.waymo_open_input_generator.WaymoLabelExtractor(*args, **kwargs)

Bases: lingvo.tasks.car.input_extractor.FieldsExtractor

Extracts the bounding box and label info from a WaymoOD tf.Example.

Emits:

labels: [p.max_num_objects] - Integer label for each bounding box object corresponding to the index in car.open_dataset.Label.Type (shifted by 1 to have 0 represent the background class).

label_ids: [p.max_num_objects] - String unique identifier for each labeled object on a per run_segment basis. This can be used for associating objects across frames (over time).

detection_difficulties: [p.max_num_objects] - DO NOT USE FOR EVALUATION. The per-box difficulty level for detection task as defined in car.open_dataset.Label.DifficultyLevel. This is the human raters difficulty level, which does NOT include information about the number of points per box. Therefore, it is an incomplete definition of difficulty and will not correspond to the leaderboard if used to calculate metrics.

single_frame_detection_difficulties: [p.max_num_objects] - The per-box difficulty level derived via both detection_difficulties (labeler defined) and metric defined (number of points in box).

tracking_difficulties: [p.max_num_objects] - The per-box difficulty level for tracking task as defined in car.open_dataset.Label.DifficultyLevel.

bboxes_3d: [p.max_num_objects, 7] - 3D bounding box data in [x, y, z, l, w, h, heading] format. x, y, z are the object center in world coordinates; l, w, h are the dimensions of the box, and heading is the rotation angle around the z-axis. See car.open_dataset.Label.Box for definitions.

bboxes_3d_mask: [p.max_num_objects] - Mask for all the above tensors (mask is the inversion of padding).

bboxes_3d_num_points: [p.max_num_objects] - Integer for each box indicating how many points are in that ground truth box.

unfiltered_bboxes_3d_mask: [p.max_num_objects] - The mask before filtering out bboxes whose labels are not in p.filter_labels.

speed: [p.max_num_objects, 2] - The object speed in x, y.

acceleration: [p.max_num_objects, 2] - The object acceleration in x, y.

classmethod Params()

Defaults params.

FeatureMap()

Return a dictionary from tf.Example feature names to Features.

_Extract(features)

The subclass-defined implementation of Extract().

Parameters

features – A dictionary of (Sparse)Tensors which includes tensors from this extractor.

Returns

A NestedMap of output Tensors whose key names match self.Shape()’s keys.

Shape()

Shape of BBoxes.

DType()

Dtypes of BBoxes.

class lingvo.tasks.car.waymo.waymo_open_input_generator.RangeImageExtractor(*args, **kwargs)

Bases: lingvo.tasks.car.input_extractor.FieldsExtractor

Extracts the range images from a Waymo OD tf.Example.

The outputs contain the following:

Let ri_shape = [H, W] of the corresponding range image.

• For every short range laser (params.cbr_laser_names):

  For every return (params.returns):

  $LASERNAME_RETURN:

  .xyz - tf.float32 of ri_shape + [3]

  .features - tf.float32 of ri_shape + [4]

  .mask - tf.float32 of ri_shape indicating whether the laser xyz and feature at each coordinate is real or padded. A coordinate has a real point iff the mask is set to 1.

  $LASERNAME_beam_inclinations: tf.float32 [2] listing the min and max beam inclinations.

  $LASERNAME_extrinsics: tf.float32 [4, 4] extrinsics matrix.

• For every longer range laser (params.gbr_laser_names):

  For every return (params.returns):

  .xyz: tf.float32 of ri_shape + [3] .features: tf.float32 of ri_shape + [4] .mask: tf.float32 of ri_shape

  $LASERNAME_beam_inclinations: tf.float32 [64] listing the non-uniform beam inclinations for the longer range laser.

  $LASERNAME_extrinsics: tf.float32 [4, 4] extrinsics matrix

  $LASERNAME_pose: tf.float32 of ri_shape + [4, 4], which is the per-pixel pose.

On laser returns:

ri1 and ri2 are the first and second returns of the sensors.

On laser sensors:

If there are 5 total sensors, there will be 5 * len(returns) outputs.

The last dimension of range image is 4, indicating the following features:

range: (if entry is -1, it means there is no laser value there).

intensity

elongation

1. if laser point entry in ‘no label zone’, 0. otherwise.

The xyz range image output is a [H, W, 3] Tensor indicating the cartesian coordinates corresponding to each range image pixel in the range image. One should use the mask computed from the ‘range’ channel of the range image to select only the points that exist.

classmethod Params()

Defaults params.

FeatureMap()

Return a dictionary from tf.Example feature names to Features.

_Extract(features)

The subclass-defined implementation of Extract().

Parameters

features – A dictionary of (Sparse)Tensors which includes tensors from this extractor.

Returns

A NestedMap of output Tensors whose key names match self.Shape()’s keys.

_XYZFromRangeImage(lidar_image, lidar_image_mask, extrinsics, inclinations, pixel_pose=None, frame_pose=None)

Extract the cartesian coordinates from the range image.

Parameters

• lidar_image – [H, W, C] range image Tensor.

• lidar_image_mask – [H, W] boolean indicating which 2d coordinates in the lidar image are present.

• extrinsics – [4, 4] float matrix representing transformation matrix to world coordinates.

• inclinations – [V] beam inclinations vector.

• pixel_pose – [64, 2650, 4, 4] tensor representing per pixel pose of GBR.

• frame_pose – [4, 4] matrix representing vehicle to world transformation.

Returns

[H, W, 3] range image cartesian coordinates.

Shape()

Shape of BBoxes.

DType()

Dtypes of BBoxes.

class lingvo.tasks.car.waymo.waymo_open_input_generator.FilterNLZPoints(*args, **kwargs)

Bases: lingvo.tasks.car.input_preprocessors.Preprocessor

Filters points that are in no-label-zones.

This preprocessor expects features to contain the following keys: - lasers.points_xyz of shape [P, 3] - lasers.points_feature of shape [P, F]

Modifies the following features: - lasers.points_xyz of shape [P2, 3] - lasers.points_feature of shape [P2, F]

where P - P2 are the number of points dropped because the corresponding

point was in a no-label-zone.

classmethod Params()

Default params.

TransformFeatures(features)

Transforms the features for one example.

Parameters

features – A NestedMap of tensors.

Returns

A NestedMap of tensors corresponding.

TransformShapes(shapes)

Sets correct shapes corresponding to TransformFeatures.

Parameters

shapes – A NestedMap of TensorShapes, corresponding to the pre-transformed features.

Returns

A NestedMap of TensorShapes corresponding to the transformed features.

TransformDTypes(dtypes)

Sets correct dtypes corresponding to TransformFeatures.

Parameters

dtypes – A NestedMap of DTypes, corresponding to the pre-transformed features.

Returns

A NestedMap of DTypes corresponding to the transformed features.

class lingvo.tasks.car.waymo.waymo_open_input_generator.WaymoSparseLaser(*args, **kwargs)

Bases: lingvo.tasks.car.base_extractor._BaseExtractor

Sparse laser input extractor for Waymo dataset.

classmethod Params()

Defaults params.