lingvo.tasks.car.starnet module

StarNet: A sparse, targeted detection network.

Point clouds and anchors are sampled from the raw points, at centers determined by a sampler. Each center can have multiple anchors bboxes. Each center’s point cloud is featurized and then used to regress and classify the associated anchors bboxes.

The featurizer could be range from a simple MLPMax network, to more complex point-based models such as PointNet++, PointCNN, PCNN, etc.

In V1, anchor_bboxes shared the same featurization around each cell (featurized_cell), but they had different regression/classification networks for each offset/dimension/rotation that produced predictions.

In V2, we bring the featurization closer to each anchor_bbox, by featurizing at each anchor_bbox (offset) location instead. Note that anchor_bboxes at the same offset, but with different rotation/dimension priors will have the same featurization.

class lingvo.tasks.car.starnet.Builder

Bases: lingvo.tasks.car.builder_lib.ModelBuilderBase

Builder for StarNet Model.

MLPMaxFeaturizer(dims)

MLP followed by Max Featurizer.

PaddedMLPMaxFeaturizer(idims, dims, use_bn=True)

MLP followed by Max on padded points.

FC(name, idims, odims, use_bn=True, activation_fn=<function relu>)

Fully-connected layer.

Linear(name, idims, odims, params_init=None)

Linear layer for predicting residuals and classification logits.

Bias(name, dims, params_init=None)

Bias layer with optional initialization.

LinearWithBias(name, idims, odims, linear_params_init=None, bias_params_init=None)

Linear with bias layer with optional initialization.

Atten(name, depth, dims, hdims, heads, odims, keep_prob=1.0, linear_params_init=None, bias_params_init=None)

Stacked self-attention followed with a projection.

GINFeaturizer(name, fc_dims, mlp_dims, num_laser_features=1)

GIN-based Featurizer.

GINFeaturizerV2(name, fc_dims, mlp_dims, num_laser_features=1, fc_use_bn=True)

GIN-based Featurizer for Model V2.

ZerosCellFeaturizer(name, dims)

Produces features with 0 values, used for disabling cell featurizer.

class lingvo.tasks.car.starnet.LossNormType(value)

Bases: enum.Enum

An enumeration.

NO_NORM = 0

NORM_BY_NUM_POS_PER_CENTER = 1

class lingvo.tasks.car.starnet.ModelBase(*args, **kwargs)

Bases: lingvo.tasks.car.point_detector.PointDetectorBase

StarNet Detection Model.

This model expects that input_batch contains:

cell_center_xyz: [N, C, 3] cell_points_xyz: [N, C, P, 3] cell_feature: [N, C, P, 1] anchor_bboxes: [N, C, B, 7] anchor_localization_residuals: [N, C, B, 7] assigned_gt_labels: [N, C, B] assigned_cls_mask: [N, C, B]

where:

N - batch size C - num centers P - num points per center B - num anchor bboxes per center

The centers for the anchor_bboxes should match those of cell_center_xyz. Specifically, num_anchor_bboxes_per_center should match that of the corresponding input generator.

Base class implements common Decoder functions, though they can be overridden if desired.

Sub-classes are expected to implement ComputePredictions.

classmethod Params(num_classes, num_anchor_bboxes_per_center, num_laser_features=1)

Returns the layer params.

ComputeLoss(theta, predictions, input_batch)

Compute loss for the sparse detector model v1.

Parameters

• theta – A NestedMap object containing variable values of this task.

• predictions – A NestedMap object containing residuals and classification_logits.

• input_batch – A NestedMap expected to contain cell_center_xyz, cell_points_xyz, cell_feature, anchor_bboxes, anchor_localization_residuals, assigned_gt_labels, and assigned_cls_mask. See class doc string for details.

Returns

• A dict containing str keys and (metric, weight) pairs as values, where one of the keys is expected to be ‘loss’.

• A dict containing arbitrary tensors describing something about each training example, where the first dimension of each tensor is the batch index.

Return type

Two dicts

ComputePredictions(theta, input_batch)

Computes predictions for input_batch.

Parameters

• theta – A NestedMap object containing variable values of this task.

• input_batch – A NestedMap expected to contain cell_center_xyz, cell_points_xyz, cell_feature, anchor_bboxes, anchor_localization_residuals, assigned_gt_labels, and assigned_cls_mask. See class doc string for details.

Returns

A NestedMap object containing residuals and classification_logits.

_BBoxesAndLogits(input_batch, predictions)

Decode an input batch, computing predicted bboxes from residuals.

class lingvo.tasks.car.starnet.ModelV1(*args, **kwargs)

Bases: lingvo.tasks.car.starnet.ModelBase

StarNet Model V1.

In this model, each center is first featurized into a single feature vector, which is then used to predict the residuals for all the bboxes at that center. Concretely, each featurized center needs to make B * (7 + num_classes) predictions.

Effectively, anchor_bboxes shared the same featurization around each cell (featurized_cell), but have different regression/classification networks for each offset/dimension/rotation that produced predictions.

classmethod Params(num_classes, num_anchor_bboxes_per_center, num_laser_features=1)

Returns the layer params.

ComputePredictions(theta, input_batch)

Computes predictions for input_batch.

Parameters

• theta – A NestedMap object containing variable values of this task.

• input_batch – A NestedMap expected to contain cell_center_xyz, cell_points_xyz, cell_feature, anchor_bboxes, anchor_localization_residuals, assigned_gt_labels, and assigned_cls_mask. See class doc string for details.

Returns

A NestedMap object containing residuals and classification_logits.

class lingvo.tasks.car.starnet.ModelV2(*args, **kwargs)

Bases: lingvo.tasks.car.starnet.ModelBase

StarNet Model V2.

This model is similar to V1 except that featurizations are computed at the location of each anchor_bbox instead of each cell center.

In V2, we don’t share the featurization among anchor_bboxes in the same cell. Instead, the model featurizes at each anchor_bbox (offset) location. Note that anchor_bboxes at the same offset, but with different rotation/dimension priors will have the same featurization.

Note: This model makes assumptions about the ordering of anchor_bboxes, and assumes that offsets correspond to the ‘outer dimensions’; see input_generator._AnchorBoxSettings.GenerateAnchorSettings for details.

Note that the different rotation/dimension settings will have their own classification/regression heads - and these are now shared across all offsets.

In summary, we want the featurizer to be location (xyz coordinate) specific, and at each location, we want to have rotation/dimension specific regressors. This makes it more natural to leverage featurizers that produce features at these anchor-offset specific locations. For example, one could apply a PCNN/PointConv model on the entire point cloud (instead of each cell) and produce features at given anchor locations.

classmethod Params(num_classes, num_anchor_bboxes_offsets, num_anchor_bboxes_rotations, num_anchor_bboxes_dimensions, num_laser_features=1)

Returns the layer params.

_CellFeaturizer(theta, input_batch)

Featurizes each center location.

ComputePredictions(theta, input_batch)

Computes predictions for input_batch.

Parameters

• theta – A NestedMap object containing variable values of this task.

• input_batch – A NestedMap expected to contain lasers.points_xyz, lasers.points_feature, lasers.points_padding, cell_center_xyz, cell_points_xyz, cell_feature, anchor_bboxes, anchor_localization_residuals, assigned_gt_labels, and assigned_cls_mask. See class doc string for details.

Returns

A NestedMap object containing residuals and classification_logits.