Interface BatchingOptionsOrBuilder

All Superinterfaces:
MessageLiteOrBuilder, MessageOrBuilder
All Known Implementing Classes:
BatchingOptions, BatchingOptions.Builder
@Generated public interface BatchingOptionsOrBuilder extends MessageOrBuilder

  • Method Details

    • getNumBatchThreads

      int getNumBatchThreads()
      Number of scheduling threads for processing batches of work. Determines
the number of batches processed in parallel. This should be roughly in line
with the number of TPU cores available.
      int32 num_batch_threads = 1;
      Returns:
      The numBatchThreads.

    • getMaxBatchSize

      int getMaxBatchSize()
      The maximum allowed batch size. Can be larger than allowed_batch_sizes to
utilize large batch splitting.
      int32 max_batch_size = 2;
      Returns:
      The maxBatchSize.

    • getBatchTimeoutMicros

      int getBatchTimeoutMicros()
      Maximum number of microseconds to wait before outputting an incomplete
batch.
      int32 batch_timeout_micros = 3;
      Returns:
      The batchTimeoutMicros.

    • getAllowedBatchSizesList

      List<Integer> getAllowedBatchSizesList()
      Optional list of allowed batch sizes. If left empty, does nothing.
Otherwise, supplies a list of batch sizes, causing the op to pad batches up
to one of those sizes. The entries must increase monotonically, and the
final entry must be equal or less than the max_batch_size.
      repeated int32 allowed_batch_sizes = 4;
      Returns:
      A list containing the allowedBatchSizes.

    • getAllowedBatchSizesCount

      int getAllowedBatchSizesCount()
      Optional list of allowed batch sizes. If left empty, does nothing.
Otherwise, supplies a list of batch sizes, causing the op to pad batches up
to one of those sizes. The entries must increase monotonically, and the
final entry must be equal or less than the max_batch_size.
      repeated int32 allowed_batch_sizes = 4;
      Returns:
      The count of allowedBatchSizes.

    • getAllowedBatchSizes

      int getAllowedBatchSizes(int index)
      Optional list of allowed batch sizes. If left empty, does nothing.
Otherwise, supplies a list of batch sizes, causing the op to pad batches up
to one of those sizes. The entries must increase monotonically, and the
final entry must be equal or less than the max_batch_size.
      repeated int32 allowed_batch_sizes = 4;
      Parameters:
      index - The index of the element to return.
      Returns:
      The allowedBatchSizes at the given index.

    • getMaxEnqueuedBatches

      int getMaxEnqueuedBatches()
      Maximum number of batches enqueued for processing before requests are
failed fast.
      int32 max_enqueued_batches = 5;
      Returns:
      The maxEnqueuedBatches.