The Lexer Decorator

Imports

To start using Lexer in your application, you'll need to import the Lexer decorator and LexerContext.

# Lexer imports
from lexer.api.lexer_context import LexerContext
from lexer.decorators.lexer import Lexer

LexerContext

The LexerContext class loads and maintains the custom settings you've configured for your application's use of Lexer. To learn more about configuring Lexer to your needs, check out Configuring Lexer.

LexerContext can then be initiated in your application via:

lexer_context = LexerContext()

The Decorator

The Lexer decorator can be applied to classes you make for your PyTorch models, connecting your models to the Lexer tools and APIs for performance reporting.

The decorator takes two parameters: A list of names for the model's inputs, and one for its outputs.

@Lexer(
    input_names=["input"],
    output_names=["output"],
)
class AmazingNet(nn.Module):
...

Benchmark

Now that the decorator has been attached to a model and you've configured Lexer, you can now start benchmarking your models.

...
lexer_context.benchmark(
  torch_model=torch_model,
  input=input_tensor,
  num_iterations=5
)
...

Parameters

• torch_model: A PyTorch model.
• input - An input tensor.
• num_iterations - Number of iterations to run through the model for benchmarking purposes.

Benchmark Results

• num_samples: Number of latency metric samples (based on the number of iterations through the model).
• throughput_per_sec: A measure of data points processed per second.
• variance: The variance in latency between sample runs, measured in milliseconds.
• mean_ms: The mean latency.
• p50_ms: The 50th percentile latency.
• p90_ms: The 90th percentile latency.
• p95_ms: The 95th percentile latency.
• p99_ms: The 99th percentile latency.
• engine: The engine used for model execution (Examples: Torch, ONNXRuntime)
• device: The type of hardware the model was run against (CPU or GPU)

📘

ms -> milliseconds

For all metrics appended with _ms, the metric is measured in milliseconds.

These results can also be viewed via this example print statement:

...
print(your_model.lexer.schema_json(indent=2))
...

Tying It All Together

You can check out an applied example using all of the Lexer functionality at our Use Lexer for Model Benchmarking tutorial.