The Lexer Decorator
The Lexer Decorator
To start using Lexer in your application, you'll need to import the
Lexer decorator and
# Lexer imports from lexer.api.lexer_context import LexerContext from lexer.decorators.lexer import Lexer
LexerContext class loads and maintains the custom settings you've configured for your application's use of Lexer. To learn more about configuring Lexer to your needs, check out Configuring Lexer.
LexerContext can then be initiated in your application via:
lexer_context = LexerContext()
The Lexer decorator can be applied to classes you make for your PyTorch models, connecting your models to the Lexer tools and APIs for performance reporting.
The decorator takes two parameters: A list of names for the model's inputs, and one for its outputs.
@Lexer( input_names=["input"], output_names=["output"], ) class AmazingNet(nn.Module): ...
Now that the decorator has been attached to a model and you've configured Lexer, you can now start benchmarking your models.
... lexer_context.benchmark( torch_model=torch_model, input=input_tensor, num_iterations=5 ) ...
torch_model: A PyTorch model.
input- An input tensor.
num_iterations- Number of iterations to run through the model for benchmarking purposes.
num_samples: Number of latency metric samples (based on the number of iterations through the model).
throughput_per_sec: A measure of data points processed per second.
variance: The variance in latency between sample runs, measured in milliseconds.
mean_ms: The mean latency.
p50_ms: The 50th percentile latency.
p90_ms: The 90th percentile latency.
p95_ms: The 95th percentile latency.
p99_ms: The 99th percentile latency.
engine: The engine used for model execution (Examples: Torch, ONNXRuntime)
device: The type of hardware the model was run against (CPU or GPU)
ms -> milliseconds
For all metrics appended with
_ms, the metric is measured in milliseconds.
These results can also be viewed via this example print statement:
... print(your_model.lexer.schema_json(indent=2)) ...
Tying It All Together
You can check out an applied example using all of the Lexer functionality at our Use Lexer for Model Benchmarking tutorial.
Updated about 2 months ago