Posted on
• Artificial Intelligence

TensorRT on Linux

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

TensorRT on Linux: Turbocharge GPU Inference the Right Way

If you’ve ever watched a great GPU sit idle while your model struggles to hit latency targets, this post is for you. TensorRT squeezes the last drops of performance out of NVIDIA GPUs for deep-learning inference—often cutting latency and boosting throughput dramatically with FP16/INT8 optimizations, layer fusion, and kernel auto-tuning. On Linux, you get first-class support, clean package installs, and a scriptable toolchain that plays perfectly with Bash.

This guide explains what TensorRT is and why it matters, shows you how to install it on Linux with apt, dnf, and zypper, and walks through practical steps to convert and benchmark a model. By the end, you’ll have a reproducible path from “trained model” to “production-grade engine.”

Why TensorRT on Linux?

  • Performance and cost: TensorRT specializes in fast inference. FP16/INT8 precision, tactic selection, and graph optimizations cut costs per query and allow denser deployments.

  • Production readiness: A stable C++/Python API, mature tooling (trtexec), and predictable engine files (.plan) make it ideal for services.

  • Ecosystem fit: Works with ONNX, PyTorch via Torch-TensorRT, and ONNX Runtime’s TensorRT Execution Provider. Containers and repos are maintained by NVIDIA.

  • Linux-native workflows: Bash-friendly install/upgrade, systemd services, and CI/CD pipelines—Linux is the natural home for GPU inference ops.

Prerequisites

  • NVIDIA GPU with recent driver. Check:
nvidia-smi
  • CUDA toolkit compatible with your TensorRT version (see NVIDIA’s compatibility matrix in the release notes).
nvcc --version

  • Optional but recommended: cuDNN installed from NVIDIA’s Machine Learning repo (we’ll include it below).

  • Python 3.8+ if you plan to use the Python API.

Tip: For clean environments, consider using a new virtualenv/conda env or containers.

Install TensorRT on Linux (apt, dnf, zypper)

TensorRT is distributed via NVIDIA’s “machine-learning” repositories. Choose the commands for your distro family and set the version variable to match your OS.

Ubuntu/Debian (apt)

Replace UBUNTU_VER with your series (e.g., 2004 for Ubuntu 20.04, 2204 for Ubuntu 22.04).

# Set your Ubuntu series number (no dot)
UBUNTU_VER=2204

# Prereqs
sudo apt-get update
sudo apt-get install -y curl gnupg ca-certificates

# Add NVIDIA Machine Learning repo (imports the repo GPG key and source list)
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu${UBUNTU_VER}/x86_64/repodata/repomd.xml.key \
  | sudo gpg --dearmor -o /etc/apt/keyrings/nvidia-ml.gpg
echo "deb [signed-by=/etc/apt/keyrings/nvidia-ml.gpg] https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu${UBUNTU_VER}/x86_64 /" \
  | sudo tee /etc/apt/sources.list.d/nvidia-ml.list

sudo apt-get update

# Install TensorRT (C++ runtime, dev headers, Python bindings, and CLI tools)
sudo apt-get install -y \
  tensorrt \
  libnvinfer-dev \
  python3-libnvinfer \
  libnvinfer-bin

# Optional: cuDNN (recommended)
sudo apt-get install -y libcudnn8 libcudnn8-dev

RHEL/CentOS/Rocky/Alma (dnf/yum)

Replace RHEL_VER with 8 or 9 to match your OS.

# Set your RHEL major version (8 or 9)
RHEL_VER=9

sudo dnf install -y dnf-plugins-core

# Add NVIDIA Machine Learning repo
sudo tee /etc/yum.repos.d/nvidia-ml.repo >/dev/null <<EOF
[nvidia-ml]
name=NVIDIA Machine Learning Repository
baseurl=https://developer.download.nvidia.com/compute/machine-learning/repos/rhel${RHEL_VER}/x86_64
enabled=1
gpgcheck=1
gpgkey=https://developer.download.nvidia.com/compute/machine-learning/repos/rhel${RHEL_VER}/x86_64/repodata/repomd.xml.key
EOF

sudo dnf clean all
sudo dnf makecache

# Install TensorRT and bindings
sudo dnf install -y \
  tensorrt \
  libnvinfer-devel \
  python3-libnvinfer

# Optional: cuDNN
sudo dnf install -y libcudnn8 libcudnn8-devel

SUSE/openSUSE (zypper)

Replace SLES_VER with 15 for SLES 15/openSUSE Leap 15.x.

# Set your SLES series
SLES_VER=15

# Add NVIDIA Machine Learning repo and import its key
sudo zypper ar -f https://developer.download.nvidia.com/compute/machine-learning/repos/sles${SLES_VER}/x86_64 nvidia-ml
sudo rpm --import https://developer.download.nvidia.com/compute/machine-learning/repos/sles${SLES_VER}/x86_64/repodata/repomd.xml.key
sudo zypper refresh

# Install TensorRT and bindings
sudo zypper install -y \
  tensorrt \
  libnvinfer-devel \
  python3-libnvinfer

# Optional: cuDNN
sudo zypper install -y libcudnn8 libcudnn8-devel

Prefer containers?

NVIDIA publishes ready-to-run containers with CUDA, cuDNN, and TensorRT preinstalled.

# Requires recent Docker and NVIDIA Container Toolkit (nvidia-docker2)
docker run --gpus all -it --rm nvcr.io/nvidia/tensorrt:24.06-py3 bash

Verify your install

# Driver and GPU
nvidia-smi

# TensorRT CLI tool (trtexec)
trtexec --version

# Python bindings
python3 - <<'PY'
import tensorrt as trt
print("TensorRT Python version:", trt.__version__)
PY

If these commands run without errors, you’re ready to build engines.

From model to engine: Convert and benchmark with trtexec

trtexec is your Swiss Army knife for building and benchmarking TensorRT engines from ONNX or UFF, and for loading .plan files.

1) Convert an ONNX model to a TensorRT engine (FP16)

# Example: ResNet50 ONNX
trtexec \
  --onnx=resnet50.onnx \
  --saveEngine=resnet50_fp16.plan \
  --fp16 \
  --workspace=4096

2) Use dynamic shapes for variable batch sizes

trtexec \
  --onnx=resnet50.onnx \
  --saveEngine=resnet50_dynamic.plan \
  --fp16 \
  --minShapes=input:1x3x224x224 \
  --optShapes=input:8x3x224x224 \
  --maxShapes=input:32x3x224x224

3) Benchmark an existing engine

trtexec \
  --loadEngine=resnet50_fp16.plan \
  --separateProfileRun \
  --iterations=200 \
  --avgRuns=10 \
  --streams=2

4) INT8 calibration (when you have a representative dataset)

  • Prepare a calibration cache with your calibrator (custom or via trtexec’s options if applicable).

  • Build with INT8:

trtexec \
  --onnx=resnet50.onnx \
  --int8 \
  --calib=calib.cache \
  --saveEngine=resnet50_int8.plan

Note: INT8 usually gives the biggest boost, but you must verify accuracy on your task.

Load and run an engine with Python

Below is a minimal example that loads a serialized engine and runs inference. It uses CUDA Python for memory management.

# pip install cuda-python numpy
import numpy as np
import tensorrt as trt
import cuda

logger = trt.Logger(trt.Logger.WARNING)
runtime = trt.Runtime(logger)

with open("resnet50_fp16.plan", "rb") as f:
    engine = runtime.deserialize_cuda_engine(f.read())
context = engine.create_execution_context()

# Example input shape and binding names may differ in your model
input_idx = engine.get_binding_index("input")
output_idx = engine.get_binding_index("output")

input_shape = context.get_binding_shape(input_idx)
h_in = np.random.rand(*input_shape).astype(np.float16)
h_out = np.empty(context.get_binding_shape(output_idx), dtype=np.float16)

# Allocate device buffers
stream = cuda.Stream()
d_in = cuda.mem_alloc(h_in.nbytes)
d_out = cuda.mem_alloc(h_out.nbytes)

# Transfer to device, execute, bring results back
cuda.memcpy_htod_async(d_in, h_in, stream)
context.execute_v2([int(d_in), int(d_out)])
cuda.memcpy_dtoh_async(h_out, d_out, stream)
stream.synchronize()

print("Output shape:", h_out.shape)

For production, wrap this in a service (FastAPI/gRPC), use pinned host memory, and reuse contexts and buffers.

4 high-impact tips to get the most out of TensorRT

  • Choose the right precision: Start with --fp16; if accuracy holds, evaluate --int8 with proper calibration for maximum speedup.

  • Optimize shapes and batching: Use dynamic shapes and size your --optShapes for your common batch to help the tactic selector.

  • Increase concurrency: Use multiple CUDA streams or instance groups in your serving layer to raise GPU occupancy.

  • Profile early: trtexec’s timing plus Nsight Systems/Compute will tell you where memory or kernel hotspots are. Fixing I/O (pinned memory, larger page-locked buffers) often wins easy latency.

Real-world patterns

  • Vision microservices: Convert ONNX classification/detection models to FP16 engines; serve via Triton Inference Server with TensorRT backend.

  • LLM/embeddings: Use TensorRT-LLM or ONNX Runtime with TensorRT EP for transformer acceleration; pin sequence lengths and GPU memory pools for consistent latency.

  • Edge devices: On Jetson, install via NVIDIA’s JetPack repo; leverage DLA cores where available for power efficiency.

Troubleshooting quick hits

  • Missing symbols or version mismatches: Ensure CUDA, cuDNN, and TensorRT versions are compatible (match major versions from release notes).

  • “No viable tactics” at build time: Reduce max workspace, adjust shapes, or update GPU driver/CUDA. Sometimes lowering batch size helps.

  • A/B accuracy drift with INT8: Improve the calibration dataset diversity; consider per-channel quantization if supported for your model.

Conclusion and next steps

TensorRT on Linux gives you a clean, scriptable route from trained model to high-performance inference. You now have:

  • A reproducible install for apt, dnf, and zypper.

  • A way to convert, benchmark, and deploy engines.

  • Practical knobs (precision, shapes, streams) to hit your latency and throughput goals.

Call to action:

  • Try converting one of your ONNX models today with trtexec, starting with --fp16.

  • Benchmark with your real input shapes and batch sizes.

  • Wrap the engine in a small Python service and measure end-to-end latency.

When you’re ready for scale, pair TensorRT with Triton Inference Server or your favorite orchestrator—and enjoy the speed.

Related posts