Deploy YOLOv11 on Jetson Orin Nano: ROS 2 C++ Guide

The Edge AI Challenge in Modern Robotics

In autonomous mobile robotics (AMRs), latency is the difference between seamless navigation and a catastrophic collision. While modern object detection models like Ultralytics' YOLOv11 offer unprecedented accuracy, running them in real-time on edge hardware like the NVIDIA Jetson Orin Nano presents a massive engineering hurdle.

Many developers default to Python-based ROS 2 nodes for rapid prototyping. However, Python's Global Interpreter Lock (GIL), high memory copy overhead, and sub-optimal serialization pipelines severely bottleneck performance. To extract every drop of the Jetson Orin Nano’s 40 TOPS of AI processing power, we must build a custom, zero-copy ROS 2 C++ node powered by TensorRT FP16 acceleration.

This guide walks you through the entire deployment pipeline, from model optimization to writing a high-throughput ROS 2 Humble/Iron C++ node for real-time obstacle avoidance.

Phase 1: Optimizing YOLOv11 with TensorRT

Before writing C++ code, we must compile the PyTorch YOLOv11 model into an optimized TensorRT engine. TensorRT performs layer fusion, kernel tuning, and half-precision (FP16) quantization tailored specifically to the Ampere GPU architecture of the Orin Nano.

Step 1: Export YOLOv11 to ONNX

First, export the pre-trained or custom-trained YOLOv11 model to ONNX format. Ensure you specify dynamic dimensions if your input resolution varies, though fixed dimensions yield the highest optimization.

# Install Ultralytics package
pip3 install ultralytics

# Export to ONNX with static 640x640 input resolution
yolo export model=yolov11n.pt format=onnx imgsz=640 half=True opset=17

Step 2: Compile to TensorRT Engine via trtexec

Using NVIDIA’s trtexec tool, we compile the ONNX graph into a serialized TensorRT engine (.engine). This step must be run directly on the Jetson Orin Nano to ensure optimizations match its hardware profile.

/usr/src/tensorrt/bin/trtexec \
  --onnx=yolov11n.onnx \
  --saveEngine=yolov11n.engine \
  --fp16 \
  --detailedProfiler \
  --workspace=3072

Phase 2: High-Throughput ROS 2 Node Architecture

To process high-framerate camera feeds (e.g., 60 FPS from an Intel RealSense or GMSL2 camera) without dropping frames, our ROS 2 C++ node must adopt the following architectural patterns:

• Intra-Process Communication: Eliminates serialization overhead when the camera driver and detection node run in the same container/process.
• CUDA Host-Pinned Memory: Prevents page-locked memory overhead during CPU-to-GPU tensor transfers.
• Asynchronous Inference Queues: Decouples frame ingestion from model execution using CUDA streams.

The diagram below illustrates the high-throughput data pipeline:

[ Camera Node ] 
       │ (Pointer Sharing via std::shared_ptr<Image>)
       ▼
[ ROS 2 Subscription Callback ]
       │
       ▼ (CUDA MemcpyAsync: Host-Pinned -> Device)
[ Pre-processing CUDA Kernel ]
       │
       ▼ (TensorRT Execution Context)
[ GPU Inference (FP16 Engine) ]
       │
       ▼ (CUDA MemcpyAsync: Device -> Host)
[ Post-processing & NMS (Non-Maximum Suppression) ]
       │
       ▼ (Publish Obstacle Avoidance Command)
[ LaserScan / Detection2DArray ]

Phase 3: Writing the ROS 2 C++ Node

Let's write a highly optimized C++ node. This implementation uses the TensorRT C++ API directly to run inference on incoming image messages.

yolo_obstacle_detector.cpp

#include <rclcpp/rclcpp.hpp>
#include <sensor_msgs/msg/image.hpp>
#include <geometry_msgs/msg/twist.hpp>
#include <cv_bridge/cv_bridge.h>
#include <opencv2/opencv.hpp>
#include <NvInfer.h>
#include <cuda_runtime_api.h>

#include <memory>
#include <vector>
#include <fstream>

class Logger : public nvinfer1::ILogger {
    void log(Severity severity, const char* msg) noexcept override {
        if (severity <= Severity::kWARNING) {
            std::cout << "[TensorRT] " << msg << std::endl;
        }
    }
} gLogger;

class YoloOrinNode : public rclcpp::Node {
public:
    YoloOrinNode() : Node("yolo_orin_node") {
        // Declare parameters
        this->declare_parameter<std::string>("engine_path", "yolov11n.engine");
        this->declare_parameter<float>("conf_threshold", 0.45f);

        std::string engine_path = this->get_parameter("engine_path").as_string();
        conf_threshold_ = this->get_parameter("conf_threshold").as_double();

        // Initialize TensorRT
        init_tensorrt(engine_path);

        // Publishers & Subscribers
        subscription_ = this->create_subscription<sensor_msgs::msg::Image>(
            "/camera/image_raw", 10, std::bind(&YoloOrinNode::image_callback, this, std::placeholders::_1));
        
        cmd_vel_pub_ = this->create_publisher<geometry_msgs::msg::Twist>("/cmd_vel", 10);

        RCLCPP_INFO(this->get_logger(), "YOLOv11 TensorRT Node Initialized successfully.");
    }

    ~YoloOrinNode() {
        cudaStreamDestroy(stream_);
        cudaFree(device_buffers_[0]);
        cudaFree(device_buffers_[1]);
    }

private:
    void init_tensorrt(const std::string& engine_path) {
        std::ifstream file(engine_path, std::ios::binary);
        if (!file.good()) throw std::runtime_error("Failed to read engine file.");

        file.seekg(0, std::ios::end);
        size_t size = file.tellg();
        file.seekg(0, std::ios::beg);

        std::vector<char> engine_data(size);
        file.read(engine_data.data(), size);

        runtime_ = nvinfer1::createInferRuntime(gLogger);
        engine_ = runtime_->deserializeCudaEngine(engine_data.data(), size);
        context_ = engine_->createExecutionContext();

        // Setup CUDA Stream and Allocations
        cudaStreamCreate(&stream_);
        
        // YOLOv11 static sizes (e.g., 1x3x640x640 input, output coordinates + classes)
        cudaMalloc(&device_buffers_[0], 1 * 3 * 640 * 640 * sizeof(float)); // Input
        cudaMalloc(&device_buffers_[1], 1 * 8400 * 84 * sizeof(float));     // Output (example dimensions)
    }

    void image_callback(const sensor_msgs::msg::Image::SharedPtr msg) {
        auto start_time = this->now();

        // Convert ROS Image to OpenCV Mat
        cv_bridge::CvImagePtr cv_ptr = cv_bridge::toCvCopy(msg, sensor_msgs::image_encodings::BGR8);
        cv::Mat frame = cv_ptr->image;

        // Preprocess: Resize and Normalize
        cv::Mat resized;
        cv::resize(frame, resized, cv::Size(640, 640));
        resized.convertTo(resized, CV_32FC3, 1.0 / 255.0);

        // HWC to CHW conversion
        std::vector<float> input_data(1 * 3 * 640 * 640);
        std::vector<cv::Mat> channels(3);
        for (int i = 0; i < 3; ++i) {
            channels[i] = cv::Mat(640, 640, CV_32FC1, &input_data[i * 640 * 640]);
        }
        cv::split(resized, channels);

        // Asynchronous Copy to Device
        cudaMemcpyAsync(device_buffers_[0], input_data.data(), input_data.size() * sizeof(float), cudaMemcpyHostToDevice, stream_);

        // Run Inference
        context_->enqueueV2(device_buffers_, stream_, nullptr);

        // Asynchronous Copy Back to Host
        std::vector<float> output_data(1 * 8400 * 84);
        cudaMemcpyAsync(output_data.data(), device_buffers_[1], output_data.size() * sizeof(float), cudaMemcpyDeviceToHost, stream_);
        cudaStreamSynchronize(stream_);

        // Process Detections and trigger Obstacle Avoidance
        process_detections(output_data, frame.cols, frame.rows);

        auto duration = (this->now() - start_time).seconds();
        RCLCPP_DEBUG(this->get_logger(), "Inference pipeline took: %.4f seconds", duration);
    }

    void process_detections(const std::vector<float>& output, int img_w, int img_h) {
        bool obstacle_detected = false;

        // Parsing logic for YOLOv11 output (Simplified bounding box heuristic for demo)
        // We inspect the central 30% of the image field of view (FOV) for safety critical zones
        for (int i = 0; i < 100; ++i) { // Inspecting top confidence scores
            float score = output[i * 84 + 4];
            if (score > conf_threshold_) {
                float cx = output[i * 84 + 0] * img_w / 640.0f;
                float cy = output[i * 84 + 1] * img_h / 640.0f;
                float w  = output[i * 84 + 2] * img_w / 640.0f;
                float h  = output[i * 84 + 3] * img_h / 640.0f;

                // Check if bounding box falls within the critical path (center strip)
                if (cx > (img_w * 0.35) && cx < (img_w * 0.65)) {
                    obstacle_detected = true;
                    break;
                }
            }
        }

        // Publish reactive navigation commands
        auto cmd_msg = geometry_msgs::msg::Twist();
        if (obstacle_detected) {
            RCLCPP_WARN(this->get_logger(), "Obstacle Detected! Halting and turning...");
            cmd_msg.linear.x = 0.0;   // Stop forward motion
            cmd_msg.angular.z = 0.5;  // Rotate in place
        } else {
            cmd_msg.linear.x = 0.4;   // Safe to proceed
            cmd_msg.angular.z = 0.0;
        }
        cmd_vel_pub_->publish(cmd_msg);
    }

    rclcpp::Subscription<sensor_msgs::msg::Image>::SharedPtr subscription_;
    rclcpp::Publisher<geometry_msgs::msg::Twist>::SharedPtr cmd_vel_pub_;
    
    nvinfer1::IRuntime* runtime_;
    nvinfer1::ICudaEngine* engine_;
    nvinfer1::IExecutionContext* context_;
    cudaStream_t stream_;
    void* device_buffers_[2];
    float conf_threshold_;
};

int main(int argc, char** argv) {
    rclcpp::init(argc, argv);
    rclcpp::spin(std::make_shared<YoloOrinNode>());
    rclcpp::shutdown();
    return 0;
}

Phase 4: Real-World Performance Benchmarks

Deploying YOLOv11 on the Jetson Orin Nano highlights the massive performance delta between various compilation states and programming environments. Below is a real-world throughput analysis at a standard 640x640 input resolution:

By migrating from PyTorch/Python to a compiled TensorRT FP16 pipeline in C++, we achieved an outstanding 8x increase in overall frame processing capacity, freeing up valuable CPU cycles on the Orin Nano for simultaneous localization and mapping (SLAM) and path planning.

Frequently Asked Questions (FAQ)

1. Can I use INT8 precision on the Jetson Orin Nano?

Yes, but INT8 requires a calibration dataset (representative images of your operational environment) to generate a dynamic range scale factor. FP16 is highly recommended for quick deployments as it provides near-lossless accuracy with substantial speedups without calibration overhead.

2. How do I resolve CUDA out-of-memory errors during startup?

Ensure that you compile the model with a defined workspace ceiling (e.g., 3GB) and configure your Jetson's swap space. The Orin Nano shares memory between the CPU and GPU; allocating too much memory to either will result in allocation failures.

3. How do we pass images between camera nodes and YOLO nodes without copying?

Use ROS 2 Intra-Process Communication (IPC). By running both the camera driver node and the YoloOrinNode in a single Component Container and publishing messages as std::unique_ptr, ROS 2 passes raw pointers directly, resulting in zero-copy memory operations.

Supercharge Your Edge AI Deployments with NikhilWorks

Building high-performance edge AI pipelines requires deep expertise across the entire stack—from hardware-level CUDA memory management to robust ROS 2 software architectures. At NikhilWorks, we help robotics companies, industrial automation firms, and AI startups build mission-critical, real-time computer vision systems.

Are you looking to optimize your model deployment, write robust ROS 2 production infrastructure, or design custom edge AI systems? Contact NikhilWorks today to schedule an engineering consultation and elevate your robotics platform to the next level.