Skip to content

Jetson perception

Pipeline and current platform

Perception runs separately from the main ROS control image. The current head-perception/main source targets JetPack 6.2 / L4T R36.4.3 on arm64, with ROS 2 Humble in the container. The GPU image uses an L4T JetPack base; CPU builds are a separate path. TensorRT engines are platform artifacts and must not be assumed portable across JetPack, CUDA, TensorRT or device variants.

The r2_perception package installs four executables: yolo, yolo_pose, fusion, and face_embedding. perception_launch.py orchestrates these and optionally the RealSense driver.

flowchart LR
    Camera[RealSense color / depth / camera info] --> Pose[YOLO pose tracking]
    Camera --> Object[Optional YOLO objects]
    Pose --> Fusion[Depth fusion]
    Object --> Fusion
    Camera --> Fusion
    Pose --> Face[Face quality / embedding]
    Camera --> Face
    Face --> Identity[WebUI identity matching]
    Identity --> Fusion
    Fusion --> Output[People2 / Objects2 / UI image]

Launch defaults and their consequences

Argument Default Meaning
use_realsense true Start camera driver
qos_reliability best_effort Sensor/detection reliability
enable_object_detection false Object node is optional and off by default
yolo_model yolov8s-worldv2.pt Object model selected by launch
yolo_process_rate, pose_process_rate 0.0 Process every available frame
enable_face_recognition true Start quality-gated embedding producer
face_process_rate 5.0 Hz Face-processing rate
face_model buffalo_m InsightFace model pack
face_min_eye_distance_px 28 Pose eye separation gate
face_min_blur_laplacian_var 80.0 Sharpness gate
camera_frame_id realsense Fused output frame
debug_overlay_enabled true UI calibration/diagnostic overlay
debug_overlay_stale_sec 1.5 Overlay state staleness threshold

Object detection's node default differs from launch: launch sets confidence 0.15 and a broader world-class list; pose uses yolov8n-pose.pt with confidence 0.5. Read the effective launch configuration rather than taking constructor defaults as deployment defaults. The complete parameter and registration reference lists source declarations.

ROS inputs and outputs

Default camera inputs are:

  • /hr/perception/jetson/realsense/camera/color/image_raw
  • /hr/perception/jetson/realsense/camera/aligned_depth_to_color/image_raw
  • /hr/perception/jetson/realsense/camera/color/camera_info

Fusion publishes /hr/perception/people (hr_msgs/msg/People2), /hr/perception/objects (hr_msgs/msg/Objects2) and /hr/perception/ui_image (sensor_msgs/msg/Image). It combines pose/object results with synchronized image/depth/camera information. Camera calibration and correct aligned depth are necessary for meaningful spatial targets; a visible 2D detection alone does not prove a valid 3D location.

Fusion's UI image QoS is reliable, while sensor/detection paths default to best effort. Remote subscribers must use compatible QoS. Identity feedback arrives from /hr/perception/face_identities and is cached with expiry; stale tracker identities are not permanent person records.

Read-only graph inspection from a correctly configured ROS 2 environment:

ros2 node list
ros2 topic list -t
ros2 topic info -v /hr/perception/people
ros2 topic echo /hr/perception/people --once --qos-reliability best_effort
ros2 interface show hr_msgs/msg/People2
ros2 service list -t

Use the installed interface definition for the target runtime. Both repositories contain hr_msgs; compare compatible definitions rather than assuming that any package with that name is interchangeable.

Face embeddings and remote extraction

The embedding producer performs quality gating, schedules retries and publishes candidates/status. The WebUI stores person records and images, matches embeddings and sends identity feedback. An image successfully saved in the WebUI can still fail embedding extraction; inspect the separate embedded result.

The extraction service is /hr/perception/extract_face_embedding using hr_msgs/srv/ExtractFaceEmbedding. Requests include a request ID, base64 image and JSON options; responses include ok, error and JSON result. Consult the exact service definition and the faces guide before building an uploader.

Launch retry defaults include one second for unrecognized retry and identity-response timeout, and three seconds of tracker-state retention. Constructor quality gates include minimum crop dimensions, brightness range, blur and detector score; these are detailed in the declaration reference. Rejected crops should be diagnosed from status/reason rather than forcing repeated recognition calls.

Maintenance and diagnosis

Check USB/device rules, NVIDIA runtime, L4T compatibility and camera frames before model inference. Then inspect pose/detection outputs, depth alignment, fused messages and DDS reachability in that order. If the UI overlay works locally but the robot receives no people, check both publishers/subscribers and interface/QoS compatibility.

Keep downloaded models and generated engines with their platform/version metadata. Use repository smoke-test scripts for the appropriate build, and run GPU/RealSense validation on a matching Jetson. Source inspection and cross-build success do not validate camera behavior or GPU inference.

Evidence: head-perception/README.md, SUPPORTED_PLATFORMS.md, ros2_ws/src/r2_perception/{setup.py,launch/perception_launch.py,src/r2_perception/*.py}.