Conversation, scenes and conversational agents

Components and ownership

ros_chatbot contains the chatbot orchestrator, scene manager, provider clients, priming service, vision/intent helpers, ElevenLabs conversational agent, browser audio bridge and optional Spark/n8n/realtime clients. Enabling the package does not mean all providers or integrations are configured or launched.

The main launch has separate enable flags for chatbot, scene manager, services, priming, tools HTTP server, ElevenLabs agent and optional integrations. Default provider availability still depends on selected world/agent configuration and credentials.

Chat input, suggestions and speech

The chatbot subscribes to /hr/interaction/chat, /hr/perception/hear/sentence and selected /hr/interaction/chatbot_response messages. It publishes candidate responses on /hr/interaction/chatbot_responses and speech requests on /hr/control/speech/say.

Hybrid/manual selection differs from automatic speaking. Operators can inspect suggestions and choose one; applications should correlate response IDs and avoid resubmitting an accepted suggestion as a new user message. The WebUI's teleops.chat.send and teleops.speech.say target different paths: one asks the conversation system for a reply, the other requests the supplied text be spoken.

The available-agent service uses hr_msgs/srv/StringArray; its name is relative to the launched node namespace. Inspect ros2 service list -t and the chatbot reference rather than guessing a fully qualified name.

Scenes and prompts

The scene manager reads configured agent IDs from the selected character/world's agents.yaml, maintains scene/prompt template configuration and exposes available_scenes. The returned catalog omits the automatic pseudo-scene. A missing or unreadable agent file logs a warning and produces an empty configured-agent list.

The priming service at /hr/interaction/prompts/response_prompt uses std_srvs/srv/Trigger. Response-style and length primer nodes support current selection and configured options/probabilities. Triggering can update shared prompt state; it is not a deterministic text-generation endpoint.

Runtime prompt settings can update conversational session context. Changing a displayed prompt selection, persisting its file, and restarting an external conversation session are distinct operations.

Intent and view-description services

The helper node provides relative get_intent (hr_msgs/srv/GetIntent) and describe_view (hr_msgs/srv/StringTrigger) services. Intent detection uses the configured NLP detector and returns intent/confidence where available; exceptions are logged.

View description consumes the latest compressed camera frame from CHATBOT_SERVICES_IMAGE_TOPIC, defaulting to the Jetson RealSense compressed color topic. It requires OPENAI_API_KEY; OPENAI_VISION_MODEL defaults in source to gpt-4o. The service can write image-feed artifacts beneath IMAGE_FEED_HOME and returns JSON content/image/time metadata in its message. Missing provider setup or request failure is reported through its response/logs. Do not confuse this OpenAI path with the separate Groq answerFromVision tool.

ElevenLabs conversation

The agent publishes status/events under /hr/interaction/elevenlabs_agent and exposes send_user_message, contextual_update and restart, each using hr_msgs/srv/StringTrigger. Requests can alter an active cloud conversation. Read the exact request/response schema before direct calls.

Streaming output is published separately for audio2lips (/hr/control/stream_voice), internal browser playback and operator monitoring. Interrupt/control channels clear queued audio. Browser playback completion participates in session/speech timing; monitor completion is deliberately isolated from that gate.

Check credentials, configured agent ID, session status and actual provider connection independently. A stale status display is not proof the cloud session remains live. Restarting a session can discard conversational continuity; do not retry it automatically after every UI reconnect.

Tool and optional service integration

The LLM tools bridge exposes expression, arm gesture and vision actions. Tool enablement/schema exposure is separate from the HTTP executor. Provider tool calls can move hardware or send camera data to an external API.

Spark integration is disabled by default in current launch/configuration unless explicitly enabled. Its ROS client exposes agent chat/feedback and response topics; a separate image supplies its HTTP service. n8n and realtime clients are also optional. Only document/enable their actual selected deployment endpoints; an installed source file does not imply a reachable service.

Diagnostics and data handling

Follow input message, selected world/agents, provider call, suggestion, selected reply, TTS request, audio and animation in sequence. Inspect metadata patches separately from duplicate input. Conversation databases and recordings may contain personal content; keep support bundles scoped and redact credentials before sharing.

Evidence: head-ros-nodes/ros/ros_chatbot/scripts/{chat.py,scene_manager.py,priming.py,services.py,11_labs_agent.py,spark_client.py}, package source/config nodes, and hr_ros2 launch files. Provider contracts here describe the inspected adapter, not current external-provider product guarantees.