ROS speech orchestration and audio2lips

Synthesis versus robot speech

The HTTP TTS server returns audio and timing metadata. ros_tts/scripts/tts_talker.py owns robot speech jobs, queueing, voice selection, playback routing, events and synchronized animation. audio_lips.py handles streamed PCM-to-viseme inference where provider phonemes are not already available.

The main stack independently enables the talker and audio-lips nodes. A live synthesis server does not establish that either ROS consumer is running.

Speech input and output

The main speech request topic is /hr/control/speech/say using hr_msgs/msg/TTS. Fields include text, lang, request_id, agent_id, audio_path and voice. Use request IDs to correlate/cancel work where supported. The current selected voice mapping and provider configuration determine actual synthesis.

Robot speech completion publishes robot_said. Speech status/events coordinate ASR suppression and behavior. TTS control topics manage interruption and queue state. The ROS TTS reference lists exact topic/service registrations and configuration declarations.

Remote ROS service surface

Names below are under /hr/control/speech/ in the normal speech namespace:

Service	Type / behavior
length	hr_msgs/srv/TTSLength; txt, lang → float length
data	hr_msgs/srv/TTSData; txt, lang → JSON string with text/word timing nodes
interrupt	hr_msgs/srv/StringTrigger; data hard/soft → success and remaining text/message
wait_for_tts	std_srvs/srv/Trigger; waits for queued work
request_queue_size	hr_msgs/srv/GetInteger; unfinished task count
clear_tts_queue	std_srvs/srv/Trigger; cancels/clears queued requests

Length/data are not purely local calculations: they can call the configured synthesis provider. Unknown language or synthesis failure yields a fallback length of 1 or empty data respectively, with logs. Do not interpret every one-second estimate as a verified utterance duration.

wait_for_tts waits for queue completion rather than providing a bounded immediate snapshot. Apply a client timeout appropriate to the application, but remember that a timeout does not stop speech. Queue size includes unfinished tasks rather than only jobs waiting behind the current one.

Interrupt accepts only hard/soft; if nothing is running it returns unsuccessful with a message. The implementation waits up to ten seconds for the executor and reports remaining text. It is not a whole-robot stop.

Control messages and queue semantics

The control handler supports shutup, ready, interrupt, resume, clear, and request-specific abort/cancel forms. Hard shutup stops at the nearest next word according to executor behavior; soft interrupt targets punctuation. Resume is a placeholder/logging path in this handler, so clients must not assume a fully paused speech job can resume.

abort:<request_id> targets current or queued work depending on the ID; cancel:<request_id> is for noncurrent requests. Inspect status and queue after cancellation. Clearing cancels pending jobs and has different scope from reading queue size.

Output routing

audio_output normalizes to local, browser or both; invalid values fall back to local. Cached speech always supplies a WebUI monitor copy, but internal browser playback is sent only for browser/both output. The monitor and internal paths maintain separate completion state.

Provider phonemes already queue synchronized visemes. The browser copy is deliberately not fed back into audio2lips in that case, avoiding duplicate/canceling lip animation. ElevenLabs streaming separately feeds /hr/control/stream_voice for audio2lips.

Audio2lips

Default input is AudioStream on /hr/control/stream_voice; default status is /hr/control/voice_streaming; output visemes go to /hr/animation/stream_visemes; command input is /hr/control/audio2lips/commands. Interrupts also cancel queued visemes through /hr/animation/cancel_visemes.

The node tracks stream IDs, format, pending audio and finish state. Defaults include idle finish timeout 0.25 seconds, idle checks every 0.05 seconds and a playback finish tail of 0.5 seconds. These are configurable values, not a guarantee of end-to-end latency.

AUDIO2LIPS_NUM_THREADS sets relevant numeric-library thread environment; otherwise the node defaults those variables to four where not already set. Model/inference configuration belongs to /hr/control/audio2lips and the installed engine. Check actual CPU load and realtime behavior when tuning.

Diagnosis

Trace speech request → queue → provider response → output routing → playback/events → visemes/animation. If duration/data succeed but the robot is silent, inspect output routing. If speech is delayed, inspect queue and provider latency. If streamed audio plays but lips stop, inspect stream IDs, interrupt/cancel events, inference engine and animation sink. Avoid feeding the same audio through both provider-phoneme and inference paths.

Evidence: head-ros-nodes/ros/ros_tts/scripts/{tts_talker.py,audio_lips.py}, src/tts, libs/audio2lips, and hr_msgs speech definitions. No live synthesis/playback was executed for this guide.