Sophia Unity runtime

Role, source and build

sophia_unity is a current platform component. Its master branch builds the Sophia WebGL scene, consuming animation packets from the ROS WebSocket streamer. It renders the virtual character; loading its assets does not establish that animation streaming or robot actuation is available.

The inspected project pins Unity 6000.3.22f1. SophiaUnity.Editor.SophiaWebGLBuild.Build builds only Assets/Scenes/sophia.unity, defaults to Builds/WebGL, and accepts UNITY_BUILD_PATH. CI runs on master pushes or manual dispatch, checks out LFS assets, builds WebGL, and publishes an immutable asset image. Consult the current workflow for artifact tags and the stack manifest for the activated digest.

The WebUI exposes installed assets only when the activation environment and directory are valid. /api/virtual-project returns the chosen loader/assets; clients should use that response rather than construct filenames. Serve .wasm as application/wasm and compressed files with their actual content encoding. A wrong MIME/encoding combination can break the player before any socket connection occurs.

Animation connection

The streamer listens on 0.0.0.0:9099 by default and subscribes to PAU data. Its default rate is 50 Hz, with --fps, --port, --topic, --config, --projection_matrix, and --world2camera_matrix controls. It translates robot animation into the Unity packet format and applies configured limits/mappings.

For a WebGL page on a nonloopback origin, ConnectionUI derives ws[s]://<page-authority>/unity_wss. For loopback pages it selects ws://localhost:9099. Outside that WebGL resolution path it uses the configured queue address. A remote proxy must upgrade /unity_wss to the stream listener; serving static assets alone is insufficient.

The current documentation host intentionally serves only static documentation. It is not a Unity animation proxy.

Packet contract

Packets are JSON objects with at least one recognized animation channel: joints, blendshapes, head_rotation, neck_rotation, eyes, or jaw. The model also accepts timestamp and optional camera matrices.

Field	Structure / handling
`head_rotation`, `neck_rotation`	Objects with float `x`, `y`, `z`; downstream accessors reorder axes
`eyes`	`leftPitch`, `leftYaw`, `rightPitch`, `rightYaw`
`joints`	Named body/arm/finger float channels
`blendshapes`	Named facial float channels
`jaw`	Float passed to head animator
`timestamp`	Float used for interpolation interval
`projection_matrix`, `world2camera_matrix`	Optional matrix model fields

AnimationPacket.TryParse rejects empty input, payloads longer than 1,048,576 UTF-16 code units, nonobject text, packets with no recognized animation channel, invalid JSON, and nonfinite public float values. Partial packets overwrite a copy of the previous model; missing major sections are initialized. A partial update therefore retains previous values rather than resetting every other channel.

Minimal visual-only injection example:

{"head_rotation":{"x":0,"y":0,"z":0},"jaw":0,"timestamp":1}

This is a structural example, not a claim about calibrated joint units or desired robot pose. Axis/unit conversions belong to ws_streamer mappings and the animator implementation. VectorModel.GetYawPitchRoll() returns (x,z,y) and GetRollPitchYaw() returns (z,x,y); eye accessors reorder yaw/pitch. Do not interpret packet axes as a direct physical robot coordinate system.

Queueing, reconnect and interpolation

PackageQueue validates ws/wss addresses, uses connection generations to discard obsolete connection attempts, and resets prior packets/interpolation state on disconnect. Each render iteration consumes at most 64 messages, merges partial updates, and uses the latest resulting packet. Intermediate frames can be dropped intentionally to keep rendering responsive.

Timestamp deltas above zero and no greater than 0.25 seconds inform the interpolation interval, clamped to 1/120–0.1 seconds; other deltas fall back to 1/30 second. The queue reports packet receive rate. Automatic reconnect is scheduled after one second by ConnectionUI when enabled. Choosing Disconnect disables automatic reconnection.

Browser/HTML integration methods

Methods are Unity component entry points, not HTTP routes. Host pages may invoke them using Unity's component-message mechanism when the scene contains the intended object.

Component method	Effect
`ConnectionUI.SetUIvisibility(int)`	Set overlay visibility to `value == 1`
`ConnectionUI.SetUIvisible(int)`	Toggle overlay; argument is not used as an absolute state
`ConnectionUI.SetVerboseMessage(int)`	Enable verbose logging for value 1
`ConnectionUI.SetKeyInputAllowed(int)`	Set WebGL keyboard capture
`ConnectionUI.SetWSurn(string)`	Set address; prepend `ws://` if string does not start with `ws`
`ConnectionUI.ForceConnectWS(int)`	Enable auto-connect and connect
`ConnectionUI.InjectJSONframe(string)`	Parse and apply an immediate frame through the queue
`ConnectionUI.SetRangesBlendshapes(int)`	Set head blendshape range
`ConnectionUI.SetRangesJaw(int)`	Set jaw range
`AnimationDriver.SetMatrixUpdateAllowed(int)`	Set projection-update flag
`AnimationDriver.SetW2cMatrixUpdateAllowed(int)`	Set world-to-camera-update flag

The inspected AnimationDriver has camera-matrix application code commented out. Setting its flags does not establish that camera matrices are applied. The optional OnAppReady browser callback is guarded so its absence does not abort startup.

UI and camera controls

Press H to toggle the connection overlay when keyboard input reaches Unity.
The overlay shows server address, connection state and receive rate; it offers reconnect behavior and explicit connection/disconnection.
Camera scripts include head-only/full-body view presets and B for background switching when those scene components are active.
Movement uses configured Unity input axes (Fire1, Vertical, Horizontal); actual bindings follow project Input settings.
WebGL starts with global keyboard capture disabled, allowing surrounding operator controls to work.

Diagnosis and acceptance

Check asset fetches and WebGL startup first, /unity_wss handshake second, packet rate third, and channel/mapping/rig behavior last. A connected socket with zero packets points upstream; packets with no motion point to parser/channel/rig issues. If the player repeatedly reconnects, inspect proxy upgrade settings, origin scheme and the stream process.

Inspect CI/build output and scene validators before publishing new character assets. Validate face, head, eye, jaw, arm and finger channels against the selected rig, then compare behavior with the intended physical profile. These visual/hardware checks are not established by source inspection.

Evidence: sophia_unity/ProjectSettings/ProjectVersion.txt, .github/workflows/webgl-deploy.yml, Assets/Editor/SophiaWebGLBuild.cs, Assets/AnimReplicator/{ConnectionUI.cs,AnimationDriver.cs,Util/PackageQueue.cs,Models/*}, camera scripts, and head-ros-nodes/ros/ros_misc/nodes/ws_streamer.py.