Stereo camera left view unavailable during teleoperation with high latency

Users VR Tele-operation
1

Hi everyone,

We are currently setting up teleoperation with Reachy 2. Everything seems to be connected correctly, and the teleoperation app starts successfully.

However, when I started teleoperation, only the right view of the stereo camera showed an image, while the left view was unavailable. There was also significant latency.

Our GPU is an NVIDIA GTX 1080 Ti, so we are wondering whether the issue could be related to GPU performance.

Has anyone experienced this before, or does anyone have suggestions for debugging it?

2

HI @Khal1L
Sorry to hear that you are having issues with your Reachy2.
Could you tell us the serial number of your Reachy? (r2-00XX)

  1. Could you check on the dashboard (http://r2-00XX.local:8000/ on your browser), Media section, that you’ve got both cameras when you press connect on Teleoperation Cameras ?
  2. Is your computer plugged when you are using teleoperation ?
  3. Are you on wifi or ethernet connection to run teleoperation ? Could you tell us the ping interval (on the mirror scene, next to the IP address on the info section) : How to Use the Reachy 2 Teleoperation App)
  4. Are you using the built release app or the code from Unity ? Which version is it (written under the Connect button at the app start) ?
  5. Which headset are you using ?

Thanks and have a good day,

Claire from Pollen Team

3

Hi Claire,

Thanks for your reply. Here are the details:

  1. Serial number
    Reachy serial number is r2-0008.

  2. Dashboard / cameras
    In the dashboard Media section, after pressing Connect on Head Camera, we can see both stereo images, but the text still says “not available.”
    The Torso page completely freezes, so we cannot press Connect there.

  3. Computer plugged in / network / ping
    We tried connecting through Ethernet, but it does not work. The dashboard shows Ethernet: no active connection, and the Ethernet IP 192.168.137.162 does not respond to ping.
    So we are currently using Wi-Fi with IP 192.168.10.172. The ping in the mirror scene is around 41 ms.

  4. Teleoperation app
    We are using the built release app, version 1.1.2.

  5. Headset
    We are using a Meta Quest 3S.

Also, the robot system keeps shutting down unexpectedly.

I’ve attached the logs here for reference.

4

In the logs, we see errors such as:

[ERROR] [hal-5]: process has died
Emergency shutdown initiated in zuuu_hal
Critical node failed : [ethercat_master_server]
Critical node failed : [dynamic_state_router]
Critical node failed : [reachy_grpc_video_sdk_server]

I have sent the full logs to support@pollen-robotics.com

5

Hi @Khal1L,

Ok that means that the teleop camera is working well (the “not available” is a known issue of the front of the dashboard, it’s not an issue of your robot).

The latency is probably caused by your wifi network, which seems to be slow. I can’t figure out why you’re not getting an Ethernet connection. Have you tried using a different Ethernet cable and a different Ethernet port on the Bedrock? Can you try connecting the robot and your computer to a standalone router (even one without an internet connection)?

On this release app version and on our Quest3S, we have the stereo vision. Could you test on another app on your Quest that you’ve got the stereo vision, to see if maybe it’s an issue of settings ?

About the logs, I guess you copy/paste from the dashboard ? Could you maybe use the ssh connexion (where the logs are clearer for us) :

  1. on a terminal, execute ssh bedrock@r2-0008.local (password: root)
  2. execute journalctl -fu reachy2-core
  3. restart the core service from the dashboard (Section services)
  4. copy/paste all the logs of the journalctl until the errors

Thanks a lot !

Claire

6

Hi Claire,

Quick update: after switching the Ethernet cable, the Ethernet connection now works.

With Wi-Fi off, r2-0008.local resolves to 192.168.137.14, ping works with 0% packet loss and <0.1 ms latency, and the dashboard is reachable at:

http://192.168.137.14:8000/

I can also enter teleoperation and control the robot normally over Ethernet now.

However, the original vision issue is still there, same as with wireless: in the Reachy app, I only get the right-eye view, and the left-eye view is black/not displayed. The camera image also jitters/shakes during teleoperation.

I also tested the Quest display outside the Reachy app. In the Quest home environment/ Reachy app mirror scene, I can see normally with both eyes, so the Quest display itself seems to be working. The issue seems to happen only inside the Reachy app during teleoperation.

So it looks like the problem is probably not caused by Wi-Fi/Ethernet connectivity or the Quest display hardware. Could you let me know what we should check next for the stereo camera stream or Reachy app rendering?

7

I couldn’t attach the log file here, so I sent the full reachy2-core log by email to support@pollen-robotics.com.

The log was recorded while entering teleoperation and reproducing the issue. It captured both the right-eye-only/jittering camera issue and a disconnection during teleop.

From what I saw, the camera seems to initialize correctly, but during teleop there are repeated [hal-4] [ERROR]: self.x_vel_goal : 0.0 messages, followed by a reachy_grpc_joint_sdk_server traceback, and then the core container stops/restarts around the disconnection.

8

Hi @Khal1L,

That’s good news for the Ethernet connection.

To be sure there is no issue with the settings of your Quest 3S, could you test on another app where there is stereo vision, and not just the Quest home environment ?

Also, it can be an issue of Gstreamer installation : did you use the installer, and did GStreamer install correctly when you installed the app? You can try to reinstall the app to be sure.

Otherwise you can download it directly : https://gstreamer.freedesktop.org/download/#windows, then check that the environment variable PATH contains C:\gstreamer\1.0\msvc_x86_64\bin (default installation) For that, look for “Edit the system environment variables” in the Windows search bar. Then, click on Environment variables. A new window shows up : double click on “Path” in the user variables. If you don’t see the gstreamer variable, select “New” and add the pathway above. setting env variables

Reboot after installation.

About the logs, I’m sorry I don’t see the mail with the new logs; I have the last one but where we can’t see the start of the error unfortunately. (the log about self.x_vel_goal is not a concern)