Running the GUI without Showing Windows Using a Virtual Display

Overview

Headless Execution without Showing the Window in Executing Scripts from the Command Line describes how to run Choreonoid without showing windows using the --no-window option. While this mode provides an easy way of headless execution, it has the limitation that rendering on the GUI, such as that of the scene view, is not performed.

On the other hand, there are cases where you want to run Choreonoid automatically with the GUI hidden, but still need the GUI itself to work. For example:

  • You want to obtain the rendering results of the scene view as images in an automated execution with scripts

  • You want to automate the checking or testing of a series of processes including the behavior of the GUI

In such cases, by using a “virtual display”, you can run Choreonoid in the same state as with a normal window system, without showing anything on the actual screen. The windows are created on the virtual display and the rendering is performed as usual, so all the functions of the GUI are available while nothing is shown on the user’s screen.

This section mainly explains how to use Xvfb, a popular implementation of the virtual display. Note that the contents of this section are intended for Linux.

What is Xvfb

Xvfb (X virtual framebuffer) is a virtual display server of the X Window System. It performs rendering on a framebuffer in memory, without using an actual display or graphics hardware.

On Ubuntu, it can be installed with the following command.

sudo apt install xvfb

Basic Usage

First, start Xvfb to create a virtual display.

Xvfb :99 -screen 0 1920x1080x24 &

“:99” is the display number, and any number can be used as long as it does not conflict with the numbers in use. “-screen 0 1920x1080x24” specifies the resolution and color depth of screen number 0.

Next, launch Choreonoid with this virtual display specified by the DISPLAY environment variable.

DISPLAY=:99 choreonoid project.cnoid script.py

Then all the windows of Choreonoid are created on the virtual display, and nothing is shown on the actual screen. The GUI is working as usual, and the rendering of the scene view is also performed.

However, in environments where the desktop is running as a Wayland session, this specification alone results in the windows being shown on the actual screen. In that case, launch Choreonoid as described in Usage in Wayland Sessions.

When you are done, terminate (kill) the Xvfb process you started.

Alternatively, the xvfb-run command automatically handles the startup and termination of Xvfb and the setting of DISPLAY.

xvfb-run -a -s "-screen 0 1920x1080x24" choreonoid project.cnoid script.py

“-a” is an option to automatically select an available display number. Since Xvfb is also terminated when the command finishes, this is more convenient for batch execution.

Usage in Wayland Sessions

In recent Linux distributions, the desktop session often runs on Wayland instead of X11. You can check the current session type with the following command.

echo $XDG_SESSION_TYPE

In environments where this shows “wayland”, Qt, which constitutes the GUI of Choreonoid, usually runs using the Wayland platform plugin. In this case, the DISPLAY environment variable is not used to determine where the windows are shown, so even if you specify a virtual display with DISPLAY, the windows are shown on the actual screen (the Wayland compositor).

Since Xvfb is a server of the X Window System, Qt must run on the X11 (xcb) platform to use it. This can be specified by the QT_QPA_PLATFORM environment variable. In a Wayland session, launch Choreonoid as follows.

env -u WAYLAND_DISPLAY QT_QPA_PLATFORM=xcb DISPLAY=:99 choreonoid project.cnoid script.py

Here, QT_QPA_PLATFORM=xcb fixes the Qt platform to X11 (xcb). In addition, the WAYLAND_DISPLAY environment variable is unset by env -u WAYLAND_DISPLAY so that no processing that assumes the use of Wayland is performed. With these specifications, the windows are created only on the virtual display.

The same specifications also apply when using xvfb-run.

env -u WAYLAND_DISPLAY QT_QPA_PLATFORM=xcb xvfb-run -a -s "-screen 0 1920x1080x24" choreonoid project.cnoid script.py

Note that in X11 sessions, Qt uses the xcb platform by default, so adding these specifications does not change the behavior. When the session type is unknown, or for scripts used in various environments, it is safer to always add these specifications.

Obtaining the Rendering Results of the Scene View

Since the rendering of the scene view is performed as usual on the virtual display, you can obtain the results as images.

One way is to save the image of the scene view from a Python script. The rendering contents of the scene view can be saved to an image file as follows.

from cnoid.Base import *

SceneView.instance.sceneWidget.saveImage("scene.png")

Note that a correct image may not be obtained in a state where the rendering has not been performed yet, such as immediately after loading a project, so execute this at a timing after the rendering, using a timer or the like if necessary.

Another way is to take a screenshot of the entire screen of the virtual display with an external tool. For example, use the import command of ImageMagick (which can be installed with sudo apt install imagemagick on Ubuntu) as follows.

DISPLAY=:99 import -window root screenshot.png

In this case, an image of the entire screen including the whole main window is obtained.

If the Pillow library of Python is installed in your environment, you can also take a similar screenshot as follows.

from PIL import ImageGrab

ImageGrab.grab(xdisplay=':99').save('screenshot.png')

Points to Note

  • The rendering on Xvfb does not use the GPU and is performed by software rendering (such as llvmpipe of Mesa). This is sufficient for checking the rendering contents, but the rendering speed is slower than with an actual GPU, so it is not suitable for measuring rendering performance.

  • If your only purpose is the simulation of vision sensors (GLVisionSimulator), you do not need to use a virtual display. As described in Headless Execution without Showing the Window, in an environment without a window system, the rendering using EGL is used, where the hardware acceleration by the GPU is also enabled. The virtual display is a means for the cases where the GUI itself needs to work.

  • For automation, the --test-mode option, which terminates Choreonoid when an error occurs, is also useful.

Other Methods

Besides Xvfb, the following virtual displays are also available and can be used depending on the purpose.

  • Xvnc (VNC server)

Xvnc provided by TigerVNC and others also works as a virtual display like Xvfb. With Xvnc, you can view the screen of the virtual display at any time from a VNC client, which is convenient when you usually run Choreonoid without showing windows but want to check the behavior from time to time.

  • xpra

xpra works as a virtual display in its “screenless mode”, and you can attach to (show) and detach from (hide) the screen of a running application afterwards. As with Xvnc, you can use it to check the screen only when necessary.

  • The offscreen platform of Qt

Since choreonoid is a Qt application, it can also be launched without a window system by setting the environment variable QT_QPA_PLATFORM=offscreen. In this case, however, QOpenGLWidget used for rendering the scene view is not supported, so the rendering of the scene view is not performed. In Choreonoid, this is equivalent to what is used internally when the --no-window mode is automatically applied in an environment without a window system, and it cannot be used for the purpose of this section (execution with GUI rendering).

If you want to perform rendering with the GPU even on a virtual display, there is a way to combine VirtualGL. The setup is somewhat advanced, but consider it when rendering speed is required.