A first User Interface (UI) for your robot application

Section

🖥️ User interfaces - tutorials

🏁 Goal of this tutorial

By the end of this tutorial, you will know how to create a user interface for your robot application, how to display it on the robot’s display, and how to connect the user interface to the robot’s main control script.

Pre-requisites

Note

As of PAL OS edge, this tutorial requires some familiarity with the command-line, and basic experience with running Docker.

• You need to have your Docker developer environment set up and running. You can also use the PAL public Docker image.

• This tutorial uses Automatic code generation with rpk. You might want to check this page first, even though we will cover rpk basic usage here as well.

What are we building?

Creating a UI with a rpk template

Step 1: generating the task skeleton

1. start your Docker container:

PAL Developer Docker imagePAL public Docker image

xhost + # this allows your Docker container to access your host X server
mkdir -p ~/exchange # this is a shared folder between your host and the container

docker run -it --rm --name pal_docker \
           --device /dev/video0:/dev/video0 \
           -e DISPLAY=$DISPLAY \
           -v /tmp/.X11-unix:/tmp/.X11-unix \
           -u user:user \
           -v ~/exchange:/home/user/exchange \
           <robot-type>-<serial>-dev:alum-<release> bash

Replace <robot-type> and <serial> with the type and serial number of your robot(eg tiago-123), and <release> with the PAL OS version you are using (e.g., 25.01).

Once inside the container, source your ROS 2 environment:

source /opt/pal/alum/setup.bash

2. go to your exchange folder and create a new workspace:

cd ~/exchange
mkdir ws
cd ws

3. run rpk to create the task with a sample GUI:

$ rpk create -p src/ task
ID of your application? (must be a valid ROS identifier without spaces or hyphens. eg 'robot_receptionist')
> gui_task

Full name of your skill/application? (eg 'The Receptionist Robot' or 'Database connector', press Return to use the ID. You can change it later)
> <leave empty to re-use the ID>

Choose a template:
1: base task template [python]
2: simple task template with a graphical user interface [python]
3: 'greet' task mock-up [python]


Your choice?
> 2

What robot are you targeting?
1: Generic robot (generic)
2: Generic PAL robot/simulator (generic-pal)
3: PAL ARI (ari)
4: PAL TIAGo (tiago)
5: PAL TIAGo Pro (tiago-pro)
6: PAL TIAGo Head (tiago-head)

Your choice? (default: 1: generic)
> 2

Choose the simple_ui template (option 2), and the generic-pal robot (option 2).

The tool will then create a simple yet complete ROS 2 task, with a graphical interface we can use as a starting point.

Note

You can also pass all the rpk parameter directly from the command-line:

rpk create -p src/ --robot generic-pal task -y --id gui_task --template simple_ui

Check rpk --help for more information on the available parameters.

4. build and source the workspace:

colcon build

Step 2: running the task

To run the task and display the user interface, you need to: 1. start the UI server, which will display the user interface; 2. start the task node. The task, however, will not run unless instructed to 3. call the task’s control server, to effectively start the task.

To do so, we need three terminals, each connected to the Docker container.

Note

Every time you need to connect to your Docker container from a different terminal, run the same two commands:

docker exec -it pal_docker bash
source /opt/pal/alum/setup.bash

1. open a new terminal, connect to your Docker container, and start the UI server.

   In a new terminal:

   docker exec -it pal_docker bash
   source /opt/pal/alum/setup.bash
   ros2 run ui_server ui_server
   

   The UI server window should open:

   

2. start the task. In the same terminal as the one use to compile your task, launch the node:

   source install/setup.bash
   ros2 launch gui_task gui_task.launch.py
   

   Nothing is displayed yet, as the task is currently ‘sleeping’: waiting to be explicitely started.

   You should however see something similar to this in the terminal, indicating that the task is correctly started:

   [INFO] [gui_task]: Initialising...
   [INFO] [gui_task]: Task gui_task started, but not yet configured.
   [INFO] [gui_task]: Task gui_task is configured, but not yet active
   [INFO] [gui_task]: Listening for UI messages on topic </gui_task/ui_msg>
   [INFO] [gui_task]: Task gui_task is active and running
   

3. run the task. In a third terminal, run:

   docker exec -it pal_docker bash
   source /opt/pal/alum/setup.bash
   source ~/exchange/install/setup.bash # needed for the 'task_msgs' package
   
   ros2 action send_goal /gui_task/control task_msgs/action/TaskControl "task_data: ''"
   

   You should now see the user interface of the task:

   

Important

You might have noticed that the ros2 action call did not return: this is expected! while the task is not complete, the action remains active, and the terminal is waiting for the task to finish.

Interacting with the task

Try to press the Do some work button a few times in the user interface. You will see the progress bar going up:

If you look at the terminal where the task is running, you will see the following messages:

[INFO] [gui_task]: Received UI message: button pressed 1 times. Increasing task completion by 10%
[INFO] [gui_task]: Completed 10% of the task
[INFO] [gui_task]: Received UI message: button pressed 2 times. Increasing task completion by 10%
[INFO] [gui_task]: Completed 20% of the task
[INFO] [gui_task]: Received UI message: button pressed 3 times. Increasing task completion by 10%
[INFO] [gui_task]: Completed 30% of the task

In reality, the task script is interacting with the user interface in a bi-directional way:

The initial visuals are set by calling the /ui/set_fragment service (exposed by the robot’s ui_server), and passing a QML fragment.

Note

QML is a declarative language used to create user interfaces, and is the language used by the ui_server to display the user interface on the robot’s screen.

Learn more about QML.

The task script then starts listening to the /gui_task/ui_msg topic, which is the topic where the user interface sends messages to the task script.

Every time the user clicks on the button, the user interface sends a message on this topic:

gui_task/res/ui/TaskUI.qml

import QtQuick 2.15
import Ros 2.0

Item {
    // [...]

    StringTopic {
        id: stringTopic
        isSubscriber: false
        isPublisher: true
        topic: "/gui_task/ui_msg"
    }

    // [...]

    MainScreen {
        id: mainScreen
        property int counter: 0

        // [...]

        // btnAction1 is defined in MainScreen.ui.qml
        btnAction1.onClicked: {
            counter += 1;
            stringTopic.value = "button pressed " + counter + " times";
            stringTopic.publish();
        }
    }
}

The task script then receives this message, and updates the task progress by publishing a message on the /gui_task/task_progress topic:

gui_task/gui_task/task_impl.py

# [...]

 class TaskImpl(Node):
     def __init__(self) -> None:
         super().__init__('task_gui_task')
         # [...]
         self.completed = 0
         self._ui_msg_sub = None
         self._task_progress_pub = None

     # [...]

     def on_ui_msg(self, msg: String) -> None:
         self.get_logger().info(f"Received UI message: {msg.data}. "
                             "Increasing task completion by 10%")
         self.completed += 10
         self._task_progress_pub.publish(Int16(data=self.completed))

     def on_activate(self, state: State) -> TransitionCallbackReturn:
         # [...]
         self._ui_msg_sub = self.create_subscription(String,
                                                     '/gui_task/ui_msg',
                                                     self.on_ui_msg,
                                                     10)

         self._task_progress_pub = self.create_publisher(Int16,
                                                         '/gui_task/task_progress',
                                                         10)
         # [...]
         return super().on_activate(state)

     # [...]

Finally, the UI listen to the /gui_task/task_progress topic, and updates the progress bar accordingly:

gui_task/res/ui/TaskUI.qml

import QtQuick 2.15
import Ros 2.0

Item {
    // [...]

    StringTopic {
        id: stringTopic
        isSubscriber: false
        isPublisher: true
        topic: "/gui_task/ui_msg"
    }

    IntTopic {
        id: taskProgressTopic
        value: 0
        isSubscriber: true
        isPublisher: false
        topic: "/gui_task/task_progress"
    }

    // [...]

    MainScreen {
        id: mainScreen
        property int counter: 0

        taskProgress: taskProgressTopic.value

        // [...]

        // btnAction1 is defined in MainScreen.ui.qml
        btnAction1.onClicked: {
            counter += 1;
            stringTopic.value = "button pressed " + counter + " times";
            stringTopic.publish();
        }
    }
}

The visuals are actually updated in the implementation of the MainScreen QML component, in MainScreen.ui.qml:

gui_task/res/ui/MainScreen.ui.qml

import QtQuick 2.15
import "js/Constants.js" as Constants

Rectangle {
    id: mainScreen
    // [...]
    // this property is updated from TaskUI.qml when a msg is received
    // on /gui_task/task_progress
    property int taskProgress: 0

    // aliasing the button so that it can be accessed from the parent component
    // (TaskUI.qml in this case)
    property alias btnAction1: btnAction1

    // [...]
    IconButton {
        id: btnAction1
        x: 95
        y: 398
        label: "Do some work"
    }
    // [...]
    Rectangle {
        id: taskProgressBar
        x: desc.x
        anchors.top: btnAction1.bottom
        anchors.topMargin: 30
        width: desc.width * (mainScreen.taskProgress/100)
        height: 10
        color: Constants.accentColor
    }

    Text {
        id: taskProgressLabel
        anchors.left: taskProgressBar.left
        anchors.top: taskProgressBar.bottom
        anchors.topMargin: 10
        text: mainScreen.taskProgress + "%"
        font: Constants.font
        color: Constants.fgColor
    }

    // [...]
}

Summary

In this tutorial, you have learned how to create a simple user interface for your robot application, using the rpk tool to generate a task skeleton with a graphical user interface, and how to run it from a Docker image.

You have also learned how to interact with the user interface, and how to connect it to your task script, so that the user interface can send messages to the task script, and the task script can update the user interface.

In this tutorial, bi-directional communication between the UI and the main script is achieved using ROS topics. Other mechanisms are available (using ROS actions, services or parameters, or using /ui/update_state): you can combine them to best fit your needs, as we might see in follow-up tutorials.

Next steps

Modify the user interfaces

The user interface is written in QML. You can edit it either with Qt Design Studio, or by directly editing the QML files in a text editor.

• 🚧 How-to: Edit QML interfaces with Qt Design Studio

• 🚧 How-to: Edit QML interfaces source code

You can modify the user interface by editing the QML files in the gui_task/res/ui/ folder. The main file is TaskUI.qml, which contains the

Install on the robot

You can also deploy this sanple task on your robot, and run it directly from the robot’s touchscreen. Check the Deploying ROS 2 packages on your robot page.

See also

• 🚧 Tutorial: Creating a interactive user interface

• Automatic code generation with rpk

• Back to the 🖥️ User interfaces main page.