Tutorial: Building a first touchscreen interaction

Section

Touchscreen - tutorials

🏁 Goal of this tutorial

By the end of this tutorial, you will know how to integrate your interactive screen with the robot’s ROS framework, by publishing to ROS topics.

Pre-requisites

• You should first have completed ARI’s Touchscreen manager tutorial. More specifically, you must know how to upload your own custom HTML page.

Note

This tutorial guides you to create interactive HTML content for the touchscreen ‘from scratch’.

Follow instead Create an application with pal_app if you wish to start from a template.

How to publish in a ROS topic from a custom HTML page

The goal is to display a question to the user and a list of buttons with possible answers. When the user presses a button, the selected answer is sent through a ROS topic.

Step 1: Start your HTML page.

Build your own custom HTML page, or download a simple example from here.

Step 2: Import the ROS library and initiliaze it.

Use rrlib.js to access the robot’s topics. The library is already installed in your robot in ~/.pal/www/webapps/pages/js/modules/.

Import the library in your code as follows:

import * as RRLIB from '../../js/modules/rrlib.js'

In the sample code, you can find this line in the main.js file.

To use the module, you need to initialize it:

let ros = new RRLIB.Ros({
         host: 'http://' + window.location.hostname
});

Step 3: Link to the topic.

Create a variable that will link to a ROS topic you want to publish in. In this example, we are using the user_input topic:

let user_input_topic = new RRLIB.Topic({
     ros: ros,
     name: 'user_input'
});

Step 4: Publish to the topic.

Publish to the topic by passing a JSON object that follows the ROS message structure. In our example:

user_input.publish({
     action: 'some action',
     args: [{
         key: 'some key',
         value: 'a value'
     }]

Step 5: Upload the .zip file containing the HTML page you just created (follow the steps in Custom HTML) and publish the page (follow the steps in Publishing a page).

Step 6: Try it! Open a terminal, log into the robot and watch the output of the user_input topic:

ssh pal@ari-0c
rostopic echo /user_input

If you’re running the sample page in this tutorial, you should observe the following output in your terminal every time you press a button on the touchscreen:

pal@ari-0c:~$ rostopic echo /user_input
action: "Yes"
args:
  -
    key: ''
    value: ''
---

That’s all!

Writing a simple interactive behavior

Let’s write a simple script that reacts to user input. In this example, the robot will answer one way or another, based on the button pressed by the user.

Here is the code of the interaction script:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import rospy

from actionlib import SimpleActionClient
from pal_interaction_msgs.msg import TtsAction, TtsGoal, Input
from pal_web_msgs.msg import WebGoTo

class RobotBase(object):

    def __init__(self):
        rospy.loginfo("RobotBase::init node")

        # Connect to the text-to-speech action server
        self.tts = SimpleActionClient('/tts', TtsAction)
        self.tts.wait_for_server()
        rospy.loginfo("RobotBase::Connected to TTS action client")

        # Create a publisher to publish web pages
        self.web_publisher = rospy.Publisher(
            "/web/go_to", WebGoTo, queue_size=10)
        rospy.sleep(0.5) # allow some time to properly connect the publisher
        rospy.loginfo("RobotBase::Publishing in: " +
            self.web_publisher.resolved_name)

        # Create a subscriber to the touch screen inputs
        self.user_input_sub = rospy.Subscriber(
            '/user_input', Input, self.user_input_touch_cb)
        rospy.loginfo("RobotBase::Subscribed to " +
            self.user_input_sub.resolved_name)


    def publish_web_page(self, web_name):
        msg = WebGoTo()
        msg.type = WebGoTo.TOUCH_PAGE
        msg.value = web_name
        self.web_publisher.publish(msg)


    def say(self, msg):
        goal = TtsGoal()
        goal.rawtext.text = msg
        goal.rawtext.lang_id = "en_GB"
        self.tts.send_goal_and_wait(goal)

    def user_input_touch_cb(self, data):
        """ Callback called when the user presses a button on the
        touchscreen.
        """
        if data.action:
            data = str(data.action).lower()
            rospy.loginfo("RobotBase::Button pressed: " + data)

            if data == "yes":
                self.say("Wonderful")
            elif data == "no":
                self.say("No problem! Maybe later")

            self.publish_web_page('question_sample')



if __name__ == '__main__':
    print("Starting robot demo...")
    try:
        rospy.init_node('first_interaction_demo')
        robot = RobotBase()
        robot.publish_web_page('question_sample')
        rospy.spin()
    except rospy.ROSInterruptException:
        pass

Important

While this script is absolutely fine, we would normally recommend to use Intents to communicate between the touchscreen and an application controller.

Check the templates generated by the pal_app tool to see an example.

Let’s go through the code above:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import rospy

from actionlib import SimpleActionClient
from pal_interaction_msgs.msg import TtsAction, TtsGoal, Input
from pal_web_msgs.msg import WebGoTo

We need to import actionlib to send actions to the robot. In our example, the robot will provide a verbal answer through speech synthesis (text-to-speech, or TTS). We thus need to import the TTS related messages TtsAction and TtsGoal. To receive the user input based on the button pressed by the user we need to import the Input message. Finally, we will also need to publish a web page on the screen. Thus, we need to import the WebGoTo message.

class RobotBase(object):

    def __init__(self):
        rospy.loginfo("RobotBase::init node")

        # Connect to the text-to-speech action server
        self.tts = SimpleActionClient('/tts', TtsAction)
        self.tts.wait_for_server()
        rospy.loginfo("RobotBase::Connected to TTS action client")

        # Create a publisher to publish web pages
        self.web_publisher = rospy.Publisher(
            "/web/go_to", WebGoTo, queue_size=10)
        rospy.sleep(2) # allow some time to properly connect the publisher
        rospy.loginfo("RobotBase::Publishing in: " +
            self.web_publisher.resolved_name)

        # Create a subscriber to the touch screen inputs
        self.user_input_sub = rospy.Subscriber(
            '/user_input', Input, self.user_input_touch_cb)
        rospy.loginfo("RobotBase::Subscribed to " +
            self.user_input_sub.resolved_name)

We define our class, where we start by creating the TTS client, to send the utterances we want the robot to reproduce.

Next, our node needs to publish the web pages we want to show on the touch screen. Finally, we subscribe to the touch screen input to get the user’s answers.

def publish_web_page(self, web_name):
    msg = WebGoTo()
    msg.type = WebGoTo.TOUCH_PAGE
    msg.value = web_name
    self.web_publisher.publish(msg)

This function publishes a message indicating the name of the custom HTML page ( web_name) that we want to show.

def say(self, msg):
    goal = TtsGoal()
    goal.rawtext.text = msg
    goal.rawtext.lang_id = "en_GB"
    self.tts.send_goal_and_wait(goal)

This function sends the text we want the robot to say. In this case, in English.

def user_input_touch_cb(self, data):
    if data.action:
        data = str(data.action).lower()
        rospy.loginfo("RobotBase::Button pressed: " + data)

        if data == "yes":
            self.say("Wonderful")
        elif data == "no":
            self.say("No problem! Maybe later")

        self.publish_web_page('question_sample')

This callback function is called every time a new messages arrives through the Input topic, i.e. every time a user presses a button on the touch screen. In this example, to simplify the code, we process the data received in the same callback and act accordingly by verbally replying to the user. However, it is good practice to leave any processing outside a callback function.

if __name__ == '__main__':
    print("Starting robot demo...")
    try:
        rospy.init_node('first_interaction_demo')
        robot = RobotBase()
        robot.publish_web_page('question_sample')
        rospy.spin()
    except rospy.ROSInterruptException:
        pass

In the main function, we start our node, publish the page we want to show and we are ready to start interacting with the robot!

Note

Remember to upload your custom HTML web page via the touchscreen manager before running your node. You can follow the steps in Publishing a page.

If you don’t know how to deploy your package in the robot, check the Tutorial: Deploying ROS packages on the robot page.

See also

• Create an application with pal_app