[Documentation] [TitleIndex] [WordIndex

(!) Please ask about problems and questions regarding this tutorial on answers.ros.org. Don't forget to include in your question the link to this page, the versions of your OS & ROS, and also add appropriate tags.

Running ROS across multiple REMOTE machines

Description: This tutorial expands on the previous Tutorial (Running ROS across multiple machines) to include the discussion on remote ROS networks. That tutorial focuses on the situation where the multiple machines are connected by the same local network, i.e. they all share the same public IP address (See figure 2 below). This expands that discussion to Remotely networked machines, i.e. machines that are connected to each other through separate internet hotspots or so that each machine has its own public IP address (See figure 3).

Tutorial Level: INTERMEDIATE

Next Tutorial: Defining Custom Messages

Overview

This tutorial covers issues that are outside the scope of ROS, such as Internet Protocols, networking, modem setup, ISP issues, and so on. This section introduces a quick review of these issues and why they matter.

The ROS network

As discussed in the previous Tutorial, ROS is designed for distributed computing, where a number of connected components (robots, computers, mobile device, etc.) are networked as seen here.

rosnet.png

Figure 1: The ROS network

Since all of these component share the same modem (or hotspot), they form a network local to this modem, and therefore this network is referred to as the local ROS network, in which all connected components have the same public IP address, which is the IP address of the modem. Every remote query (from the internet) to any component of this network is blocked at the modem. This is a fundamental feature of Internet and a security protocol to protect the local components against hacking and other threats.

localip.png

Figure 2: The Local ROS network

Within the local ROS network, every component is identified with its unique local IP addresses (see figure 2). These are the IP addresses used to run ROS across multiple local machines, as discussed in the previous Tutorial

The remote ROS network

In many robotics applications, such as outdoor robots, there comes a need for the robot to cover vast areas and be separated from the operator. Examples include agriculture robots, search & rescue Robots, and others. In these situations, the robot and operator might not share the same modem (or internet hotspot) and instead each connect to the internet separately, and then connect remotely to each other. This setup is referred to as the remote ROS network.

remoteip.png

Figure 2: The Remote ROS network

Figure 2 also shows the challenge of remote ROS networks. We can no longer use local IP addresses (as suggested in the previous Tutorial) to identify components since each component resides in a separate local network. Also, we cannot use their public IP addresses either, since these IP addresses refer to their modems rather than to the components thesmselve.

For example, if the laptop in figure 2 attempts to contact the robot using the robot's public IP address, it would not be able to do so, because that IP address refers to the robot’s modem and not the robot itself, and vice versa.

Therefore, there is a need to Forward that connection from modems onto their internal components, thus facilitating the requirements of ROS and so establishing a successful remote ROS network. This brings the need for PortForwarding

PortForwarding (PF)

There has been a number of approaches to set up remote ROS networks, some include Cloud Robotics, Third-party web solutions, Android, and/or combinations. An extensive review of these efforts can be found here. This tutorial focuses on using PortForwarding as the tool to enable the Remote ROS networks. PortForwarding has its advantages and limitations over other methods. These are summarized below:

Advantages of PortForwarding

Cloud-based solution facilitate remote ROS network by transferring the contents of the ROS messages through the cloud. This would require a two-way data conversion, or rosmsg-webformat-rosmsg bridge for each rosmsg being transferred throug the cloud.

A simple Caller/Listener application contains few rosmsgs so it might require few bridges, but a complex outdoor robot application might contain hundreds, or even thousands, of rosmsgs that each would require an individual bridge to be transferred through the cloud.

Figure 4: PortForwarding vs. Cloud-based solutions for remote ROS networks

Furthermore, should there be any changes or modifications to the system (introducing a new sensor, modifying robot hardware, etc.) new bridges would be needed as well.

Port forwarding offers an alternative approach, as seen in Figure 4, it opens up a remote ROS-to-ROS connection where ALL rosmsgs in the application can be remotely transferred without any conversion or bridges needed. PF also supports any future system changes or updates.

Challenges of PortForwarding (and how to overcome them)

Unfortunately, the use of Portforwarding has its limitations, but these limitations can be managed:

  1. As shown below, PF require the beforehand knowledge of all public IP addresses of the networked components. This is further problematic if these IP addresses were dynamically changing (as set by the ISP). However, this tutorial provides a solution to manage this problem.
  2. Implementing PF is multi-layered proces that involves settings with the ISP, Modem/Routers setup, Operating System configuration, setup within ROS itself, and in that order. This makes setup and troubleshooting a challenge, especially for beginners.
  3. Not all ISPs allow or support the use of PF, those that do might require permission/special access rights.

The next section details the steps needed to configure, troubleshoot, and overcome the challenges of implementing PortForwarding for remote ROS networks.

Configuration / Troubleshooting Steps

It is important that the following steps are tackled in the given order: ISP => Modem => Operating System => ROS => Dynamic IPs. Some settings in later steps depend on the successful implementation of the earlier steps. The steps below might sound trivial/redundant to some, but for best results, it is recommended that you follow steps as recommended here. The following sections show the outline/summary of the configuration/Troublshooting steps. For detailed, line-by-line steps with screenshots and explanations, refer to the attachedPaper. (henceforth, this will be referred to as "The Paper")

STEP 1: ISP Settings

Configuration Steps:

  1. Verify with your provider/network Admin that PF is supported/allowed/enabled on your network.
  2. Optional (but recommended): Obtain a fixed Public IP address for your modem.

Verification Steps (verify that PF is allowed on your network):

  1. Nothing to be verified here, apart from confirmation from your provider/admin that PF is allowed/enabled.
  2. Once Verified, proceed to STEP 2

STEP 2: Modem/Router Settings

At this stage, your ISP has enabled/allowed Portforwarding, next we need to configure the modem so it can allow remote queries to be forwarded to your robot (or other components from the ROS network, see figures 1 & 3)

Configuration Steps:

  1. On your browser/App, Login to your modem homage, navigate to firewall settings => enable Port Forwarding, DMZ, and other filtering settings

  2. Optional (but recommended): Set the local IP address and port range (see the Paper for details)

  3. Repeat this step for every modem of every machine of this remote network (see figure 3)

routerpf.png

Figure 5: Modem Configuration for PortForwarding

Verification Steps (verify that your modem/router successfully allowed PF):

  1. First insure that your connection is up, open a browser and connect to the internet
  2. Select a Port number between 1024 and 65000
  3. Host a netcat chat session on the selected port (nc -l portNumber)
  4. On the browser, open a PortChecker website and check the status of the portNumber you selected

  5. If the status is OPEN, then this step is completed and you have successfully enabled PF on your modem
  6. Once Verified, you can proceed to STEP 3

STEP 3: Operating System (Linux) Settings

At this stage, your ISP has allowed/enabled Portforwarding, and your modem/router was properly configured to forward queries to your machine. Next we need to configure your system so it can identify and interact with all members of the remote ROS network (See figures 1 & 3), as per ROS networking requirements. The following steps are to be conducted on every member of the remote ROS network (robot, laptop, etc.).

For now, let us assume that all Public/Local IP addresses are fixed. Fixed public IPs can be granted by the ISP, fixed local IPs are defined/reserved in the modem's home page. Dynamic IPs will be covered later.

Configuration Steps:

  1. In the hosts file, define all public/local IP addresses and hostnames for all components in the ROS network.

  2. Optional (but recommended): Define as many definitions as possible, be it for local/remote networks, through fixed/mobile modems, etc. (see Figure 6)

hosts_actual2.png

Figure 6: Hosts definitions for multiple network configurations

Verification Steps (verify that hosts were defined properly for PF):

To verify that hostnames and IP addresses have been correctly configured, we will use three tests; the Ping test, the SSH test, and the netcat test. These tests are chosen to verify that ROS network requirements has been meet, namely: 1) each member can ID all other members in the network by their IP Public addresses (Ping), 2) each member can can access other members in the network (SSH), and 3) each member can act as server and/or client within the network (netcat).

STEP 4: ROS Settings

At this stage, your ISP has enabled/allowed PF, your modem/router is properly configured, and each member of your network has been configured to recognize and interact with all other members of the network remotely through PortForwarding. Next, we need to configure ROS within each machine to facilitate the remote ROS nework. The Previous Tutorial begins from this point. Configuration Steps:

  1. In the ROSmaster member of the network, open a terminal ...

  2. Run this command: export ROS_MASTER_URI=ROSmaster_local_IP:11311 (replace ROSmaster_local_IP with the local IP address of the ROSmaster machine)

  3. In all other components of the network, open a terminal ...
  4. Run this command: export ROS_MASTER_URI=ROSmaster_public_IP:11311 (replace ROSmaster_public_IP with the public IP address of the ROSmaster machine)

Now, every machine in the remote network knows which machine is the ROSmaster, which is another requirement of ROS networking requirements.

Verification Steps:

Once again, there will be three tests conducted to verify that everything is configured properly, and once again, these tests are to be run on every member of the remote ROS network.

If all is OK up to this point, then congratulations, the remote ROS network is functioning through PortForwarding! Next, you will need to verify it further by testing it with a real robot. Using Turtlebot's interaction as an example.

STEP 4: Managing Dynamically Changing IPs

Some ISPs may not provide fixed Public IPs to subscribers. Instead, they would provide a range of values for their IPs, such as 173.17.XXX.XXX, where XXX could range from 000 to 255. This would obviously complicate the setup and configuration steps discussed above as steps 2 through 4 would need to be revisited every time the IP addresses are changed. The Solution is to automated this step through the use of environment scripts and cloud solutions (ironic!), as seen in figure 7.

auto_graph2.png

Figure 7: Auto-Updating IPs to overcome Dynamic IP addresses

Current IP definitions (similar to those in the hosts files) would be stored in a cloud-storage facility such as Dropbox or other. Each member would have access to the file and can read it /write to it. Environment scripts in the ROSmaster would check the current IP values and update configuration autonomously through out the network. The degree of autonomy can be defined by the user.


Configuration Steps (Auto-Updating IP addresses)

The actual scripts and further discussion can be found in the Paper (and its accompanying Github page), the following is the Auto-update algorithm and logic.

  1. Upon starting up the ROSmaster machine, the master-script is run, which does the following:
    1. Checks current Public IP address of the machine
    2. Compares the IP address with the values stored in the hosts file in the cloud.

    3. If values are the same, then exit (no changes/updates needed)
    4. If values are different, then update PF settings
      1. update hosts values

      2. Update ROSmaster_Public_IP and ROSmaster_Local_IP
      3. Once values are changed in file on storage facilities, all members in network are notified.
      4. This triggers environment scipts in all member machines to update values accordingly.

The degree of Autonomy (completely autonomous or User-verified) can be defined by tweeking the scripts (and commenting/uncommenting some specific lines)


2019-10-19 12:27