|Note: This tutorial assumes that you have completed the previous tutorials: ROS tutorials.|
|Please ask about problems and questions regarding this tutorial on answers.ros.org. Don't forget to include in your question the link to this page, the versions of your OS & ROS, and also add appropriate tags.|
Configuring a Robot's Diagnostic AggregatorDescription: This tutorial will help you set up and configure a diagnostic aggregator, using the PR2 as an example. The Aggregator can be configured for a robot or type of robot to perform basic analysis on the diagnostics.
Tutorial Level: INTERMEDIATE
Next Tutorial: Creating a Diagnostic Analyzer
Diagnostic Aggregator Basics
The diagnostic_aggregator package contains the tools for collecting, categorizing, and analyzing diagnostics data from a robot. The Aggregator class subscribes to raw data on the /diagnostics topic, and publishes /diagnostics_agg at 1Hz. To process the data, the aggregator loads "diagnostic analyzers" at startup. These analyzers can examine any data item, and output processed diagnostics data.
To keep this simple, we'll only use the diagnostic_aggregator::GenericAnalyzer and diagnostic_aggregator::AnalyzerGroup analyzer types, which are both in the diagnostic_aggregator package. It's useful to know a little about the GenericAnalyzer, so you may want to look at the tutorial Using the GenericAnalyzer if you have questions.
We'll go through the process of setting up a diagnostic aggregator, or aggregator_node, for a PR2. We'll divide up the diagnostics data into different types, and set up our analyzers to categorize the raw data. We can test this using bag files of raw data and the robot_monitor.
Looking at the Raw Diagnostics
To start with, we can view the raw data from a PR2. We can grab a bag file of recorded diagnostics data, or just subscribe to an actual robot. To view it, use the runtime_monitor. All raw diagnostics is published on the /diagnostics as a diagnostic_msgs/DiagnosticArray topic.
Looking at the raw diagnostics data on a PR2, we can see that we have broad categories:
- EtherCAT devices (motors)
- Power System
- Sensors and Drivers
- Forearm/Dual Stereo Camera
The easiest way to analyze the diagnostics is to simply group everything into categories. We'll set up an analyzer for each group or sub group above.
Setting up the Aggregator
To configure our aggregator_node, we'll start by grouping all the EtherCAT Devices together. These devices are the PR2 actuators, plus the LED projector board and EtherCAT hubs.
To get the aggregator_node to load our analyzers, we'll need to set up the analyzers in the private parameter space of the aggregator_node. Make a YAML file to store the parameters. You can make it in any package, or just use the existing "diagnostic_aggregator/demo/pr2_analyzers.yaml". This configuration file will load into the private parameter namespace of an aggregator_node.
pub_rate: 1.0 # Optional base_path: '' # Optional, prepended to all diagnostic output analyzers: motors: type: GenericAnalyzer path: Motors startswith: 'EtherCAT'
In the analyzers namespace, we made our first analyzer in the motors namespace. Let's look at each parameter under motors.
type: This is mandatory for all analyzers. It tells the aggregator_node which plugin class to load.
path: All output from this GenericAnalyzer be under this name. (Ex: "EtherCAT Device (head_pan_motor)" -> "Motors/EtherCAT Device (head_pan_motor)")
startswith: This tells the GenericAnalyzer to analyze any component that starts with 'EtherCAT'. See the tutorial Using the GenericAnalyzer for details.
As noted above, the pub_rate and base_path parameters are optional. They default to 1.0 and "", respectively.
When we categorized the diagnostics, the sensors went under "Sensors", but in different sub categories. To use sub categories in the aggregator_node, we use the AnalyzerGroup.
To the same YAML file above, add:
sensors: type: AnalyzerGroup path: Sensors analyzers: base_hk: type: GenericAnalyzer path: Base Hokuyo timeout: 5.0 find_and_remove_prefix: base_hokuyo_node num_items: 3 tilt_hk: type: GenericAnalyzer path: Tilt Hokuyo timeout: 5.0 find_and_remove_prefix: tilt_hokuyo_node num_items: 3 imu: type: GenericAnalyzer path: IMU timeout: 5.0 find_and_remove_prefix: imu_node num_items: 3
This sets up an AnalyzerGroup to cover "Sensors", and moves the Hokuyo's and IMU into a sub category. An incoming message "imu_node: Connection Status" will become "Sensors/IMU/Connection Status", and be 2 levels down in the Robot Monitor.
The AnalyzerGroup will initialize "sub-analyzers" for any Analyzer specified in its analyzers namespace.
Adding Remaining Analyzers
To add more analyzers to the aggregator_node, add the parameters to YAML setup file.
computers: type: GenericAnalyzer path: Computers contains: [ 'HD Temp', 'CPU Usage', 'CPU Temperature', 'HD Usage', 'NFS'] joints: type: GenericAnalyzer path: Joints startswith: 'Joint' power: type: GenericAnalyzer path: 'Power System' timeout: 5.0 startswith: [ 'Smart Battery', 'Power board'] expected: [ 'IBPS 1', 'IBPS 2', 'IBPS 3', 'IBPS 0']
This will "fill out" the Analyzer with the categories for the power system, computers, and joints.
Launching the Aggregator
When you're done making the above YAML file, add it to a roslaunch file so it loads in the aggregator_node's namespace.
Your launch file should look something like this:
<launch> <node pkg="diagnostic_aggregator" type="aggregator_node" name="diagnostic_aggregator" > <!-- Load the file you made above --> <rosparam command="load" file="$(find my_robot_package)/analyzers.yaml" /> </node> </launch>
Save this launch file as "aggregator.launch", and launch it.
It's important to test your newly configured diagnostic aggregator. You'll have to view it with the robot_monitor. Make sure you have the robot monitor up to date and built. See Using the Robot Monitor for details.
To test, pull a diagnostics logfile from your robot. On the PR2, these files are recorded automatically (stored in /hwlog on the robot). If your robot does not automatically record, you will have to manually record one using rosrecord. Copy the file to your desktop.
Play back and verify that the log file works using the runtime_monitor. Open three terminals.
roscore # Terminal 1 rosplay YOUR_LOGFILE # Terminal 2 rosrun runtime_monitor monitor # Terminal 3
You should see raw diagnostics data from the robot. Leave these terminals up.
Now, launch the file you just made:
View with the robot_monitor:
rosrun robot_monitor robot_monitor
Compare the robot_monitor and the runtime_monitor. The robot_monitor should be categorized according to your configuration. Diagnostic items in error or warning state should make their parents in error or warning as well.
Shutdown your aggregator with Ctrl-C, and the robot_monitor should report stale and turn gray in a few seconds.
If you wish to alter categories, modify your YAML file and re-launch. If you're satisfied with the aggregator configuration, add the launch file to the robot's main launch file.
<include file="$(find my_pkg)/aggregator.launch" />
See pr2_bringup for existing diagnostics aggregators on robots. The file pr2_bringup/config/analyzers.yaml on the PR2 robot holds the configuration instructions, and the file pr2_bringup/pr2.launch starts the aggregator_node on startup.