One Size Fits All?
The first thing I wanted for the robot was a good way to control it so I can drive the robot as smoothly and comfortably as I drive my mobility scooter or my wheelchair. I also wanted to show off how flexible this can be, so I put together a system that lets you mix-and-match inputs with outputs, for trillions of possible ways of putting together a personalized way of controlling the robot. Don’t worry if that seems like a lot; it’s more like a lot of ways to personalize. I’ve tried maybe a dozen, and only have a few favorites.
Anything that requires me to hold my arms out in front of me (i.e., pretty much all minor household chores) tires me out super quickly, so being able to operate the robot while sitting on the couch would save me a ton of energy. Doing teleoperation with the built-in interface last time got me pretty excited about the range of chores I could get the robot to do if I practice – the wrist/gripper setup is really handy (ha!) for doing surprisingly fiddly tasks.
The Accessible Design Brief
At its heart, teleoperation (a person controlling the robot) involves matching some input interface, like a controller or a keyboard, to some output scheme like controlling the angle of each of the motors on the robot or controlling where the gripper is in 3D space. There aren’t really any limits on what those input interfaces or control schemes can be, and one of the key principles of accessible design is flexibility – there’s no such thing as one setup that will work for everyone. So a key design goal for this project for me was to make the demo as flexible as possible. I implemented a half dozen or so [check numbers] interfaces and about the same number of control schemes, and an application to customize how the interface maps to the control scheme. If you’ve ever played a video game that lets you customize its controls, it’s exactly the same principle: the interface is the selection of “buttons” and the control scheme is the slots/controls you can set the “buttons” to control.
Implementation Approach
The key to make this as flexible as possible was to take advantage of ROS 2 topics and remapping. The connection between control schemes and interfaces is managed using the Joy message type (short for Joystick, not an expression of delight), which carries information about the state of axes (continuous values) and buttons (integer values). To simplify the mappings, we assume that axes are in [-1,1] and buttons are either 0 (un-pressed) or 1 (pressed). An interface is a node that publishes a Joy message on a topic, and for convenience, provides a service that returns lists of names for the buttons and axes, and a control scheme is a node that listens for a Joy message and moves the robot, and similarly for convenience, provides a service that returns lists of names for what the buttons and axes do.
Side quest: Sim/Real Unification
To make it easy to test this in simulation, I needed to bring the ROS 2 APIs for the simulated and real robots into sync, preferably in such a way that it would be easy to keep them synchronized. I added a superclass to hello-helpers that does this, in addition to exposing a wider range of parameters to the user. This new API will be released soon with its own blog post. If you want to try out my code before it’s released, you’ll need to make sure you check out the feature/sim-real-api-alignment branch of stretch4_ros2.
The main node provides a UI that maps button and axis states from the interfaces to button and axis states for the control scheme(s), so that the user can adjust what button maps to what control scheme behavior (e.g., “right stick X > robot arm extension/retraction”). For now, the system only allows one control scheme at a time, but as long as there’s no conflict in topic names, you can have as many input interfaces as you’d like. So you could control the robot with a combination of keyboard, head tracking, gaming controller, etc.
Finally, I thought it would be helpful to support some basic multi-button controls, like holding the trigger and moving the stick to switch from controlling the base x/y to controlling the arm height/extension. So I added the idea of “virtual buttons” and “virtual axes”, which are button/axis controls that are composed from combinations of or transformations on other controls.
I’ve put a reference to all of the operations, interfaces, and control schemes I implemented at the end of this post. You can find all of the code in the multi_teleop_demo package.
What’s left?
Both the simplified control scheme and the kinematic control scheme are comfortable to use without too many degrees of freedom, so they’re definitely my favorites on that side. On the input side, I like the space mouse, controller, and head control, depending on the specific application. I’m looking forward to using them more to try to get used to them – like using any assistive technology (or driving a car or riding a bike or…), teleoperation has a learning curve before you get really good at it. I’m also looking forward to getting some of Hello Robot’s collaborators with disabilities to try out some of the interfaces and give feedback. Keep an eye out for updates in future blog posts!
Overall, I’m a big believer in having at least some minimal teleoperation available for home robots: teleop can get robots out of trouble, solve problems that can’t be solved autonomously, and allow users to communicate their preferences in an unambiguous way. That said, teleop can be clunky and slow: it’s worth it for people like me who need to be careful about conserving their energy or for folks who wouldn’t be able to do the task at all without it, but it’s not going to be a good solution for nondisabled people who just want a robot butler to help out around the house. I’d argue that home robotics hasn’t really solved the problem of a robot that’s “fun to drive” yet, or for the car people out there, robots that have good handling (the drone community knows what’s up here). Where my wheelchair and scooter are fun and responsive, and quickly have come to feel like an extension of my body, robot teleoperation just isn’t quite there. I don’t think I’ve fully solved it yet, but having these flexible interfaces feels like a step in the right direction.
AI Usage
As computing works out how best to use AI assistance, I think it’s helpful to hear how others use these tools. My approach for this project was as follows:
The sim/real unified API was mostly written by me to make sure that the interfaces worked the way I wanted them to, with AI filling in some functions that were straightforward to describe. The rest of the code is a mix of Antigravity (using Gemini Flash 3.5) and hand-coded, depending on which was faster and more useful (I have enough ROS experience that sometimes guiding the AI to generate code is slower than just writing it myself). Most UI stuff is AI, but superclasses and libraries are mostly hand-written by me and reviewed by AI for style and completeness. Tests are generally AI-generated with my supervision because writing tests is extremely boring (sorry QA/testing engineers, you’re amazing but I am not you). Docs are AI-generated and reviewed by me, and install instructions and supporting files are mostly AI-generated from the code. Every word on the blog is written/edited by me, although I have Gemini review the posts and provide suggestions on clarity, accuracy, and messaging (some, but not all of which I take – AI is not allowed to touch my writing directly). I also used Gemini as a “rubber ducky” for making a plan for the scenes in the video, but nothing in the video is AI generated – it’s all done by me with my phone, onboard robot cameras, and kdenlive (we love a good open-source piece of software).
Reference:
Composition/Transformation Operations:
Axes:
Squaring the magnitude: since axes are in [-1,1] this makes the axis feel more sluggish/calm (i.e., the same tilt angle results in a smaller output).
Square-root of the magnitude: does the inverse of squaring: makes the axis feel more responsive/sporty, where the same tilt angle results in a larger output.
Add deadband: Most controllers and steering wheels have a small range where a small value results in sending zero instead of the value – this keeps the system from feeling “twitchy”. To make things feel smooth, the math for this transformation is a little more complicated than just clamping to zero below some magnitude: instead, we adjust a piecewise linear transformation so that the value sent is zero in the deadband, then grows linearly from zero at the edge of the deadband to 1 at the limits as usual.
Axis to button: Sets a cutoff for transforming an axis to a button – above the value it’s 1, below it’s 0.
Axis override: Lets one axis override another – if the first axis has a zero value, sends the value of the second axis, otherwise sends the value of the first axis.
Average axes: Sends the average value of two axes.
Invert axis: Additive inverse of the axis (0.5 becomes -0.5, for example)
Buttons:
Logical operations on 1 or 2 buttons: AND, OR, NOT, NOR, NAND. Since button values are 0 or 1, we can do basic logical operations on one or more buttons to allow for complex button combinations. I’m not sure why you’d want to have a 4-button combination to get the robot to do something, but the important thing is you could.
Button override: like axis override, returns 1 if button A is pressed, regardless of button B, and 0 if neither button is pressed. Effectively equivalent to OR.
Button to axis: The inverse of axis to button – either two different values where 0 and 1 for the button map to the two values, or mapping two buttons to the axis such at if Button A is pressed it sends -1, neither is pressed it sends 0, and if Button B is pressed it sends +1.
Button-Axis combinations:
Axis 0 if Button: Sends the axis value normally unless the button is pressed, in which case it only sends 0.
Axis Value if Button: Sends the axis value normally only if the button is pressed, otherwise sends 0.
Interfaces:
Gaming Controller: I play a lot of video games, so the controller is a pretty intuitive interface, even if House Cleaning Simulator 3000 is the world’s most boring video game. Reading off a controller or joystick is one of the easiest interfaces to implement: it’s already built into ROS 2 in the joy package, although to pass axis names around I also implemented a controller node.
Mouse: This isn’t really a practical interface for most people since you’re probably using your mouse to control the UI. The mouse buttons get mapped to buttons, and mouse x/y on the screen gets translated to two axes. This interface has some potential for use with assistive devices that only control the mouse, especially being able to capture mouse clicks.
Space Mouse: The space mouse is my personal favorite for controlling robots – all the benefits of a joystick but with more degrees of freedom. If you’re not familiar, it’s a 6D (6-axis) mouse that can be rotated and translated in all directions (x, y, z, roll, pitch, yaw). It takes a little getting used to, but unlike a gaming controller, which usually only has 4 continuous axis (right stick x/y and left stick x/y), you get control over all 6 standard degrees of freedom at once, and with only one hand. I’ve used the space mouse a lot with the Kinova Jaco arm, and I like it here too. Fortunately, there’s a handy Python library for reading off the mouse, which makes it easy to implement a node that reads from the mouse and publishes Joy messages.
Face tracking: Using MediaPipe, this node tracks facial landmarks. Using a configuration file, you can select landmarks and map the distance between them to axes (e.g., mouth vertical aperture or eye opening). Alternatively, you can also set a cutoff for distance and map to a button press (e.g., blink). This node also sends overall 6D headpose as axes, with configurable min-zero-max ranges for each axis. A cool thing about this input interface is that it is highly accessible – all of Hello Robot’s partners have some head movement, so this is one that a lot of people could use, potentially in combination with one of the other interfaces.
Hand tracking: I have two nodes for hand tracking – a single hand tracker and a two-hand tracker. Like with face/head tracking, you get 6 axes for 6DOF position of either the one tracked hand or the centroid of the two hands. Then the same distance to axis or button mappings. I like pinching as a gesture for opening and closing the gripper: it feels very intuitive.
GUI: The most boring interface, but a useful complement to the others. This is just a GUI configured to have as many axes as sliders and buttons as buttons as you’d like. It’s useful for testing and useful if you have an assistive device that lets you manipulate user interfaces, but it’s pretty tedious to use with control schemes that have a lot of axes since you can only manipulate one axis at a time.
Control Schemes:
Motor position control: Motor position control is probably the most obvious way of mapping axes to robot behavior. Each axis controls a motor between a reconfigurable min and max value, and the published axis value puts the motor in that position.
Motor velocity control: Like motor position control but sends a velocity instead of a position. Velocity works well with interfaces that snap back to zero easily (like joysticks, controllers, and the space mouse). It’s less fun to use with interfaces that tend to stay where you put them and you have to search for the zero point (mouse, head pose) because it’s easy to accidentally move the robot by having the input slightly off zero.
Mixed control: Switches between position and velocity control depending on button state.
Kinematic control: Closely related to the flying gripper control used in the flying gripper demo, this control scheme maps your inputs to movement of the gripper from the perspective of a pre-specified frame or frames. If the frame is the gripper grasp point, this works exactly like flying gripper, but you can choose other points, like the base link, wrist link, or a fixed point in the environment. I haven’t tested it, but if you had a frame tracking the user, this would give you control that felt more like a video game where your head was the “camera” in the scene and your controls were interpreted relative to the camera.
Single button flying gripper: There are lots of folks out there who interact with computers using only a single button. One way of doing this is to use time to make your single button press do more: first a line scans from left to right, and a button press selects the horizontal position, then a second line scans top to bottom so that your button press selects a vertical position, giving a precise virtual click. To do something similar on the robot, in this mode, the griper scans left/right, then up/down, then forward backward along that line, and then moves until the button is pressed. This lets you fly the gripper around using only a single button, which is surprisingly fun!
Simplified motor control: I came up with this control scheme while I was feeling frustrated with the built-in gamepad control. It combines wrist pan with base rotation and wrist tilt with the shoulder lift, by having the wrist move until it hits its limit then moving the base/lift. This saves a couple of degrees of freedom in the mapping, which makes it easier to map to a gamepad controller.