uch has been written about the emergence of robotics, chief among which has been their impact on society, the economy, and the future of work. Robots are already being used for many purposes, including serving as remote-controlled agents in dangerous environments, manufacturing processes, and delivery tasks.
Advancements in Artificial Intelligence (AI) allow machines to function increasingly independently of this remote control. The AI stack can be broken down into two stages: a sensing stage to extract information about the robot’s environment and internal state, and a decision-making stage which uses this information to determine the robot’s next actions. For more information about the sensing stage, please check out our Computer Vision Selects which delves into using imagery to help computers understand the world around us. For this decision-making layer, Reinforcement Learning has been gaining strong momentum as a method to teach robots to solve problems in an unsupervised manner.
In this week’s ACM Selects, we provide a snapshot of current advancements in AI that impact robotics. Our selections briefly cover relevant articles regarding Reinforcement Learning and user experience design for robotics, as well as discussions on applications being used today.
While in this article we implicitly define a ‘robot’ as a single or few actuated devices able to sense and interact with its surroundings, the rise of the internet of things and advances of multi-agent systems may soon expand our understanding of “a robot” to a system potentially encompassing vast numbers of sensors and actuated parts. We will discuss one such IOT application, creating city-sized robots, in an upcoming ACM Selects on Smart Cities.
We invite you to consider participating in ACM’s activities on these topics, be it through our professional community, global policy activities, ongoing work in professional ethics, and/or through our chapters, SIGs, local meetups and/or conferences.
We value your feedback and look forward to your guidance on how we can continue to improve ACM Selects together. Your suggestions and opinions on how we can do better are welcome via email through selects-feedback@acm.org.
Reinforcement Learning
"Reinforcement learning is a way of not needing labels, or labeling automatically by who’s winning or losing—by the rewards, so you can learn to play the Go game by playing the moves and winning or losing, and no one has to tell you if that was a good move or a bad move because you can figure it out for yourself; it led to a win, so it was a good move.”
--Richard Sutton, Distinguished Research Scientist at DeepMind
Reinforcement renaissance
First published in Communications of the ACM, Vol. 59, No. 8, July 2016.
The article describes the emerging interest in reinforcement learning (RL) in combination with deep learning, and its application to games and robotics, especially after Deepmind's success on complex games such as Go. It has unique perspectives from different experts in the field and provides a great overview of how they think about the problems, challenges and limitations, and the future work involved to reach human-like capabilities.
[ Learn more ]
Reinforcement learning in Robotics
First published in Medium, July 2016.
Víctor Mayoral Vilches, CTO at Alias Robotics, describes that RL works for complex problems where there is no easily programmable solution e.g. game playing and robotic control. He mentions via examples how similar a robot learning via RL and discovering optimal behavior through trial-and-error is similar to humans' way of learning. Later, challenges about "curse of dimensionality" are being discussed, and how we can use the effective representations, prior knowledge and approximations to make robot RL yield good results in real world scenarios.
[ Learn more ]
“Deep learning is the greatest thing since sliced bread, but it quickly becomes limited by the data. If we can use reinforcement learning to automatically generate data, even if the data is more weakly labeled than having humans go in and label everything, there can be much more of it because we can generate it automatically, so these two together really fit well."
--Richard Sutton, Distinguished Research Scientist at DeepMind
The Ingredients of Real World Robotic Reinforcement Learning
First published in Berkeley Artificial Intelligence Research, April 2020.
Abhishek Gupta, a PhD student in EECS at UC Berkeley, shows a way of training robots directly with RL in the real world as opposed to the typical simulation to real-world transfer paradigm. He describes ways of addressing constraints of the RL in the real world such as "no reset mechanism", "non-availability of a low dimensional Markovian state", "no reward function", and demonstrates successful results on simplistic robotic tasks.
[ Learn more ]
An Introduction to Deep Reinforcement Learning
First published by Now Publishers, 2019. Available on arxiv.
A great resource for people who are further interested to learn more about Deep reinforcement learning.
[ Learn more ]
Applications and Human-Robot Interactions
Healthcare robotics
First published in Communications of the ACM, Vol. 60, No. 11, October 2017.
A fantastic review article on the usage of Robotics in Healthcare by UCSD Professor Laurel D. Riek. The article provides great insights into how healthcare robotics could help the over 20% of the world’s population facing physical, cognitive, or sensory impairments, and fill the gaps in an overburdened healthcare system.
The structure of this article is also a framework for any product design workflow: 1. First, build a profile for the potential beneficiaries of the technology, settings in which it will be used, and tasks it should perform. 2. Create a taxonomy of different types of solutions (in this case healthcare robots), and 3. List the challenges of the field and how to mitigate them. The article is filled with lessons in user-centric design to create technology that is actually beneficial and useful and is a highly recommended read.
[ Learn more ]
A Roadmap for US Robotics: 2020 Edition
First published as the US National Robotics Roadmap by the Computing Community Consortium, September 2020.
A comprehensive report on the state of Robotics applications in the United States from its leading research universities. A thorough coverage of societal drivers, research efforts, technology contexts, and the legal and societal contexts of the field.
[ Learn more ]
A Primer for Conducting Experiments in Human–Robot Interaction
First published in ACM Transactions on Human-Robot Interaction, October 2020.
A comprehensive tutorial on conducting studies with human participants, from a Human-Robot Interaction research perspective. As we start to work with, drive alongside, and be cared for by robots, it is increasingly important to understand Human-Robot interactions. Example studies in improving human-robot collaboration include understanding proper timings for interactions, how to create interactions for multiple people with a robot controlling multiple devices, and how pedestrians interact with autonomous vehicles. Through this tutorial, we hope that young practitioners will gain 1. A critical thinking for determining the methodological rigor, reproducibility, and impact of the studies they read about in academic papers and popular media, and 2. An authoritative and comprehensive toolset for conducting their own research with human participants.
[ Learn more ]
