Dexterous use of hands (with fingers) is one defining characteristic of humans. Replicating dexterity of the human hand would markedly improve efficiency, intuitiveness, and richness of many real-world human-robot interaction (HRI) applications, including (i) robotic hand teleoperation (Fig. 1B), particularly that of anthropomorphic robotic hands (1, 2), where a remote user can fully use their hand and fingers with haptic feedback for complex manipulation tasks, instead of relying on [typically only up to 6 DOFs (degrees of freedom)] conventional haptic devices (3); (ii) collaborative robot interaction (Fig. 1C and Movie 1), where a user can quickly and intuitively provide rich commands and cues to the robot using their hands and fingers, thereby making the interaction safer and smoother compared to the case of conventional pendant programming (4); and (iii) 3D (three-dimensional) drone swarm control (Fig. 1D and Movie 2), where a user in the field can efficiently control the complex 3D swarm behavior by simply nudging their formation or quickly defining 3D virtual walls to avoid dangerous regions. All the tasks mentioned above are difficult when relying on conventional 2D tablet interfaces (5). This use of the hand will also greatly improve the user experience of virtual reality (VR) and augmented reality (AR), which is currently dominated by 6-DOF “fist-based” controllers (6, 7).

DISCUSSION