This work introduces a novel approach to physical Human-Robot Interaction (pHRI) by leveraging the joint torque sensors of standard collaborative robots. By mounting a passive, uninstrumented plexiglass touchpad to the robot's flange, we transform the robot into a handwriting-based input interface. Users can communicate complex intentions by drawing characters, words, or full sentences with their finger directly on the touchpad. As the user writes, the system captures raw joint torque signals and transforms them into end-effector forces and moments. These signals are then processed and classified in real-time using a Bidirectional Long Short-Term Memory (Bi-LSTM) neural network.
To ensure high recognition accuracy and system robustness, we established a standardized drawing protocol for 36 character classes (26 letters and 10 digits). This protocol guides users on stroke direction and the number of strokes per character, as demonstrated in the video below.
We collected two comprehensive datasets to train and evaluate the Bi-LSTM neural network (available via the links at the top of this page):
As illustrated the figure below, the wrench profiles captured at the flange remain remarkably consistent while different individuals write the same letter. While the signal amplitude varies based on the pressure applied by the user, the underlying patterns remain stable, which is key to the system's high generalization capability.
To classify the complex time-series data generated during handwriting, we utilize a Bidirectional Long Short-Term Memory (Bi-LSTM) network. This architecture is particularly effective at capturing the spatio-temporal relationships within the wrench signals by processing the sequences in both forward and backward directions. The system achieves a high online classification accuracy of approximately 95% for users who contributed to the training set and maintains a robust 92% accuracy for completely new users. As shown in the confusion matrix below, misclassifications primarily occur between dynamically similar characters, such as 'E' and 'F' or 'D' and 'P'.
Even with high-accuracy classification models, potential misclassifications can still occur. To address this, our system employs a Hierarchical Finite State Machine (HFSM) to manage task execution and ensure safe interaction. The system continuously records signals, classifies individual characters, and forms full words or sentences. The recorder string is firstly corrected using a spelling corrector and is then fed into an LLM-powered layer that interprets the user's intent. This allows for significant flexibility in communication; for example, phrases like "Apple," "I want an apple," or "I would like an apple" are all translated into the same underlying action code by the robot. To ensure safety during execution, the robot audibly confirms the recognized command and requests a physical confirmation via a tap on the robot’s arm. This multimodal approach combines the naturalness of handwriting with voice and touch feedback, adding a critical layer of safety and user control.
To demonstrate the practicality and intuitiveness of our system, we implemented a real-world breakfast preparation task. In this scenario, users write their desired food or drink items on the touchpad, and the robot identifies, retrieves, and delivers them. This showcase highlights the potential of intrinsic tactile sensing to enable accessible and natural human-robot collaboration in everyday environments.
