Attention guided In-Hand Mechanical Tools Recognition in Human-Robot Collaborative Process

Guo Wu; Xin Shen; Vladimir Serebrenny

Guo Wu Bauman Moscow State Technical University http://orcid.org/0000-0001-8424-4421
Xin Shen Bauman Moscow State Technical University http://orcid.org/0000-0002-8191-9654
Vladimir Serebrenny Bauman Moscow State Technical University http://orcid.org/0000-0003-1182-2117

Keywords: Human Robot Collaboration, Hand Object Interaction, Attention Mechanism, Occlusion-Aware Object Detection

Abstract

The task of recognition of human behavior in a collaborative robotic system is crucial for the organization of seamless and productive collaboration. We design a vision system for the industrial scenario for riveting a metal plate, and concentrate on the task of recognizing in-hand mechanical tools. However, there is a severe occlusion problem during hand-object interaction process. Incorporating attention modules into the backbone part are often utilized to handle occlusion and enhance the ability of extract features with contextual information. In view of that, three modified occlusion-aware models based on YOLOv5 for in-hand mechanical tools recognition are proposed: by adding SimAM into each of bottleneck network in the backbone part, inserting a Criss-Cross attention layer between the last C3 block and the SPPF block of the back-bone network, and replacing the last C3 block of the backbone network with Criss-Cross attention layer. We create a dataset specifically for our task of in-hand mechanical tools recognition and validate four modified models after training separately, which proves the effectiveness of SimAM module and ineffectiveness of Criss-Cross attention module. The real-time detection is still imperfect under the occlusion of various directions of the hands.