P. Achlioptas, A. Abdelreheem, F. Xia, M. Elhoseiny and L. Guibas, ReferIt3D: Neural Listeners for Fine-Grained 3D Object Identification in Real-World Scenes, European Conference Computer Vision (ECCV), 2020 (Oral).

Abstract:

In this work we study the problem of using referential language to identify common objects in real-world 3D scenes. We focus on a challenging setup where the referred object belongs to a fine-grained object class and the underlying scene contains multiple object instances of that class. Due to the scarcity and unsuitability of existent 3D-oriented linguistic resources for this task, we first develop two large-scale and complementary visio-linguistic datasets: i) Sr3D, which contains 83.5K template-based utterances leveraging spatial relations among fine-grained object classes to localize a referred object in a scene, and ii) Nr3D, which contains 41.5K natural, free-form utterances collected by deploying a 2-player object reference game in 3D scenes. Using utterances of either datasets, human listeners can recognize the referred object with high (>86%, 92% resp.) accuracy. By tapping on this data, we develop novel neural listeners that can comprehend object-centric natural language and identify the referred object directly in a 3D scene. Our key technical contribution is designing an approach for combining linguistic and geometric information (in the form of 3D point clouds) and creating multi-modal (3D) neural listeners. We also show that architectures which promote object-to-object communication via graph neural networks outperform less context-aware alternatives, and that fine-grained object classification is a bottleneck for language-assisted 3D object identification.

Bibtex: 

@article{aaxeg-rnlfgoirws-20,
author={Achlioptas, Panos and Abdelreheem, Ahmed and Xia, Fei and Elhoseiny, Mohamed and Guibas, Leonidas},
title={ReferIt3D: Neural Listeners for Fine-Grained 3D Object Identification in Real-World Scenes},
journal={16th European Conference on Computer Vision (ECCV)},
year={2020}
}