Multispectral Object Detection

Defense type
Thesis
Starting date
End date
Location
IRISA Rennes
Room
Metivier
Speaker
Heng Zhang (LACODAM)
Theme
Only using RGB cameras for automatic outdoor scene analysis is challenging when, for example, facing insufficient illumination or adverse weather. To improve the recognition reliability, multispectral systems add additional cameras (e.g. infra-red) and perform object detection from multispectral data. Although multispectral scene analysis with deep learning has be shown to have a great potential, there are still many open research questions and it has not been widely deployed in industrial contexts.
In this thesis, we investigated three main challenges about multispectral object detection:
  • (1) the fast and accurate detection of objects of interest from images;
  • (2) the dynamic and adaptive fusion of information from different modalities;
  • (3) low-cost and low-energy multispectral object detection and the reduction of its manual annotation efforts.

In terms of the first challenge, we first optimize the label assignment of the object detection training with a mutual guidance strategy between the classification and localization tasks; we then realize an efficient compression of object detection models by including the teacher-student prediction disagreements in a feature-based knowledge distillation framework. With regard to the second challenge, three different multispectral feature fusion schemes are proposed to deal with the most difficult fusion cases where different cameras provide contradictory information. For the third challenge, a nouvel modality distillation framework is firstly presented to tackle the hardware and software constraints of current multispectral systems; then a multi-sensor-based active learning strategy is designed to reduce the labelling costs when constructing multispectral datasets.

Composition of the jury
Reviewers:
M. Vincent LePetit, Professor Université de Bordeaux, Researcher ENPC ParisTech
Ms. Tinne Tuytelaars, Professor, ESAT KU Leuven
Members:
M. Patrick Bouthemy, Research Director, INRIA
M. Patrick Perez, Senior Researcher, Valeo AI
M. Jakob Verbeek, Senior Researcher, Facebook
Directors:
Ms Elisa Fromont, Professor, Université de Rennes 1
M. Sébastien Lefèvre, Professor, Université de Bretagne Sud