Multispectral Object Detection

Type de soutenance
Date de début
Date de fin
IRISA Rennes
Heng Zhang (LACODAM)
Only using RGB cameras for automatic outdoor scene analysis is challenging when, for example, facing insufficient illumination or adverse weather. To improve the recognition reliability, multispectral systems add additional cameras (e.g. infra-red) and perform object detection from multispectral data. Although multispectral scene analysis with deep learning has be shown to have a great potential, there are still many open research questions and it has not been widely deployed in industrial contexts.
In this thesis, we investigated three main challenges about multispectral object detection:
  • (1) the fast and accurate detection of objects of interest from images;
  • (2) the dynamic and adaptive fusion of information from different modalities;
  • (3) low-cost and low-energy multispectral object detection and the reduction of its manual annotation efforts.

In terms of the first challenge, we first optimize the label assignment of the object detection training with a mutual guidance strategy between the classification and localization tasks; we then realize an efficient compression of object detection models by including the teacher-student prediction disagreements in a feature-based knowledge distillation framework. With regard to the second challenge, three different multispectral feature fusion schemes are proposed to deal with the most difficult fusion cases where different cameras provide contradictory information. For the third challenge, a nouvel modality distillation framework is firstly presented to tackle the hardware and software constraints of current multispectral systems; then a multi-sensor-based active learning strategy is designed to reduce the labelling costs when constructing multispectral datasets.

Composition du jury
M. Vincent LePetit, Professor Université de Bordeaux, Researcher ENPC ParisTech
Ms. Tinne Tuytelaars, Professor, ESAT KU Leuven
M. Patrick Bouthemy, Research Director, INRIA
M. Patrick Perez, Senior Researcher, Valeo AI
M. Jakob Verbeek, Senior Researcher, Facebook
Ms Elisa Fromont, Professor, Université de Rennes 1
M. Sébastien Lefèvre, Professor, Université de Bretagne Sud