Deep continual learning for satellite image time series

Context
During the last decades, the number and characteristics of the imaging sensors on board the satellites have constantly increased and evolved allowing access (often free of charge) to a large amount of Earth observation data. Recent constellations frequently revisit the same regions at a high spatial resolution. For example, the two Sentinel-2 satellites capture all land surfaces every five days at the equator at a 10-meter spatial resolution at best. The data cubes acquired by these sensors, commonly referred to as satellite image time series (SITS) [8], combine high spectral, spatial, and temporal resolutions, facilitating precise monitoring of landscape dynamics [1].

The automatic transformation of these data cubes into meaningful information (e.g., deforestation maps or land cover land use maps) usually relies on supervised learning techniques. Recent advances in this field have been marked by a shift towards deep learning methods, owing to their state-of-the-art results across various domains, including computer vision and natural language processing. The ability of temporal neural networks to handle sequential data (e.g., text or audio) and to detect time-invariant characteristics results in various achievements for time series classification in several domains [2], including remote sensing [6].

However, models are often trained statically. In other words, either a model is fine-tuned, leading to forgetting the knowledge gained previously, or a new model is trained for each new dataset neglecting the opportunity to leverage insights from prior training instances. For example, the French scientific panel on land cover mapping (CES OSO) produces annual land cover maps by retraining a model every year, overlooking the potential utility of previously trained models. This approach is not only computationally intensive and time-consuming but also suboptimal given the rapid availability of satellite imagery for model updates. A compelling alternative lies in dynamic learning paradigms, wherein models are updated from a data stream, enabling the accumulation of knowledge over time while mitigating the risk of catastrophic forgetting. In the deep learning era, this strategy is known as continual learning [4]. Traditional scenarios view each observation sequentially and process them independently [10], which is an issue for SITS whose temporal structure (e.g., crop growth rate) is crucial to model landscape dynamics.

Objectives
While the formal definition of continual learning is much debated in machine learning and computer vision communities, it is non-existent for SITS. The PhD aims at developing for the first time continual learning techniques adapted to the specificities of SITS data by leveraging both continual learning and SITS analysis research. It will consist of two main objectives: (1) devising and evaluating new robust continual learning paradigms for SITS, and (2) refining the continual learning strategy to discover new classes over time. We aim to demonstrate the potential of continual learning applied to SITS for forest monitoring, especially to help monitor Amazon deforestation and degradation on a large scale.

1. Developing new continual learning algorithm for sequences of satellite images. In this regard, we aim to evaluate existing state-of-the-art techniques and their ability to recall dynamics in SITS (e.g., vegetation growth). Among several ideas, we first plan to study continual learning strategies on temporal neural networks (e.g., Transformers with regularized attention weights) when subtime-series are inputted. This scenario requires studying how catastrophic forgetting will impact temporal neural architectures and potentially how its effect can be mitigated. It also requires to determine the optimal number of past observations to find a trade-off between the precision of the method and memory used to store the data. 

2. Discovering new classes over time without forgetting previous class [3]. The inability of continual learning approaches to discover new classes limits considerably their application in real-world remote sensing settings where land cover changes over time and labels cannot be easily collected. A possible idea is to represent each existing class through prototypes, that
can be extracted for SITS [9], and maintain them over time. New classes could be identified when embedding of newly acquired observations are dissimilar from existing prototypes.

Expected outcomes
The main expected outcome is the implementation for the first time of continual learning algorithms for SITS data, validated in the context of forest monitoring at a large scale. The produced code will be released through open-source code (Python/PyTorch) to allow reproducibility of the results, with possible integration into dedicated toolboxes for machine learning on time series or into existing packages for end-users (e.g., R-SITS). The results will be disseminated through open-access peer-reviewed publications in major international conferences and top journals in machine learning and computer vision (ICML, CVPR, ICLR, Neurips, or ICCV) and remote sensing (ISPRS PHOTO, IEEE TGRS, RSE, or Earth Vision).

Candidate profile and skills
We are looking for a candidate

• with a computer science, (geo)data science, or statistics master degree (or equivalent),
• with strong data analysis, machine learning, and computer vision knowledge,
• who is familiar with deep learning techniques,
• with excellent programming skills in at least one language (C/C++, Python, etc.),
• with good communication skills (at least in English) are required,
• with interest in Earth observation applications.
• Knowledge of time series analysis and remote sensing techniques will be appreciated.

Supervision and Collaboration
The Ph.D. candidate will be supervised by Dr. Charlotte Pelletier (UBS/IRISA) and Prof. Laetitia Chapel (Institut Agro Rennes Angers/IRISA), who are specialized in time series analysis and machine learning for structured data with applications in remote sensing. This PhD position is funded by the ANR JCJC DECOL on deep continual learning for satellite image time series. The PhD thesis will be conducted at Université Bretagne Sud, IRISA lab, in Vannes, France. Interactions with Brazilian researchers from INPE are also envisioned with the possible organisation of a visit stay.

Application
To apply for this position, please send a resume, a motivation letter explaining your understanding and interest in the topic, your Bachelor (Licence) and Master (or equivalent) transcripts, and a reference letter to charlotte [*] pelletieruniv-ubs [*] fr, and laetitia [*] chapelirisa [*] fr.
Application deadline: until the position is filled.