Attention-based models for time series classification

More and more data is monitored 24h a day in many different domains including business, medicine, ecology, etc., leading to an incredible amount of temporal data available. Numerous research works on the domain temporal data analysis and learning have been proposed these last years.

Amongst these works, time series classification (TSC) has received a lot of attention. Many different techniques for TSC have recently emerged. A detailed review of the main TSC techniques can be found in [1].

Inspired by the success of convolutional neural networks for image and video analysis, some accurate models for TSC have been designed using neural networks, as detailed in the review [2].

In [3], it is shown that the importance of localization information is really dependent on the dataset. It seems hence essential to design models capable of learning when and how to use localization information for TSC.

Most of state-of-the-art models are not specifically designed to focus on the importance of the localization information, as they either do not take it specifically into account, or just partially (only absolute localization of patterns and not differential one, or the other way around), or they don’t have the ability to learn which kind of localization information is important for every particular dataset.

The Attention mechanism [4] aims at giving a model the ability to put the focus on certain parts of the data to quantify and learn the temporal dependance between the input and output of the model.

Attention-based models led to great improvement in domains such as natural language processing, audio and video analysis in which the temporal information is at stake.

Transformer models (that include the attention mechanism) are state of the art models for many tasks using such kind of data.

The objective of this PhD is to design attention-based models for time series classification.

The first step in this PhD will be to examine how neural-network-based models for TSC can be extended with the attention mechanism to have a performance baseline.

Then, more advanced structures will be explored inspired from Transformer models and pay particular attention to positional encodings and the information they convey when time series are at stake.

Another main objective of this PhD will be to tackle the issue of the quantity of data. Indeed, such models might need huge amount of data to be accurately learned, which is not always the case with classical TSC benchmarks. We will hence study how to cope with this issue by adopting for instance techniques based on distillation, as used in [5] for image classification tasks.