Advanced Visual-Language Transformer for Historical newspapers understanding

This thesis is in line with the domain of handwriting recognition and document analysis. The goal is to design an end-to-end model able to understand the inherent structure of documents, combined with text recognition. We focus on historical newspapers for the valuable data they contain. Historical newspapers offer a very interesting study case. 1) This is challenging, as older documents present increased difficulties due to the old language, the style of writing, the deterioration of the document or low-quality digitizations, which often make the task of transcribing a specific document difficult and troublesome, even for an expert. 2) The layout of newspapers is highly codified, with several levels of hierarchy, leading to a complex document structure, but with high variabilities depending on pages or editors and guided by some implicit rules regarding the reading order.

Large Visual-Language Models offers interesting generic capacities to deal with text recognition from images. However, they generally suffer from hallucinations and the performance drops dramatically when dealing with historical documents or non-latin alphabets. 

Traditional systems for dealing with historical documents generally focus either on text recognition task only, or on document layout analysis task only. Text recognition approaches rely on the combination of a text line extraction model, an optical model recognizing characters in images, and a language model providing linguistic corrections [Soullard et al., 2019]. The emergence of end-to-end models gradually replaced these multi-step systems by a unified model performing these different tasks simultaneously [Yousef and Bishop, 2020, Coquenet et al., 2022]. These systems are based on fully convolutional architectures [Yousef and Bishop, 2020] and/or on Transformers [Coquenet et al., 2023], which become more and more popular in many fields, such as handwriting recognition, thanks to the efficiency of the attention mechanisms they include [Barrere et al., 2024, Kang et al., 2022]. When dealing with newspapers, document layout analysis approaches focus on localizing the structural components of the document (title, article, advertising, columns, paragraphs,...), linking them together through membership relations or reading order. Rule-based strategies (Gutehrlé and Atanassova [2022]) have been investigated to consider the complex structure of the document. Deep learning-based approaches are mainly based on fully convolution approach through pixel-level segmentation tasks Soullard et al. [2020], Yang et al. [2017] or object detection strategies Kumar and Lehal [2024]. 

An attempt to combine text recognition and document analysis was proposed in Coquenet et al. [2023] by relying on an autoregressive Transformer. The ground truth mixes layout tokens and character tokens to associate text blocks to some categories (page, page number, annotations, main body). However, the model does not localize the recognized item in the input image and does not take advantage of the structural knowledge.

Thus, we are interested in designing an advanced Transformer-based model, able to localize, recognize and understand the content of historical newspapers. The aim is to leverage from external knowledge about how a newspaper is structured to constrain the model’s behavior, making training easier. One way could be to prevent the Transformer attention to attend to specific parts of the image, because it does not make sense to look at them during the recognition process. Another way could be to propose adaptative loss function for training specific parts of the networks.

This thesis is conducted in connection with the LabCom Synapses in partnership with the Ouest-France company. We are considering publications in high-level conferences or journals covering the Machine Learning domain (e.g. CVPR, NeurIPS, ICML) and the document analysis field (ICDAR, IJDAR).