Post-doctoral research visit f/m adapting vision-language and multi-modal large language models

Post-Doctoral Research Visit F/M Adapting vision-language and multi-modal large language models

Le descriptif de l’offre ci-dessous est en Anglais

Type de contrat : CDD

Contrat renouvelable : Oui

Niveau de diplôme exigé : Thèse ou équivalent

Fonction : Post-Doctorant

A propos du centre ou de la direction fonctionnelle

The Inria Grenoble research center groups together almost 600 people in 27 research teams and 8 research support departments.

Staff is present on three campuses in Grenoble, in close collaboration with other research and higher education institutions (University Grenoble Alpes, CNRS, CEA, INRAE, …), but also with key economic players in the area.

Inria Grenoble is active in the fields of high-performance computing, verification and embedded systems, modeling of the environment at multiple levels, and data science and artificial intelligence. The center is a top-level scientific institute with an extensive network of international collaborations in Europe and the rest of the world.

Mission confiée

Tasks will involve producing and disseminating high-quality research. This includes submitting papers to top conferences and journals in the field, and also sharing software associated with such work. Accepted papers should be presented at the conference; such travel will be funded by the research team.

The work will be done in collaboration with the PI and his students as well as potentially other members in the team. Collaborations with external members is also possible.

Principales activités

This postdoctoral position focuses on doing research into Vision-Language Models (VLMs) and Multimodal Large Language Models (MLLMs), in particular to ensure that existing models evolve over time. The first part of this research axis will study the problem of continual learning, i.e., to enable these models to acquire new knowledge over time without suffering from catastrophic forgetting. The second part will target the problem of reinforcement learning (RL) based post-training. The latter part of the postdoc will consider research directions at the intersection of these two topics.

Continual learning: When a pre-trained VLM faces a stream of new tasks or datasets, it suffers from catastrophic forgetting. Unlike standard unimodal models, VLMs experience cross-modal feature drift—where updating the visual or textual encoder breaks thegeometric alignment between the modalities [1]. Thus, it is essential to understand how visual and textual knowledge are forgotten differently in order to develop modality-aware continual learning strategies. Potential research directions include the use of adapters, LoRA techniques to more sophisticated ideas such as semantic-geometry preservation to move past generic regularisation [2], task-specific projection to combine frozen layers with trained cross-modal projection layers [3].

RL-based post-training: Classical supervised fine-tuning, although effective, has its limitations, such as the reliance on manually-annotated examples. One alternative to this paradigm is to use RL to push past the ceiling of human-provided demonstrations, improving step-by-step logic, grounding, and interaction. Potential research directions include reward modelling to ensure faithfulness-aware reward signals [4], sample-efficient RL.

References

[1] Liu, Y., Hong, Q., Huang, L., Gomez-Villa, A., Goswami, D., Liu, X., van de Weijer, J., & Tian, Y.. Continual learning for VLMs: A survey and taxonomy beyond forgetting. arXiv.

[2] He et al.. Continual Learning with Vision-Language Models via Semantic-Geometry Preservation.. arXiv.

[3] Zhou et al.. Learning without forgetting for vision-language models. TPAMI. [4] Zhao et al.. On robustness and chain-of-thought consistency of RL-finetuned VLMs. CVPR workshop.

Compétences

Experience in topics such as domain adaptation, studying distribution shifts would be helpful.

Avantages

1. Subsidized meals
2. Partial reimbursement of public transport costs
3. Leave: 7 weeks of annual leave + 10 extra days off due to RTT (statutory reduction in working hours) + possibility of exceptional leave (sick children, moving home, etc.)
4. Possibility of teleworking (90 days / year) and flexible organization of working hours
5. Professional equipment available (videoconferencing, loan of computer equipment, etc.)
6. Social, cultural and sports events and activities
7. Access to vocational training
8. Complementary health insurance under conditions

Rémunération

2788€ gross salary / month