01
Multimodal Machine Learning and Vision-Language Models
How can models learn jointly from images, volumes, speech and text, and ground what they generate in what they perceive?
Most of the data that matters in the real world does not arrive as a clean 2D image or a short string of text. I am interested in models that combine modalities, including volumetric and spectral data, audio and language, and in generation methods whose outputs can be checked against their inputs rather than merely sounding plausible. That includes the practical side: how such systems behave when they run in real time and interact with people.
- Themes
- Vision–language and speech–language models
- Cross-modal representation learning
- Multimodal generation and evaluation
- Embodied agents that perceive, reason and act
- In my work
- Radiology report generation from 3D chest CT (Springer LNICST, PAAISS 2025), and real-time speech–language systems for an embodied humanoid at IQAI.