Paper
HEALNET: Multimodal Fusion of Heterogeneous Biomedical Data
Technological advances in medical data collection, such as high-throughput genomic sequencing and digital high-resolution histopathology, have contributed to the rising requirement for multimodal biomedical modelling, specifically for image, tabular and graph data. Most multimodal deep learning approaches use modality-specific architectures that are often trained separately and cannot capture the crucial cross-modal information that motivates the integration of different data sources. This paper presents the Hybrid Early-fusion Attention Learning Network (HEALNet) – a flexible multimodal fusion architecture, which: a) preserves modality-specific structural information, b) captures the cross-modal interactions and structural information in a shared latent space, c) can effectively handle missing modalities during training and inference, and d) enables intuitive model inspection by learning on the raw data input instead of opaque embeddings. We conduct multimodal survival analysis on Whole Slide Images and Multi-omic data on four cancer datasets from The Cancer Genome Atlas (TCGA). HEALNet achieves state-of-the-art performance compared to other end-to-end trained fusion models, substantially improving over unimodal and multimodal baselines whilst being robust in scenarios with missing modalities.
Download: NeurIPS 2024
Format
- Paper shared 1 week ahead for pre‑reading
- 5–10 min summary by a volunteer (no slides — we’ll scroll through the PDF together)
- Open discussion on methodology, strengths/weaknesses, clinical relevance, and future directions
- Papers chosen collaboratively to balance foundational ML research and cutting‑edge medical AI
- Who should attend? Researchers, clinicians, students — anyone curious about ML/AI in medicine.
- How to join? Simply show up (no registration required). Looking forward to seeing you there!
Organizers
Questions or paper suggestions? Contact us via Email.
Postdoc
Institute of Clinical Epidemiology, Public Health, Health Economics, Medical Statistics and Informatics
Faculty Member
Postdoc
Institute of Clinical Epidemiology, Public Health, Health Economics, Medical Statistics and Informatics
Faculty Member
Postdoc
Digital Cardiology Lab, University Clinic of Internal Medicine III - Cardiology and Angiology
Faculty Member
Postdoc
Digital Cardiology Lab, University Clinic of Internal Medicine III - Cardiology and Angiology
Faculty Member