Published in NeurIPS, 2026
We show that projector training in multimodal LLMs primarily performs coarse alignment, and introduce PORTAL, a training-free initialization method that achieves this alignment without paired data or gradient-based optimization.
Published in ICML, 2026
Neural networks, trained with different objectives on different data and modalities, are converging to shared local neighborhood relationships.
Published in bioRxiv, 2026
COSMIC (Cross-modal generation between Single-cell RNA-seq and MICroscopy images) is a bidirectional generative framework that quantitatively links single-cell nuclear morphology and gene expression.
Published in bioRxiv, 2025
IRIS (Interconnected Robotic Imaging and Single-cell transcriptomics) is a deterministic single-cell platform that couples high-resolution microscopy with droplet-based single-cell RNA-seq to capture multimodal images and matched transcriptomes from the same cell, directly linking morphology to gene expression.
Published in NeurIPS, 2025
STRUCTURE is a framework for building multimodal models in low-data regimes by aligning frozen unimodal foundation models, enabling strong performance on zero-shot classification and retrieval tasks. It introduces a simple, plug-and-play regularization technique that preserves the geometric structure of each modality’s latent space and aligns layers with the highest representational similarity across modalities.
Published in ICML, 2024
This work CROW is a cross-domain open-world discovery method that enables automatic assignment of samples to seen classes and discovery of novel classes under a domain shift. CROW introduces a cluster-then-match strategy enabled by a well-structured representation space of foundation models.
Published in AAAI, 2019
This work aims to transfer human action from the source person to the target person using 3D reconstruction.