Yifeng Xu

I am a first-year PhD student in the MMLab@HKU, supervised by Prof. Xihui Liu. Previously, I earned my MS from the VIPL group at ICT in 2026 under the supervision of Prof. Shiguang Shan, and worked closely with Prof. Zhenliang He. Before that, I received my BEng from HITsz in 2023 under the supervision of Prof. Wenjie Pei.

My current research interests lie in generative models and their applications to computer vision, multimodal learning and world models.

Email Google Scholar GitHub Blog

News

  1. CtrLoRA is accepted to ICLR 2025.
  2. HCL is accepted to ICCV 2023.

Research

* Equal contribution, † Corresponding author

Jodi: Unification of Visual Generation and Understanding via Joint Modeling

Yifeng Xu, Zhenliang He, Meina Kan, Shiguang Shan, Xilin Chen

arXiv 2025

Visual generation and understanding are two deeply interconnected aspects of human intelligence, yet they have been traditionally treated as separate tasks in machine learning. In this paper, we propose Jodi, a diffusion framework that unifies visual generation and understanding by jointly modeling the image domain and multiple label domains. Specifically, Jodi is built upon a linear diffusion transformer along with a Role-Switch mechanism, which enables it to perform three particular types of tasks: (1) joint generation, where the model simultaneously generates images and multiple labels; (2) controllable generation, where images are generated conditioned on any combination of labels; and (3) image perception, where multiple labels can be predicted at once from a given image. Furthermore, we present the Joint-1.6M dataset, which contains 200K highquality images collected from public sources, automatic labels for 7 visual domains, and LLM-generated captions. Extensive experiments demonstrate that our Jodi excels in generation tasks and performs competitively in understanding tasks. Besides, Jodi exhibits strong extensibility to new visual domains. Codes, data, and model weights are available at https://github.com/VIPL-GENUN/Jodi. arXiv Code Project BibTeX

CtrLoRA: An Extensible and Efficient Framework for Controllable Image Generation

Yifeng Xu, Zhenliang He, Shiguang Shan, Xilin Chen

ICLR 2025

Recently, large-scale diffusion models have made impressive progress in text-toimage (T2I) generation. To further equip these T2I models with fine-grained spatial control, approaches like ControlNet introduce an extra network that learns to follow a condition image. However, for every single condition type, ControlNet requires independent training on millions of data pairs with hundreds of GPU hours, which is quite expensive and makes it challenging for ordinary users to explore and develop new types of conditions. To address this problem, we propose the CtrLoRA framework, which trains a Base ControlNet to learn the common knowledge of image-to-image generation from multiple base conditions, along with condition-specific LoRAs to capture distinct characteristics of each condition. Utilizing our pretrained Base ControlNet, users can easily adapt it to new conditions, requiring as few as 1,000 data pairs and less than one hour of single-GPU training to obtain satisfactory results in most scenarios. Moreover, our CtrLoRA reduces the learnable parameters by 90% compared to ControlNet, significantly lowering the threshold to distribute and deploy the model weights. Extensive experiments on various types of conditions demonstrate the efficiency and effectiveness of our method. Codes and model weights will be released at https://github.com/xyfJASON/ctrlora. arXiv Code BibTeX

Hierarchical Contrastive Learning for Pattern-Generalizable Image Corruption Detection

Xin Feng*, Yifeng Xu*, Guangming Lu†, Wenjie Pei†

ICCV 2023

Effective image restoration with large-size corruptions, such as blind image inpainting, entails precise detection of corruption region masks which remains extremely challenging due to diverse shapes and patterns of corruptions. In this work, we present a novel method for automatic corruption detection, which allows for blind corruption restoration without known corruption masks. Specifically, we develop a hierarchical contrastive learning framework to detect corrupted regions by capturing the intrinsic semantic distinctions between corrupted and uncorrupted regions. In particular, our model detects the corrupted mask in a coarse-to-fine manner by first predicting a coarse mask by contrastive learning in low-resolution feature space and then refines the uncertain area of the mask by high-resolution contrastive learning. A specialized hierarchical interaction mechanism is designed to facilitate the knowledge propagation of contrastive learning in different scales, boosting the modeling performance substantially. The detected multi-scale corruption masks are then leveraged to guide the corruption restoration. Detecting corrupted regions by learning the contrastive distinctions rather than the semantic patterns of corruptions, our model has well generalization ability across different corruption patterns. Extensive experiments demonstrate following merits of our model: 1) the superior performance over other methods on both corruption detection and various image restoration tasks including blind inpainting and watermark removal, and 2) strong generalization across different corruption patterns such as graffiti, random noise or other image content. Codes and trained weights are available at https://github.com/xyfJASON/HCL. PDF Supp arXiv Code BibTeX

Education

The University of Hong Kong

PhD student

Supervisor: Prof. Xihui Liu

HKSAR, China

Institute of Computing Technology, CAS

MS in Computer Science and Technology

Supervisor: Prof. Shiguang Shan

Beijing, China

Harbin Institute of Technology, Shenzhen

BEng in Computer Science and Technology

Supervisor: Prof. Wenjie Pei

Shenzhen, China

Awards

  1. Silver Medal, The 46th ICPC Asia Regional Contest, Jinan Site
  2. First Prize, China Undergraduate Mathematical Contest in Modeling (CUMCM), Guangdong
  3. First Prize, The 12th Chinese Mathematics Competition (CMC)
  4. Silver Medal, The 2020 China Collegiate Programming Contest (CCPC), Weihai Site

Honors

  1. First-Class Academic Scholarship, Institute of Computing Technology, CAS
  2. First-Class Academic Scholarship, Harbin Institute of Technology, Shenzhen
  3. First-Class Academic Scholarship, Harbin Institute of Technology, Shenzhen
  4. China National Scholarship, Ministry of Education of China