CorrFill: Enhancing Faithfulness in Reference-based Inpainting with Correspondence Guidance in Diffusion Models

Kuan-Hung Liu1, Cheng-Kun Yang2, Min-Hung Chen3, Yu-Lun Liu1, Yen-Yu Lin1,
1National Yang Ming Chiao Tung University, 2National Taiwan University 3NVIDIA
Paper Supplementary arXiv Code

WACV 2025

teaser

Abstract

In the task of reference-based image inpainting, an additional reference image is provided to restore a damaged target image to its original state. The advancement of diffusion models, particularly Stable Diffusion, allows for simple formulations in this task. However, existing diffusion-based methods often lack explicit constraints on the correlation between the reference and damaged images, resulting in lower faithfulness to the reference images in the inpainting results.

In this work, we propose CorrFill, a training-free module designed to enhance the awareness of geometric correlations between the reference and target images. This enhancement is achieved by guiding the inpainting process with correspondence constraints estimated during inpainting, utilizing attention masking in self-attention layers and an objective function to update the input tensor according to the constraints.

Experimental results demonstrate that CorrFill significantly enhances the performance of multiple baseline diffusion-based methods, including state-of-the-art approaches, by emphasizing faithfulness to the reference images.

Pipeline

pipeline

CorrFill Inpainting Pipeline. The inpainting process of Stable Diffusion involves iterations of progressive denoising. During each denoising step, CorrFill estimates the correspondence between the target and reference images and guides the subsequent denoising step accordingly. This approach ensures that the denoising process and correspondence estimation converge towards a pattern that remains faithful to the reference image.

Results

result

Results Comparisons. The comparisons between four baselines and their CorrFill-enhanced counterparts on two datasets. Problematic regions in the results of the baseline methods effectively addressed by CorrFill are highlighted in red boxes.

BibTeX

@inproceedings{liu2025corrfill,
      title={CorrFill: Enhancing Faithfulness in Reference-based Inpainting with Correspondence Guidance in Diffusion Models},
      author={Liu, Kuan-Hung and Yang, Cheng-Kun and Chen, Min-Hung and Liu, Yu-Lun and Lin, Yen-Yu},
      booktitle={WACV},
      year={2025}
    }