MatteFormer

MatteFormer: Transformer-Based Image Matting via Prior-Tokens

https://arxiv.org/abs/2203.15662

https://github.com/webtoon/matteformer

29 March, 2022

CVPR2022

Authors: GyuTae Park, SungJoon Son, JaeYoung Yoo, SeHo Kim, Nojun Kwak

Abstract: In this paper, we propose a transformer-based image matting model called MatteFormer, which takes full advantage of trimap information in the transformer block. Our method first introduces a prior-token which is a global representation of each trimap region (e.g. foreground, background and unknown). These prior-tokens are used as global priors and participate in the self-attention mechanism of each block. Each stage of the encoder is composed of PAST (Prior-Attentive Swin Transformer) block, which is based on the Swin Transformer block, but differs in a couple of aspects: 1) It has PA-WSA (Prior-Attentive Window Self-Attention) layer, performing self-attention not only with spatial-tokens but also with prior-tokens. 2) It has prior-memory which saves prior-tokens accumulatively from the previous blocks and transfers them to the next block. We evaluate our MatteFormer on the commonly used image matting datasets: Composition-1k and Distinctions-646. Experiment results show that our proposed method achieves state-of-the-art performance with a large margin. Our codes are available at https://github.com/webtoon/matteformer.

文摘：本文提出了一种基于Transformer的图像抠图模型MatteFormer，该模型充分利用了Transformer块中的trimap信息。我们的方法首先引入一个先验标记，它是每个trimap区域（例如前景、背景和未知）的全局表示。这些先验标记被用作全局先验标记，并参与每个块的自注意机制。编码器的每一级都由过去（先前注意的Swin Transformer）块组成，该块基于Swin Transformer块，但在两个方面有所不同：1）它有PA-WSA（先前注意的窗口自注意）层，不仅使用空间标记执行自注意，还使用先前标记执行自注意。2） 它具有先验内存，该内存累积地保存先前块中的先前令牌，并将它们传输到下一个块。我们在常用的图像抠图数据集：Composition-1k和Differentions-646上评估MatteFormer。实验结果表明，我们提出的方法在很大程度上实现了最先进的性能。我们的代码位于https://github.com/webtoon/matteformer.