Pan-Sharpening with Customized Transformer and Invertible Neural Network
man zhou, Jie Huang, Yanchi Fang, Xueyang Fu, Aiping Liu
[AAAI-22] Main Track
Abstract:
Pan-sharpening in remote sensing imaging systems refers to generate a high-resolution multispectral image from a high-resolution panchromatic (PAN) image and its corresponding low-resolution multispectral (MS) image. Over the recent years, convolution neural network (CNN)-based methods have demonstrated superior learning capability and dominated this field. However, due to the limitation of the convolution operator, long-range spatial features are often obtained imprecisely, leading to the degradation of overall performance. To this end, we propose a novel method by exploiting a customized transformer architecture for long-range dependencies modeling, and an information-lossless invertible neural module for effective feature fusion. Specifically, the customized transformer formulates the PAN and MS features as queries and keys to encourage joint feature learning across two modalities while the designed invertible neural module enables effective feature fusion to generate the expected pan-sharpened results. To the best of our knowledge, this is the first attempt to introduce transformer and invertible neural network on pan-sharpening. Extensive experiments over different satellite datasets demonstrate that our method outperforms the state-of-the-art algorithms both visually and quantitatively with fewer parameters and flops. Furthermore, the ablation experiments also prove the effectiveness of the proposed customized long-range transformer and the superiority of invertible neural feature fusion module for pan-sharpening.
Introduction Video
Sessions where this paper appears
-
Poster Session 4
Fri, February 25 5:00 PM - 6:45 PM (+00:00)
Red 3
-
Poster Session 8
Sun, February 27 12:45 AM - 2:30 AM (+00:00)
Red 3