Integrated system for image restoration and augmentation using deep transformer-based neural networks

Abstract

The paper presents a system for automatic restoration and augmentation of scene images distorted by various types of noise, primarily atmospheric precipitation and other weather-related degradations, based on deep neural network transformer models. Unlike traditional approaches, where the tasks of image restoration and the generation/synthesis of artificial distortions are solved separately, the proposed integrated system treats them as mutually inverse processes and combines them within a single architecture. The complex includes an augmentation module that synthesizes weather and other distortions in images and a restoration module that removes these distortions; both modules are built on transformer models with a modified attention mechanism. The system is trained in a cyclic scheme: the augmentation module produces distorted images that are then restored by the restoration module via an inverse transformation, while loss functions and perceptual error metrics are jointly optimized to ensure the realism of both generated and restored images. Experimental results demonstrate that the proposed approach outperforms many state-of-the-art models in terms of quality metrics such as PSNR, SSIM, FID, and others, while its novelty lies in the simultaneous solution of image augmentation and restoration tasks within a single framework capable of effectively modeling diverse weather conditions and distortions and improving images captured under unstable shooting conditions