Codes and dataset for "A Lightweight Recurrent Aggregation Network for Satellite Video Super-Resolution", Journal of Selected Topics in Applied Earth Observations and Remote Sensing (JSTARS), 2023.
Intelligent processing and analysis of satellite video has become one of the research hotspots in the representation of remote sensing, and satellite video super-resolution (SVSR) is an important research direction which can improve the image quality of satellite video. However, existing approaches for SVSR often underutilize a notable advantage inherent to satellite video: the presence of extensive sequential imagery capturing a consistent scene. Presently, the majority of SVSR methods merely harness a limited number of adjacent frames for enhancing the resolution of individual frames, thus resulting in suboptimal information utilization. In response, we introduce the Recurrent Aggregation Network for Satellite Video Super-Resolution (RASVSR). This innovative framework leverages a bidirectional recurrent neural network to propagate extracted features from each frame across the entire video sequence. It relies on an alignment method based on optical flow and deformable convolution (DCN) to realize the alignment of the features, and a Temporal Feature Fusion Module (TFF) to realize effective feature fusion over time. Notably, our research underscores the positive influence of employing lengthier image sequences in SVSR. In the context of RASVSR, with better alignment and fusion, we make the perceptual field of each frame spanning 100 frames of the video, thus acquiring richer information, and information between different images can be complementary. This strategic approach culminates in superior performance compared to alternative methods, as evidenced by a noteworthy 1.15 dB improvement in PSNR, with very few parameters.
SAT-MTB-VSR is a large-scale dataset for satellite video super-resolution made from original videos of Jilin-1, which is a subset of the satellite video multitasking dataset SAT-MTB.It contains 431 videos, each of which is 100 consecutive frames, of which 413 are used as the training set and 18 as the validation set, and all the 18 validation sets are from different original videos, while the size of the images is 640 × 640. These images are downsampled 4× by bicubic interpolation to get 160 × 160 low-resolution images, thus obtaining the LR-HR training pairs.
The dataset is publicly available at zenodo and Baidu Netdisk.
-
Clone the repo
git clone https://github.com/Alioth2000/RASVSR.git
-
Install dependent packages
cd RASVSR pip install -r requirements.txt -
Install BasicSR
Please run the following command in the root path of the project to install BasicSR:python setup.py develop
Download the SAT-MTB-VSR dataset from zenodo or Baidu Netdisk and unzip it to datasets/SAT-MTB-VSR/.
We recommend using the following command to convert the files to lmdb format to speed up training:
python scripts/data_preparation/create_lmdb.pyDownload the pretrained models from Baidu Netdisk and put them in experiments/pretrained_models/SAT-MTB-VSR/.
- Single GPU
python basicsr/train.py -opt options/train_RASVSR.yml - Multiple GPU
CUDA_VISIBLE_DEVICES=0,1 ./scripts/dist_train.sh 2 options/train_RASVSR.yml
- Single GPU
python basicsr/test.py -opt options/test_RASVSR.yml - Multiple GPU
CUDA_VISIBLE_DEVICES=0,1 ./scripts/dist_test.sh 2 options/test_RASVSR.yml
@ARTICLE{10316591,
author={Wang, Han and Li, Shengyang and Zhao, Manqi},
journal={IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing},
title={A Lightweight Recurrent Aggregation Network for Satellite Video Super-Resolution},
year={2024},
volume={17},
number={},
pages={685-695},
doi={10.1109/JSTARS.2023.3332449}}
This work is built upon BasicSR.