neural networks for low level image processing

Until recently, machine learning (ML) or neural networks (NN) are mainly used in high level vision tasks, such as image segmentation, object recognition and detection. Low level image processing such as denoising, demosaicing, white balance still mainly rely on signal processing based methods which uses expert designed filters. There are usually a long list of filters in the whole processing pipeline which is run on a dedicated ISP chip. In the past one or two years, there are two new trends. One trend is that more and more researchers propose to apply NN for low level image processing and achieved fascinating performance in term of image quality and processing speed. The other trend is that neural network chip becomes more and more popular at various mobile platform, such as the latest Apple A11 Bionic chip and Huawei Kirin 970 chip. I believe in the near further, NN based methods will play important roles at some low level image processing tasks also some ISP chip may include some NN computing units.

This is a personal collection of works using neural networks for low level image processing. The list will be regularly updated. You are welcome to contribute. Papers of significance are marked in bold. My comments are marked in italic.

For a up-to-date list, please follow this github repository.

Table of Contents

• Review and comments
• Color constancy
• Denoising
• Demosaicing
• Superresolution
• Automatic adjustment
• Artefacts removal
• Pipeline
• Image quality evaluation
• Others

Review and comments

• Deep, Deep Trouble
  • This is an interesting comments from Michael Elad about the impact of Deep Learning on image processing, mathematics and humanity.
• Deep learning for image/video processing
  • Dr. Huang Yu’s summary of DL on image/video processing.

Color constancy

• Deep Specialized Network for Illuminant Estimation (ECCV, 2016, CUHK)
• Single and Multiple Illuminant Estimation Using Convolutional Neural Networks (TIP, 2017, Italy)
  • A three-stage method for illuminant estimation from RAW images
• Recurrent Color Constancy (ICCV, 2017, TUT)
  • Recurrent network for temporal color constancy using multiple frames instead of single frame
• FC4: Fully Convolutional Color Constancy with Confidence-weighted Pooling (CVPR, 2017, THU)
  • CNN based method using image patches
• Natural Image Illuminant Estimation via Deep Non-negative Matrix Factorisation (IET-IP, 2017, OceanUofChina)

Denoising

• Image denoising with multi-layer perceptrons (arXiv, 2012)
  • This is the very first paper using NN to image denoising tasks.
• Can a Single Image Denoising Neural Network Handle All Levels of Gaussian Noise? (SPL, 2014, France)
  • This paper proposal a way to apply NN on image denoising with different noise level of Gaussian noise.
• Deep convolutional architecture for natural image denoising (WCSP, 2015, ZJU)
• Learning Deep CNN Denoiser Prior for Image Restoration (arXiv, 2017, HKPolyU)
• Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising (TIP, 2017, HKPolyU)
• FFDNet: Toward a Fast and Flexible Solution for CNN based Image Denoising (arXiv, 2017, HKPolyU)
• Image Restoration: From Sparse and Low-Rank Priors to Deep Priors (IPM, 2017, HKPolyU)
  • This is a review paper.
• Dilated Deep Residual Network for Image Denoising (arXiv, 2017, SIU)
• IDEAL: Image DEnoising AcceLerator (ACM, 2017, UToronto&Algolux)
  • A NN based approximations of BM3D on an NN accelerator

Demosaicing

• Demosaicing using artificial neural networks (SPIE, 2000)
• A multilayer neural network for image demosaicking (ICIP, 2012)
• Deep Joint Demosaicking and Denoising (Siggraph, 2016, MIT&Adobe)
  • This paper propose to use more difficult patches for the training.
  • Data and code are available on Github.

Automatic adjustment

From the publications, we can find that Adobe has done a lot of work pushing the usage of machine learning in low-level image processing especially automatic photo adjustment.

• Learning Photographic Global Tonal Adjustment with a Database of Input / Output Image Pairs (CVPR, 2011, MIT&Adobe)
  • Adjustment personalization
• Automatic Photo Adjustment Using Deep Neural Networks (ACM, 2016, UIUC&Adobe&Microsoft)
  • This technique is more accurate and supports local edits.

Superresolution

Super-resolution is one of the areas that NN has been applied extensively and achieved great success.

• Deep Networks for Image Super-Resolution with Sparse Prior (ICCV, 2015, UIUC)
• Image superresolution using deep convolutional networks (TPAMI, 2015, HKUST)
• Accurate image super-resolution using very deep convolutional networks (CVPR, 2016, Seoul Univ.)
• Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network (CVPR, 2016, ICL)
  • An order of magnitude faster than previous CNN-based methods
• Image Super-Resolution via Deep Recursive Residual Network (CVPR, 2017, MSU)
• Photo-realistic single image super-resolution using a generative adversarial network (CVPR, 2017, Twitter)

Artefacts removal

• Deep Generative Adversarial Compression Artifact Removal (arXiv, 2017, UoFlorence)
• Real-time Deep Video Deinterlacing (arXiv, 2017, CUHK)

Pipeline

• Learning the image processing pipeline (arXiv, 2016, Stanford)
  • Propose to learn the filter parameters by ML
• Learning Adaptive Parameter Tuning for Image Processing (arXiv, 2016, UCLA&Nvidia)

Image quality evaluation

• Deep Learning for Blind Image Quality Assessment (2017, Unimib)

Others

• Fast Image Processing with Fully-Convolutional Networks (ICCV, 2017, Intel)
  • Operator approximation to accelerate image processing tasks
• Learning Proximal Operators:Using Denoising Networks for Regularizing Inverse Imaging Problems (arXiv, 2017, TUM)
  • This paper proposes a way to use NN for the the general fidelity plus regularization optimization problem which may find applications in many problems.