Presentation of thesis results

Compared with ordinary RGB imaging system, hyperspectral has high dimensional features, information redundancy, significant uncertainty, small samples, and space-spectrum unity, etc., and it faces great challenges for data processing. In recent years, with the emergence of new techniques of deep learning, hyperspectral images based on deep learning have been fully developed in the fields of dimensionality reduction, hybrid image element decomposition and classification.

The School of Marine Science and Technology, Northwestern Polytechnical University used Jiangsu Shuangli HePu's GaiaField-V10 hyperspectral imager with Gaia-Sorter hyperspectral sorter to conduct the following three aspects of research.

I. The School of Marine Science and Technology, Northwestern Polytechnical University collected 2048 hyperspectral images with hyperspectral cameras, in which 6659 targets (3.19 targets per image on average) were identified, especially since the images contained many real objects and objects displayed on tablet screens. It is possible to carry out the test of distinguishing the authenticity of the targets. And based on this database, an object-level hyperspectral image detection method based on SSD deep learning target detection framework is proposed. In addition, the corresponding strategies of data augmentation, network initialization, and migration learning of the existing RGB detection network are proposed for the high dimensionality of hyperspectral data and the difficulty of data acquisition.

The figure below shows the results of data acquisition and analysis work using the GaiaField-V10 hyperspectral imaging system, and it can be seen from Figure 1 that the information contained in the hyperspectral is superior to that of the RGB imaging system.

Figure 1 Data capture scenes and data examples

Two typical category discrimination results (false color images) on the hyperspectral dataset. The left image was taken on the LED screen and the right image was taken in a real scene

The target-based hyperspectral detection problem was solved and the corresponding convolutional neural network was designed according to its characteristics. In addition, a hyperspectral dataset containing more than 400 high-quality images was created for target detection, as showed in Figure 2. Experimental results test the effectiveness of the framework and show its superior performance.

Figure 2 Demonstration of two typical detection results on the hyperspectral dataset

II. This research center proposes a hyperspectral de-shadowing method based on cyclic consistent adversarial network using GaiaField-V10 hyperspectral imaging system from Jiangsu Shuangli Hepu. The method does not need to calibrate the range of the shadowed region or find the paired spectra of uniform substances, but only needs to provide some random sample sampling in the shadowed and unshaded regions to achieve excellent de-shadowing effect, which significantly improves the practicality of the de-shadowing algorithm, and the results are shown in Fig. 3.

Figure 3 On the effect of the method on the de-clouding of Houston data

In addition, the experiments were conducted with real calibration data acquisition, which contains images of uniform scenes with and without light occlusion, allowing for de-shadowing tests and quantitative evaluation of the de-shadowing effect using shade-free images, Figure 4 shows the experimental results of the paper.

Figure 4 Data collected for the de-shadowing study using GaiaField-V10

III. Hybrid image element unmixing is one of the important problems urgently needed in hyperspectral image information processing, aiming to decompose the image element spectra in hyperspectral data into the fundamental component spectra (end elements) and the corresponding component ratios (abundance). This research center has discussed many specific nonlinear mixing models using GaiaField-V10 hyperspectral imager combined with Gaia-Sorter, and established the relationship of linear mixing + nonlinear weighted perturbation generic model constructed based on structured self-encoder, and researched a nonlinear unmixing method based on depth self-encoder, and the method architecture is shown in Figure 5. And a 3D-CNN is used to obtain the vectorial spatial information about the image in the decoder part.

Figure 5 Structured self-encoder structure for nonlinear unmixing

This research center proposes the unmixing for the linear mixing + post-nonlinear model through the further exploration and employs the LSTM-DNN structure in the decoder to mine the correlation information of the spectrum. The scheme is general and effective compared to the state-of-the-art methods and is validated by creating colored quartz sand particle mixing data in the laboratory, and the validation results are shown in Figure 6.

Figure 6 Mixing and imaging in known proportions using colored quartz sand particles for unmixing tests on hyperspectral

The above proposed method of using a nonlinear model to represent the relationship between end elements and abundance in this image element effectively and efficiently overcomes the difficulty that the linear model cannot accurately represent the relationship between end elements and abundance in the mixed image element.

References

[1] Yan, Longbin & Zhao, Min & Wang, Xiuheng & Zhang, Yuge & Chen, Jie. (2021). Object Detection in Hyperspectral Images. IEEE Signal Processing Letters. PP. 1-1. 10.1109/LSP.2021.3059204.

[2] M. Zhao, M. Wang, J. Chen and S. Rahardja, "Hyperspectral Unmixing for Additive Nonlinear Models With a 3-D-CNN Autoencoder Network," in IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1-15, 2022, Art no. 5509415, doi: 10.1109/TGRS.2021.3098745.

[3] M. Zhao, L. Yan and J. Chen, "LSTM-DNN Based Autoencoder Network for Nonlinear Hyperspectral Image Unmixing," in IEEE Journal of Selected Topics in Signal Processing, vol. 15, no. 2, pp. 295-309, Feb. 2021, doi: 10.1109/JSTSP.2021.3052361.

[4] Zhao, M. ,  Yan, L. , &  Chen, J. . (2021). Hyperspectral image shadow compensation via cycle-consistent adversarial networks. Neurocomputing, 450(1). 10.1016/j.neucom.2021.04.017