Ⅰ. Introduction

Considering the geographical characteristics of South Korea, surrounded by the sea on three sides, coastal surveillance plays a critical role in national security. However,incidents such as the North Korean fishing boat’s entry into Samcheok Port and the Chinese small boat’s illegal entry in Taean, Chungnam, highlight the ever present risk of human error in surveillance. If these issues are not addressed, failures in coastal surveillance operations could pose serious threats to national security^[1]. Additionally, the aging of current military equipment has made coastal surveillance operations more difficult. Approximately 70% of the Army’s coastal surveillance radar and 60% of TOD systems have exceeded their lifespan, making it harder to detect infiltrations. Furthermore, with a declining population due to low birth rates and short- ened military service periods, the shortage of person- nel for surveillance operations is expected to worsen in the future. In this context, the introduction of artifi- cial intelligence(AI) and Big data technologies to the military could provide solutions to these challenges. This study focuses on the automatic identification of unidentified objects along the coast. Building on exist- ing research, this study integrates several specific technologies to enhance the accuracy and efficiency of detection. First, the Dehaze algorithm is used to remove background noise caused by sea fog, improv- ing image clarity. Next, the water segmentation and refinement maritime obstacle detection Network(WaSR) algorithm is employed to segment the background by separating the sea, sky, and land based on the horizon. Using this segmented data, the object detection algorithm You only look once (YOLO) is applied to detect objects specifically in the sea region. Previously, sea fog removal and object de- tection models were developed independently, but this study combines these approaches to perform object detection on clearer, noise reduced images. By com- paring this approach with previous studies, it is ex- pected to offer a more efficient and accurate identi- fication of unidentified objects. This will help provide solutions to the aforementioned issues, including hu- man error, reduced military personnel, and aging mili- tary equipment.

Ⅱ. References

With the rise of the Fourth Industrial Revolution, many tasks previously performed by humans are now being replaced by AI and Big data technologies. The military is also preparing plans to develop an ad- vanced, cutting edge force by integrating the latest technologies. This can provide solutions to the afore- mentioned problems and help strengthen future na- tional defense capabilities. Thus, the adoption of AI technology can automatically detect and swiftly re- spond to potential threats such as infiltration through coastal areas or ab- normal behaviors of ships. This paper will discuss measures for automating coastal surveillance operations and will explain three essential algorithms required for the automation of coastal op- erations an automatic object detection algorithm, a background separation algorithm, and a fog removal algorithm.

2.1 Object Detection

As Vision AI technology advances, numerous ob- ject detection techniques have emerged, and these technologies are currently being utilized in various fields, such as autonomous vehicles, healthcare, se- curity and surveillance, agriculture, and manufacturing. Based on these technologies, there is a need for a system that can identify ships, floating objects, and submerged objects on the sea and provide early warnings in the event of unidentified underwater objects that have not been trained. Major examples of object detection algorithms that can be mentioned include Faster R-CNN^[2], Yolo^[3], and Single shot mul- tibox detector(SSD)^[4]. Faster R-CNN, one of the ob- ject detection technologies, is an improved model of the previous Fast-RCNN^[5], employing a 2-stage ap- proach for inference. Since it independently handles object proposal regions and bounding boxes, it has a relatively slower inference speed but offers the ad- vantage of high accuracy. Another model, Yolo, uses a 1-stage approach, predicting the object type and lo- cation simultaneously, which results in faster in- ference speeds. This makes Yolo highly suitable for real time processing, unlike traditional object de- tection systems, and it is widely used in various fields for various applications. Furthermore, Yolo continues to receive version updates, with improvements ad- dressing the accuracy issues found in earlier versions, making it one of the most popular object detection models today.

2.2 Background Subtraction

In order to efficiently carry out coastal surveillance operations, it is necessary to distinguish and separate the background elements such as the sky, sea, and land. If detection is performed without separating these elements, objects in the sky and on land may also be indiscriminately recognized, making back- ground separation a critical measure. Examples of background separation methods include the BackgroundSubtractor approach^[6] used in OpenCV and the Semantic Segmentation method. The back- ground removal algorithms provided by OpenCV con- sist of BackgroundSubtractorMOG, Background SubtractorMOG2, and BackgroundSubtractorGMG, BackgroundSubtractorMOG is a Gaussian mixtur- ebased background separation algorithm.This method removes the background by applying a mixture of Gaussiandistributions (with K values of 3 or 5) to the background pixels. The weights of the mixture com- ponentsaredeterminedbytheamountoftimeapar- ticular pixel remains in the same location in the video for background removal^[7]. BackgroundSubtractor GMG is a model that combines statistical background image removal with pixel level Bayesian segmentation. This algorithm uses the first few frames (usually120) for background modeling and employs Bayesian inference to detect moving objects or fore- ground elements instead of the background.

Fig. 1.

An Example of Background Removal Using BackgroudSubtractorMOG.

Fig. 2.

An Exampleof Background RemovalUsingBackgroud SubtractorGMG.

Looking at background separation methods using segmentation techniques, this approach has become increasingly popular in recent background separation technology. Segmentation based background separa- tion allows the removal or composition of specific parts of segmented objects, and the U-net^[8] algorithm is widely used for this purpose. Recently, various models such as Facebook’s Segment Anything(SAM)^[9] have also been developed. Among these, the WaSR^[10] model is lighter compared to other segmentation models and, especially when consider- ing the maritime environment, demonstrated com- petitive performance compared to other models. Most notably, it achieved clear separation of obstacles such as the sea and sky. The WaSR model benefits from employing the ResNet 101 backbone network, which provides high accuracy in background separation

Fig. 3.

WaSR Frame Work.

2.3 Dehaze

Sea fog refers to the fog that forms near the surface of the sea. The low visibility caused by sea fog can reduce the visibility range, making fog minimization a crucial factor in coastal surveillance operations. The algorithms used for sea fog removal include DCP (Dark Channel Prior)^[11] and EDN-GTM^[12]. DCP, atraditional fog removal method,works by identifying areas in a foggy image where the intensity of the mini- mum channel among the R, G, and B channels sig- nificantly decreases. These areas are defined as the Dark Channel, and the pixel values from this channel are applied to the entire image. This process results in an image composed only of the channel with the lowest luminance. From this image, the darkness level of specific pixels is extracted and defined as the target luminance, and the boundary value of this luminance is estimated. The target luminance is set as a top per- centile, typically between 0.1 percent and 1 percent For pixels with luminance below the target boundary, a binary mask is generated to highlight the dark regions. As a result, the dark regions are marked as 1, and the remaining regions are marked as 0. From these marked regions, the target luminance is reesti- mated, and the input image is then restored

In the case of the EDN-GTM model, it utilizes the U-net network as its core backbone for image segmentation. This model also adopts the regions where the RGB values decrease as the Dark Channel, which is then input into the U-net network. The U-net structure used in EDN-GTM incorporates mod- ifications such as adding spatial pyramid pooling and using the Swish activation function to enhance fog removal performance. Additionally, since the model is trained directly on foggy data, it shows superior performance in fog removal and low visibility im- provement compared to traditional models. The results indicate a significant performance improvement of up to 4.73% in terms of mAP measurements, and it has good compatibility with the Yolo model, making it applicable to real time video processing

Fig. 4.

An image processed for fog removal using the DCP algorithm.

Fig. 5.

Detection performance on blurred and dehazed images (left in each pair: blurred, right: dehazed, red: ground truth, green: detection).

Ⅲ. Research Methodology

By combining the aforementioned technologies, a system will be developed for identifying unidentified underwater objects along the coast. The algorithm model is structured in the following order Dehaze model for noise (sea fog) removal, Background Subtraction model for background separation, and Object Detection model for automatic object detection. Let me know if you need any further adjust- ments

3.1 Select Dehaze model

For sea fog removal, the EDN-GTM model was used as the Dehaze model. This model showed the best performance during inference with a pretrained model and has been previously combined with real time inference models like Yolo, making it the most efficient option for this experiment. In scientific sur- veillance operations, when the coastal environment’ s CCTV system passes through this model, it detects foggy areas and minimizes noise, producing clearer images. The resulting images are then fed into the next task, the object detection algorithm

3.2 Select Object Detection model

Is then processed by the object detection algorithm. The most suitable object detection model for this task is the Yolo model. Since Yolo operates using a 1-stage approach, it offers the advantage of exceptional in- ference speed. This makes it highly specialized for processing real time maritime footage received via CCTV, which is crucial for scenarios requiring rapid responses, such as maritime detection. Furthermore, Yolo continues to evolve with new versions, sig- nificantly improving the accuracy previously a limi- tation of earlier models and adding new features with each update. Considering these advantages, the Yolo model is used for object detection. As the Yolo model requires images to be in tensor format for inference, the images should be converted into tensor format af- ter passing through the sea fog removal task before being processed by the Yolo model for the second task, object detection

3.3 Select Background Subtraction model

The next step is to separate the background into sea, sky, and obstacles to improve the detection accu- racy of unidentified objects. Recent studies have pro- posed the WaSR approach as an effective algorithm for identifying unidentified underwater objects in ma- rine environments^[13]. The WaSR architecture consists of two main components: an encoder and a decoder. The encoder is responsible for extracting features and contextual information from the input image, utilizing the depth of ResNet-101 to extract rich and distinct features. The decoder reconstructs the segmented im- age from the encoded features, ensuring spatial reso- lution is maintained and the boundaries of the seg- mented regions are clear and well defined. As shown in Fig3, the WaSR network’s overall structure illus- trates the interaction between the encoder, which cap- tures the scene’s semantics, and the decoder, which refines this under standing into a detailed segmentation map. Here, the encoder handles the semantic under- standing of the scene, while the decoder translates this into a segmentation map that clearly distinguishes be- tween the sea, sky, and obstacles. Because WaSR uses ResNet-101 as its backbone network, it offers the ad- vantage of high accuracy in background separation. Additionally, it is well suited for handling challenges in marine environments, such as water reflections and various weather conditions, as it includes a function to adjust brightness preprocessing. Therefore, the WaSR algorithm is deemed highly suitable for back- ground separation. The application of WaSR in this paper is crucial, as its high accuracy in segmentation ensures optimal performance for the subsequent de- tection algorithms, making the WaSR model the pre- ferred choice. Since WaSR is capable of adjusting brightness preprocessing in response to various marine conditions like water reflections and weather, it was selected as the background separation algorithm. The background separation algorithm is applied simulta- neously when receiving the dehazed image, rather than after object detection, so it is more accurate to refer to it as the second task. Therefore, the dehazed image undergoes object detection and background separation simultaneously, and the combined results are outputted.

3.4 Coastal surveillance system Architecture

Based on CCTV images input from the coastal en- vironment, a framework is constructed by combining the aforementioned technologies to remove sea fog and identify objects floating on the sea. First, after receiving the maritime image, noise elements such as sea fog are removed from the image. Then, the image undergoes object detection and background separation algorithms, during which only objects on the sea are inferred using the Yolo algorithm.

Fig. 6.

Model Frame work.

3.5 Train dataset

The dataset used for training consists of MaSTr1478, a representative set of images used for maritime object detection, and 24,000 images from the AIhub marine open dataset. Among these, approx- imately 10,000 images were processed using the Haze Image Synthesis^[14] algorithm to create foggy images for training. The foggy images generated through Haze Image Synthesis will be used for training the Dehaze algorithm, EDM-GTM, while the remaining data will be used for training WaSR and Yolov7.

As shown in Fig7, this dataset is divided into three natural regions for each image: sky, horizon, and sea, making it a suitable dataset for detecting obstacles in the sea area. To use this dataset for object detection, it was categorized into 10 classes: fishing boat, naval vessel, submarine, rubber boat, person, vehicle, heli- copter, airplane, animal, andbuoy. Among these, any objects not trained, along with buoys, will be assumed to be unidentified underwater objects

Fig. 7.

Model Frame work.

Ⅳ. Experiment

4.1 Experimental environment

The PC environment used for model training is Red Hat Enterprise Linux Server Release 7.9. The PC specifications include an Intel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz, and the training was con- ducted using two NVIDIA A40 graphics cards.

4.2 Experimental Results

The purpose of this study is to develop a deep learning based approach to improve object detection in marine environments, and experiments were con- ducted using the framework implemented according to the previously mentioned research methodology. In the first experiment, inference was performed using the pretrained models of EDN-GTM, Yolov7^[15], and WaSR, and the results are as follows

Table 1.

Sample Original Images

Table 2.

Detection Results

The results showed that although object detection for unidentified objects on the sea was performed, there were frequent issues where reflected images on the water were mistakenly identified as unidentified underwater objects. Additionally, the sea fog removal step in the noise reduction process did not produce satisfactory results. The resolution in the noisy areas slightly changed, but the noise was not properly removed. This issue is attributed to the use of a pre- trained model instead of a model trained directly with the relevant data and weights. To address this, a re- experiment was conducted using a model trained with custom data. The dataset used for training included around 10,000 images with added fog effects from the Haze Image Synthesis Dataset. When the results were rechecked using the weight files obtained from this trained model, the output was as follows

The inference results, as shown in Table 4, indicate that the model trained with images containing sea fog was able to handle foggy areas more flexibly com- pared to the model used in the initial experiment. Additionally, the sea fog removal effect contributed to an increase in the detection rate of unidentified objects. The next experiment will compare object de- tection rates before and after sea fog removal.

As seen in Table 5 and Table 6, the object detection accuracy is higher in the images where sea fog was removed compared to the results from images without fog removal. Next, the inference speed will be measured. Since inference on a GPU environment was not possible, the speed comparison was conducted us- ing CPU inference.

Table 3.

Example Images from the Haze Dataset Used for Inference

Table 4.

Validation Results

Table 5.

Validation Results (Before Haze Removal)

Table 6.

Validation Results (After Haze Removal)

4.3 Differences in Inference by Model

The object detection models used in the experiment were Yolov5^[16] and Yolov7. Looking at Yolov5’s backbone, it uses the CSP-Darknet^[17] network for fea- ture map extraction, and in the neck section, it utilizes SPP to upsample the image before passing it to the head for inference.In contrast, Yolov7 has a simpler structure where the output goes directly from the backbone to the head without passing through the neck. Although YoloV6^[18] was released later than Yolov7, in research related to floating objects in the sea, Yolov3^[19], Yolov5, Scaled Yolov4^[20], and Yolov7 have been primarily used. Yolov7, in partic- ular, demonstrated high recognition rates and fast in- ference speeds in the object detection competition held in July 2022.In this experiment, we will explore the differences between Yolov5 and Yolov7, for which prior research on object detection exists

When comparing the object detection rates of Yolov5 and Yolov7 using the same images, there was no significant difference in detection accuracy. However, in terms of inference speed, some noticeable differences between the two models were observed When using the model, the inference speed was meas- uredat 4 seconds. In addition to Yolov7, other com- monly used models for marine environment detection include Scaled Yolov4 and Yolov5. Therefore, Yolov5 will also be used as the object detection model to compare the inference speeds

Table 7.

Object Detection Results with EDN-GTM, Yolov5, and WaSR

Table 8.

Object Detection Results with EDN-GTM, Yolov7, and WaSR

When using Yolov5, it took approximately 10sec- onds for the sea fog to be removed from the image and for the unidentified underwater object to be in- ferred, with a slight variation of around 1 second. In contrast, when using Yolov7, the inference speed in the same situation was around 4 seconds, which is about 6 seconds faster. It can be assumed that if the inference had been conducted in a GPU environment rather than a CPU environment, the difference in in- ference speed would have been even greater. Next, to achieve faster inference speeds, I set the weight file of Yolov7 to Yolov7-tiny. The tiny version uses significantly fewer layers and filters compared to the standard weight file, and employs the LeakyReLU ac- tivation function in each layer, resulting in faster in- ference speeds for object detection. Therefore, it is expected that using the tiny model will lead to even faster inference speeds

Fig. 8.

EDN-GTM + Yolov7 + WaSR.(using CPU)

Fig. 9.

EDN-GTM + Yolov5 + WaSR.(using CPU)

Fig. 10.

Inference results using Tiny (Using CPU)

Contrary to expectations, applying the tiny model did not result in any difference in inference speed. This leads to the conclusion that the factor affecting inference speed is the sea fog removal algorithm, the EDNGTM model. This model is based on the U-net network, which, due to its deep and complex layers, is not suitable for real time processing. However, it is known to produce efficient results when performing tasks that require high accuracy.

Ⅴ. Conclusion and Future Research

This study presents an object detection method- ology for identifying unidentified underwater objects in marine areas, based on sea fog removal technology. In the preprocessing stage, fog removal was applied to enhance image clarity, followed by object detection using the YOLO algorithm, with background segmen- tation limited to the marine area to improve detection accuracy. Experimental results show that fog removal improved the identification accuracy of unidentified objects. however, some limitations were observed, such as reduced accuracy in images with high noise and the complexity of the model, which hinders real time processing. Future work will focus on addressing detection errors caused by noise, such as reflections on water, and enhancing real time detection performance. To achieve this, model optimization methods, including UNet model lightweighting using TensorRT^[21], will be explored for potential applica- tion in real time noise removal and object detection. Additionally, we aim to secure higher detection accu- racy to develop a solution that can be efficiently uti- lized in coastal surveillance operations