Real Time 3D Segmentation System Based on YOLOv8 and SAM for Sorting Recyclable Plastic Bottle Waste

Ⅰ. Introduction

Plastic materials are an essential component of modern society due to their low cost, lightness, and excellent physical properties. According to the 2022 OECD report, global plastic production increased from approximately 1.5 million tons in 1950 to approximately 460 million tons in 2019, and is projected to reach 1.23 billion tons in 2060 It is known that incinerating 1 ton of plastic produced from petroleum raw materials generates approximately 7 tons of carbon dioxide^[1]. Transforming our lives into a circular economy by recycling plastic materials is essential to protecting the environment and conserving resources. Many countries, including the EU and the US, have recently implemented regulations to promote plastic recycling. These regulations include mandating the use of a certain percentage of recycled plastic materials and improving product design to facilitate recycling, all aimed at ensuring environmental protection and sustainability^[2-4].

Plastic recycling technology encompasses processes that recover, sort, and process waste plastic for reuse or utilization as raw materials or fuel. Plastic waste is collected through various methods, including household collection, curbside pickup, drop-off centers, or recycling bins in public places. Subsequently, the collected plastic waste is transported to a sorting facility or recycling center, where it undergoes sorting based on size and material. This sorting process can be conducted manually by operators or with the assistance of machinery such as conveyor belts, trommels, and optical sorters. Once sorting is complete, recyclable materials are processed to prepare them for reuse^[5,6]. In particular, plastic PET bottles have a high recycling rate because they are clean and made of excellent material. However, the actual collection rate remains significantly low, and extensive human labor is required due to the lack of mature intelligent recognition and picking systems for automated sorting.

Recently, due to the development of artificial intelligence technology, it is being applied to sort recycling waste, but many studies focus only on object detection technology for waste objects. However, in the case of plastic bottle waste, it is difficult to accurately pick them because they are easily bent and distorted. To overcome these problems, it is necessary to accurately recognize the shape of recycled plastic bottle waste and obtain 3-dimensional information.

In order to overcome these issues, this paper proposes a real-time 3D segmentation system based on YOLOv8 and segment anything model (SAM) for sorting recyclable plastic bottle waste. The proposed system consists of 4 steps: data preprocessing, plastic bottle waste detection, 3D segmentation, and picking priority decision. The proposed method initially captures RGB images and depth data from a stereo camera. Subsequently, it undergoes a preprocessing procedure for background subtraction and noise reduction using both RGB and depth information. The preprocessed image data is then fed into the YOLOv8 model to detect bottle objects within the waste pile. YOLOv8 outputs the detected results, including labels, probabilities, and bounding boxes of the bottles. These data are then passed to the SAM model to obtain the contour and mask of the detected bottles. The mask is further merged with depth information to extract 3D segments of the detected bottle objects. This method determines the centroid coordinates of the detected bottles with depth and calculates their priorities for picking them up accordingly.

Some recycling waste sorting plants have started using robots to sort recycling waste. In many cases of artificial intelligence object detection, bounding boxes are used to recognize objects. However, the recycled plastic bottles that need to be picked are more flexible and are often crushed. Because of this, a problem arises in which the center of the plastic bottle cannot be picked up. Also, during the sorting process, if the robot goes down too far to pick up plastic waste, the bottles will jump out of the conveyor belt, and if it goes down short and tries to pick, it will not reach the waste and cannot pick it up. When using the 3D segmentation technology proposed in this paper, we believe that it will be able to overcome these problems and provide the advantage of increasing the overall sorting rate by providing accurate and fast picking.

This paper is structured as follows: Section 2 provides an overview of related work, including plastic waste recycling, segregation systems based on deep learning, YOLO, and SAM. Section 3 presents the proposed real-time 3D segmentation system based on YOLOv8 and SAM. Section 4 discusses the experimental results and performance evaluation. Finally, Section 5 concludes with future directions for further research.

Ⅱ. Related Work

2.1 Plastic waste recycling

This section outlines the functioning of waste recycling systems, focusing on plastic recycling, which can be categorized into three types: physical recycling, chemical recycling, and energy recycling. Physical recycling involves converting recycled raw materials into a physical state, such as pellets, through mechanical processes including crushing, sorting, separation, extrusion, and molding. Chemical recycling entails completely reverting polymer-type plastics back to their original raw material form, monomers, through chemical reactions. Energy recycling involves utilizing plastic waste as an alternative fuel for power generation facilities, cement processes, boilers, and other applications^[7].

Figure 1 illustrates a plastic waste recycling process. Plastic waste collected through various methods in diverse environments is transported to recycling centers where it undergoes sorting by size and material. Sorting facilities categorize waste into various groups, including paper, plastic, glass, metal, and organic waste. Once sorted, recyclable materials undergo processing to prepare them for reuse. Depending on the material, this processing may involve washing, shredding, grinding, dissolving, or other techniques. Processed materials are then reintegrated into the production cycle to manufacture new products^[8,9].

Fig. 1.

Plastic waste recycling process.

2.2 Segregation system using deep learning

Several previous studies have utilized deep learning techniques to recognize waste materials. Susanth^[10] introduced an image classifier that identifies objects and detects the type of waste material using convolutional neural networks (CNNs) such as ResNet50, DenseNet169, VGG16, and AlexNet. Sudha^[11] proposed an automatic classification method for environmentally friendly waste segregation using deep learning, where objects are classified as biodegradable and non-biodegradable. Sousa^[12] suggested automating waste sorting with deep learning. Their two-step approach combines recent object detectors like Faster R-CNN with higher resolution bounding boxes to support the classification task effectively. Additionally, a study^[13] experimented with well-known deep convolutional neural network architectures, achieving a 90% test accuracy using Inception-ResNet.

Existing studies primarily focus on object detection, which may not fully address the challenges posed by contaminated or distorted plastic bottles in waste recycling centers. To enable robots to accurately pick up such bottles, it's crucial to accurately extract the segment of the bottle rather than just the bounding box. Moreover, providing both x, y-coordinate and z-axis information is essential for efficient waste sorting. This requires more advanced segmentation techniques that can handle irregular shapes and provide depth information, ensuring precise localization and manipulation of plastic bottles within the recycling process.

2.3 YOLO

Absolutely, the ability to recognize and locate waste is crucial for robots or sorters in waste management systems. Object detection technology plays a significant role in achieving this goal. YOLO (you only look once) is a popular choice for real-time object detection tasks, including those in video analytics and surveillance. Over time, YOLO has evolved through various versions, each building upon the previous one with improvements in accuracy, speed, and handling of small objects. Among these versions, YOLOv8 stands out as it ensures real-time processing while maintaining high accuracy. Ultralytics YOLOv8, in particular, is designed to be fast, accurate, and user-friendly, making it suitable for a wide range of applications such as object detection, tracking, instance segmentation, image classification, and pose estimation tasks. Its versatility and efficiency make it a valuable tool for waste recognition and location in waste management systems.

2.4 Segment anything model

The recent announcement of the segment anything model (SAM) by a Meta AI research team marks a significant advancement in image segmentation technology. SAM is capable of producing high-quality object masks from input prompts such as points or bounding boxes, and it can generate masks for all objects within an image. SAM has been trained on an extensive dataset consisting of 11 million images and 1.1 billion masks, resulting in strong zero-shot performance across various segmentation tasks. Its architecture, illustrated in Figure 2, incorporates a powerful image encoder, prompt encoder, and lightweight mask decoder. This unique design allows for flexible prompting, real-time mask calculation, and recognition of ambiguity in segment operations. SAM's capabilities hold promise for a wide range of applications, including object recognition, scene understanding, and image editing, where accurate segmentation is essential. Its ability to generate high-quality object masks efficiently and effectively positions it as a valuable tool in the field of computer vision and artificial intelligence.

Fig. 2.

Architecture of the segment anything model.

Ⅲ. Real Time Bottle Detection and 3D Segmentation based on YOLOv8 and SAM

3.1 System overview

Figure 3 provides an overview of the robot system utilized for sorting recyclable waste. Previously, we introduced a robot system for depth extraction and object positioning in ^[18], where a k-nearest neighbors (kNN) algorithm was employed for object tracking. In this paper, we extend upon this system by integrating the latest deep learning models for precise object identification. In the sorting process, plastic waste travels along a conveyor belt beneath the delta robot. At the entrance of the delta robot system, a stereo vision camera is positioned to capture images of the waste materials. For our proposed real-time 3D segmentation system for sorting recyclable plastice bottle waste, we utilize the Jetson Orin NX board and Stereolabls's ZED X mini camera. The proposed method operates sequentially, starting with data preprocessing using depth information, followed by bottle detection using YOLOv8, and 3D segmentation using SAM. Additionally, the method determines the centroid coordinates of the detected bottles with depth and calculates their priorities for picking them up accordingly. This comprehensive approach enhances the efficiency and accuracy of the waste sorting process.

Fig. 3.

System architecture for real time bottle detection and 3D segmentation: (a) whole system process and (b) delta robot for recycling.

3.2 Data preprocessing using depth

The system proposed in this paper primarily relies on RGB images and depth information. While technologies like LiDAR and infrared (IR) structured light are commonly used for depth extraction, they face challenges when dealing with transparent objects such as plastic bottles, as their wavelengths penetrate through them. The ZED camera, on the other hand, utilizes stereo vision technology to extract depth and is equipped with neural networks to enhance depth accuracy. Many object detection technologies solely utilize RGB images for object detection. However, in the context of waste sorting, images often contain multiple waste objects along with noise elements such as contamination. This complexity necessitates advanced techniques for accurate object detection and segmentation, which is addressed in the proposed system through the integration of depth information alongside RGB images and the utilization of deep learning models like YOLOv8 and SAM. This comprehensive approach enhances the system's capability to effectively identify and segment waste objects in real-time.

Once the system captures RGB images and depth information, it establishes a target distance for background removal. Subsequently, the system generates a mask based on the measured depth, followed by performing an “AND” operation between this mask and the original image. This operation results in a new image where the background or complex noises are eliminated, as depicted in Figure 4^[18]. The preprocessed image, free from background clutter and noise, is then utilized for object detection using YOLOv8. This approach enhances the accuracy and reliability of object detection by focusing on the relevant regions of the image while minimizing interference from irrelevant background elements.

Fig. 4.

Original image and preprocessed image: (a) raw RGB image and (b) preprocessed image using depth.

3.3 Bottle detection and instance segmentation

Figure 5 illustrates how the system detects plastic bottles and obtains their instance segments using YOLOv8 and SAM. YOLOv8 is employed to detect waste objects by analyzing the preprocessed image and generating bounding boxes around the detected waste.

Fig. 5.

Flow chart for plastic bottle detection and 3D segmentation.

Traditional sorting systems typically utilize object detection techniques that solely rely on bounding boxes for waste classification. However, in the case of waste, plastic bottles come in various irregular forms. Failure to accurately identify the center of these irregular bottles can lead to issues such as improper picking by the delta robot or bottles popping out of the system. To address this challenge, this paper adopts instance segmentation techniques. After applying a bottle pass filter to select plastic bottle candidates based on the output labels from YOLOv8, the preprocessed image and bounding boxes are input into SAM to obtain masks for each instance.

The recycled plastic bottles that need to be picked are more flexible and are often crushed. Because of this, a problem arises in which the center of the plastic bottle cannot be picked up as shown in Figure 6. In order to decide the centroid of the bottle, the paper utilizes the pretrained SAM-b model, trained on a dataset of 11 million images. The output masks are then used to identify the contours and center points of each object, which are combined with depth information to generate 3D segments. This approach ensures more accurate and reliable identification and segmentation of plastic bottles within the waste stream.

Fig. 6.

Picking problem with crushed bottles.

where, the weighting function between probability and depth, denoted by α, is a constant used to balance the influence of probability and depth in the selection process. It ensures that both factors contribute appropriately to the overall score used for prioritizing the detected bottles. The probability of the detected bottle, represented by [TeX:] $$P_i$$, is provided by the output of YOLOv8 and reflects the confidence level of the detection. The normalized distance value, denoted by [TeX:] $$\text { Dist}_{\text {Nor }}$$, is derived using Equation (2) and represents the distance of the detected bottle from the camera, normalized to a suitable range for the weighting calculation.

To address the slower speed of SAM compared to YOLOv8, the system employs a strategy to handle multiple objects detected in a single image. Instead of processing each object individually in real-time, the system identifies multiple objects, sorts them based on priority, and stores them in a queue. This allows the system to efficiently manage the object selection process while ensuring that the most relevant objects are prioritized for pickup in Equation (3). Once the objects are sorted and queued, the system then transfers the object information, including the picking point coordinates (x, y, z), to the delta robot controller. This information enables the delta robot to precisely position itself and pick up the objects in the designated order. By implementing this approach, the system optimizes the utilization of resources and ensures smooth coordination between the object detection system and the delta robot, ultimately improving the overall efficiency of the waste sorting process.

Ⅳ. Experimental Results

Figure 7 depicts a graphical user interface (GUI) designed for real-time 3D segmentation. The GUI is comprised of several components: (a) raw RGB image view, (b) depth image view, (c) mask view, depth and contour view, (d) mask view for background subtraction, (e) input image for inference, (f) corrected depth map, and (g) YOLOv8 result view. (a) The raw RGB image shows the left image captured from the stereo camera, and (b) shows the grayscale image for depth extracted through stereo vision. (c) converts the extracted depth image into JET format, and overlays the bottle contour obtained through the segmentation process. Then, the center point of each bottle was found and marked. This center point refers to the point where the robot will pick. (d) The image view shows a mask image to remove the background or object if it is farther than the threshold distance based on the measured depth, and (e) shows the result image to remove the background using the mask image and the original input image. This image is used as an input image to detect actual recycled plastic bottles. (g) shows the results of plastic objects detected through YOLOv8, and (f) shows the results of correcting the depth information of the detected and segmented plastic bottles.

Fig. 7.

The graphic user interface for real time plastic bottle detection and 3D segmentation.

Additionally, the GUI includes an interface that enables users to adjust parameters such as the distance threshold for noise and background removal. This interactive feature allows users to fine-tune the system settings according to specific environmental conditions and requirements, enhancing the system's adaptability and usability.

4.1 Result by plastic trash dataset.

In the initial application of the proposed technique, the instance segmentation trash plastic dataset provided by Roboflow^[19] was utilized for training. However, as depicted in Figure 8, it was observed that the detection performance and depth accuracy were considerably low. One of the primary reasons for this low accuracy is attributed to the composition of the training dataset.

Fig. 8.

Experimental results by YOLOv8-seg: (a) snippet of training dataset, (b) real time detection result, and (c) depth image and segmented instances.

The dataset used for training contained not only plastic bottles but also various other types of plastic waste. This heterogeneity in the dataset likely introduced challenges for the model, making it difficult to accurately distinguish plastic bottles from other types of plastic waste. As a result, the model's performance suffered, leading to reduced detection accuracy and depth estimation.

4.2 Result by COCO dataset

Upon applying the pretrained model with the COCO dataset^[20] to the proposed system, notable improvements were observed in both the detection performance and depth accuracy compared to the plastic trash dataset. Figure 9 illustrates the results of this application, showcasing a composite image combining depth and contour based on the YOLOv8 bottle detection results, as well as the outcomes of 3D segmentation. Upon detection of a plastic object by YOLOv8, it becomes evident that SAM performs segmentation with precision. This improved performance can be attributed to the utilization of a more diverse and comprehensive dataset like COCO, which contains a wider variety of object classes and scenarios. The pretrained model, having been trained on such a dataset, exhibits better generalization capabilities and is better equipped to handle a broader range of objects and environmental conditions. By leveraging the strengths of YOLOv8 for object detection and SAM for instance segmentation, the proposed system demonstrates enhanced accuracy and reliability in identifying and segmenting plastic objects, contributing to the overall effectiveness of the waste sorting process.

Fig. 9.

Experimental results by YOLOv8 and SAM: (a) raw RGB image, (b) depth image with contour and (c) 3D segments.

As mentioned before, this paper used a pretrained model using the COCO dataset to classify plastic bottles. Nevertheless, we evaluated the performance of the YOLOv8 with COCO dataset to verify its superiority. The Microsoft common objects in context (MS COCO) dataset contains 91 common object categories with 82 of them having more than 5,000 labeled instances. In total the dataset has 2,500,000 labeled instances in 328,000 images^[20]. Figure 10 shows the training and validation loss curves for the COCO dataset during training to verify the superiority in classification. For performance evaluation, the epoch was set to 30 and the batch size was set to 8.

Fig. 10.

Training and validation loss curves.

In this paper, the depth of various plastic bottles was measured and compared. Figure 11 displays the types of plastic bottles utilized for depth measurement and segmentation. It is observed that plastic bottles with numerous edges, corners, and textures tend to exhibit higher accuracy in both detection and segmentation. Conversely, if a plastic bottle lacks distinct features or is relatively smooth, the accuracy of detection and segmentation may decrease. The expectation is that highly contaminated and crumpled plastic bottles, commonly encountered at actual recycling waste sorting sites, would present challenges for detection and segmentation due to their irregular shapes and obscured features. However, it's worth noting that the proposed system, equipped with advanced deep learning models like YOLOv8 and SAM, is designed to address such challenges by leveraging both RGB and depth information to accurately identify and segment plastic bottles, even under challenging conditions. By evaluating the system's performance across a variety of plastic bottle types and conditions, this paper provides valuable insights into the capabilities and limitations of the proposed approach, paving the way for further improvements in waste sorting technology.

Fig. 11.

3D segmentation with various bottle types.

4.3 picking accuracy by segmentation

The detected information must ultimately be used for picking by the robot. However, if the picking is not performed even though the detection is accurate, time is wasted during the off-picking, which reduces the sorting speed. Figure 12 (a) shows the result of calculating the center of a distorted bottle using the bounding box used in traditional object detection techniques, and Figure 12 (b) shows the result of calculating the center point after performing segmentation. The result in (a) shows a situation in which the robot cannot completely pick, and in case (b), we can see that the picking point is located on the plastic bottle.

Furthermore, the proposed method is expected to increase the sorting speed of the entire system by detecting that the picking point is located at the border of the plastic bottle, avoiding it, and not including it in the picking target.

Fig. 12.

Picking accuracy: a) bounding box and (b) instance segment.

Furthermore, the proposed method is expected to increase the sorting speed of the entire system by detecting that the picking point is located at the border of the plastic bottle, avoiding it, and not including it in the picking target.

Ⅴ. Conclusion and Future work

While efforts such as carbon-neutral policies are underway to mitigate global warming, reducing plastic production, which is ubiquitous worldwide, presents significant challenges. Plastic PET bottles, in particular, hold greater value in terms of quality compared to other plastic waste. However, optically judging and classifying transparent PET bottles pose limitations. To address these challenges, this paper proposes a novel approach leveraging object detection and segmentation techniques. By incorporating distance- dependent background and noise removal, along with the fusion of state-of-the-art YOLOv8 and SAM technologies for segmentation, the proposed method aims to accurately identify and segment plastic waste, particularly PET bottles. Additionally, a target determination method is proposed for delta robots to enable precise collection of plastic waste. The proposed technology holds promise for practical application in recycling waste classification, offering a comprehensive solution to the complexities associated with plastic waste sorting. Furthermore, its versatility suggests potential applications beyond waste management, highlighting its significance in addressing the broader environmental challenges posed by plastic pollution.

The current system detects both glass and plastic bottles. In the future, we will upgrade the system so that we can classify according to the type of plastic materials by securing data according to material and learning new things.