GDR: A LiDAR-Based Glass Detection and Reconstruction for Robotic Perception

Ⅰ. Introduction

To be able to perform its automation functions, a robot is required to independently collect data, process information, accurately identify conditions, and make and execute appropriate decisions^[1]. One of the first key aspects, and one of the most important, is the ability to perform independent data collection.

Data collection by robots is carried out to obtain information like the state of the robot itself or the conditions of their surrounding environment. This in- formation is then processed to produce decisions and actions that areaccurate and fit the actual needs^[2].

For example, in the application of robots as un- manned any vehicle (UXV), robots are required to un- derstand their surrounding environment to achieve various automation functions, which typically include robot localization, environment mapping, and robot navigation^[3,4].In such cases, robots are often equipped with a range of sensors such as inertial measurement unit (IMU), camera, global positioning system (GPS), sonar, light detection and ranging (LiDAR), or ra- dar^[5,6]. Although these sensors serve similar purposes, each has its own advantages and limitations, making the choice of sensor application dependent.

For example, in terms of environmental conditions, the choice of sensors differs between indoor and out- door applications. In indoor environments, LiDAR and camera are often preferred over GPS because GPS signals are difficult to obtain due to the lack of satellite visibility. Conversely, in open outdoor environments, GPS is usually favored over LiDAR due to challenges such as the scarcity of geographic features in LiDAR data, which complicates localization, or the high intensity of sunlight that can cause significant noise in camera data. In some cases, specific environmental conditions pose unique challenges to robots, limiting their ability to perform autonomous functions effectively.

In indoor environments, experimental results have shown that in urban buildings with a high prevalence of glass planes, data from camera^[7] and LiDAR can become inaccurate in representing the actual surroundings (Figure 1). This is problematic because LiDAR and camera are typically the primary sensors relied upon in such environments. The issue arises because LiDAR, as an optical sensor, relies on light waves for environmental perception^[8], and camera capture light reflected from objects in the surroundings. Glass planes naturally refract light, leading to incorrect or incomplete perceptions by LiDAR and camera. Instead of detecting the glass plane itself, these sensors often perceive the objects behind it. This in- complete and incorrect perception of the glass plane, if not addressed, could lead to the robot incorrectlyassuming that an area in the environment is free instead of occupied. Naturally, this condition couldcause issues in the implementation of autonomous navigation in the robot, such as the possibility of collisions with glass obstacles.

Fig. 1.

Incomplete glass plane perception by LiDAR. (a) Real environment view. (b) LiDAR point cloud. (c) LiDAR point cloud with estimated glass plane and glass boundary view.

Furthermore, in localization and mapping systems, additional issues may arise, such as landmark recognition failure or loop closure detection failure, due tothe high noise generated by glass objects as laser beams scatter unpredictably^[9]. This could result indecreased reliability of the robot's localization results, which in turn could cause the autonomous navigationsystem on the robot to not perform as expected.

Therefore, a system is needed that can first detect the presence of glass planes in the environment and subsequently reconstruct the glass plane in the perception data, so that the resulting perception is complete, including portions of the glass plane that are inherently undetected by the robot sensor.

Based on that motivation, this paper proposes GDR, a LiDAR-based glass detection and reconstruction, aimed at improving LiDAR perception capabilities in glass-rich environments. GDR addresses the limitations of glass perception by LiDAR through the contributions of this paper as follows:

1. The introduction of two feature extraction method based on the distance and intensity information, for detecting glass planes based on LiDAR data

2. A simple linear equation-based approach to efficiently reconstruct glass planes in the point cloud data.

3. GDR’s LiDAR-based perception-level approach enables its use in various applications that relyon LiDAR point clouds, including localization, mapping, SLAM, and obstacle avoidance for robot navigation systems.

Ⅱ. Related Works

Some researchers have attempted to identify the presence of glass planes using the small number of LiDAR points generated by glass surfaces (glass points). There is a unique characteristic in LiDAR points from smooth objects such as marbles, polished woods, shiny metal planes, and glass planes (Figure 2), where at glass points that are close to the normal angle, a significant spike in intensity value occurs. The difference lies in the height of the spike, the intensity gradient of glass points, which tends to be higher, and the profile width, which is generally narrower than points on other smooth objects. Using this phenomenon, [10][11][12][13] proposed thresholding methods on these three characteristics and attempted to implement them on 2D LiDAR. Meanwhile, [14] using 3D LiDAR, also proposed a thresholding method on the intensity value of glass points, but only on points in areas where spikes occur. The intensity gradient is detected using a divergence method. However, this phenomenon only occurs when the LiDAR laser beam strikes the glass surface at or near the normal angle. The challenge arises because LiDAR mounted on robots are often imperfectly aligned, rarely remaining perpendicular to the glass plane in the environment. Additionally, uneven floors can prevent the laser beam from hitting the glass surface at an angle close enough to the normal angle, resulting in glass points failing to exceed the intensity threshold and thus not being classified as potential glass points. This issue persists even with 3D LiDAR, despite its higher number of channels and smaller gap between them, which theoretically increases the likelihood of the beams hitting glass near its normal angle. Experimental results show that even 64-channel 3D LiDAR sometimes fails to generate glass points withintensity values exceeding the threshold. Consequently, this method does not consistently detect glass surfaces in the environment. If glass detectionfails, glass reconstruction that will include glass surfaces as LiDAR perception results cannot be achieved.

Fig. 2.

Intensity value characteristics on glass and other reflective-material objects.

Furthermore, [15] proposed another method usingthe standard deviation of distance values from LiDAR points. This method assumes that points behind glass objects exhibit greater noise in their distance values compared to those not obscured by glass. By applyinga threshold, points with a standard deviation exceeding the threshold are categorized as being behindglass. Glass reconstruction then begins and ends at the points where the standard deviation exceeds the threshold. However, this method is not entirely reliable, as glass detection depends on the presence of objects behind the glass, which is not always the casein real world environments. Additionally, errors inglass reconstruction may occur when objects are positioned in front of the glass, causing the reconstruction to begin or end at the object in front rather than at the true boundary of the glass. As a result, this methodis also unable to correctly reconstruct the point cloud.

Other researchers have proposed combining RGBand depth information from RGB-D camera^[16,17], sensor fusion between RGB-D camera and LiDAR^[18], using polarization camera and LiDAR^[19], or sonar/ultrasonic sensors with LiDAR^[20,21]. However, methods that rely on RGB-D camera have significant limitations, primarily due to the minimal distance information between the robot and the glass plane. This limitation makes it difficult to estimate the position and profile of glass planes in the environment. Additionally, camera naturally depend heavily on lighting conditions, and there is no guarantee that lighting will always be sufficient to support glass detection via camera. In contrast, glass detection methods using ultrasonic sensors are unaffected by lighting conditions. However, these methods face the common drawback of a narrower field of view (FOV) compared to LiDAR. This limitation increases the difficulty of estimating the position and profile of glassplanes and reduces the likelihood of successfully detecting them. Moreover, the use of multiple sensors adds computational complexity to the robot's system, slowing down autonomous processes and potentially impairing its perception capabilities.

To address these issues, this paper proposes a LiDAR-based glass detection method that is more robust to changes in lighting conditions, offers a wider FOV compared to other sensors, and provides more comprehensive depth information about the glass plane. This enables more accurate estimation of the glass plane's position and profile. Additionally, a new distance feature and intensity feature extraction approach is proposed to overcome the limitations of previous methods, which have not adequately considered the effects of uneven floor contours or the loss of LiDAR's vertical perpendicularity to the glass plane.

Ⅲ. Proposed Method

In this chapter, a detailed explanation will be provided on how GDR performs glass detection and glass reconstruction on LiDAR point cloud inputs, up to the generation of a corrected point cloud. As illustrated in Figure 3, the objective of GDR is to receive LiDAR point cloud input, process it, and produce are fined, corrected point cloud.

Fig. 3.

GDR’s system model.

3.1 Glass Detection

The process begins with the Glass Detection module, which comprises two feature extractors based on the distance and intensity information of the input point cloud. Based on those extracted features, glass points are identified, representing the presence of glass planes in the environment. The glass detection starts with distance feature extraction which essentially assesses the variance of a group of points that are considered clustered. In Figure 2, it is evident that only a few glass points are generated by the LiDAR. In this condition, since the glass points are primarilylocated near the normal angle, the variance of the distance values from these points tends to be lower compared to the points from many other objects with different materials. This occurs because the distance values of points on the other objects are typically more varied compared to those on a glass plane, where the points tend to have more similar distance values to each other.

The first step in this feature extractor involvesgrouping the LiDAR points from the input point cloud P_S, which contains a set of points p, into several clusters C, with some noise points N. The clustering process is based on the spatial positions of the points in 2D space. This system employs the DBSCAN clustering method for grouping points. DBSCAN is chosen because, unlike clustering methods such as K-Means, it does not require a parameter k, allowing the clustering results to yield n clusters that reflect the true structure of the data, especially for 2D LiDAR data.

Next, for each cluster, the variance of the distance values for all points within it is calculated. If this value is less than the variance threshold [TeX:] $$\begin{equation} \sigma_{\text {tres }}^2 \end{equation}$$, the cluster is categorized as candidate glass points and added to P_cg. If the value exceeds [TeX:] $$\begin{equation} \sigma_{\text {tres }}^2 \end{equation}$$, the points in that cluster are categorized as non-glass points and will not be processed further in the subsequent feature extractor.

Next, in the intensity feature extractor, glass points are identified based on their intensity values. As shown in Figure 4, the intensity values on a glass plane exhibit two types: values within the intensity spike and values outside of it. The values within the intensity spike are further divided into two segments: those before the peak point, p_peak, which is the point with the highest intensity, and those after the peak point. As seen in Figure 4a, the values before the peak tend to follow an upward trend, with a nearly always positive gradient. Conversely, values after the peak generally show a downward trend, with a nearly always negative gradient. Additionally, intensity values outside the spike are almost always more numerous than those within the spike. Based on these characteristics, the intensity feature extractor will refine the candidate glass points produced by the distance feature extractor, denoted as P_cg.

The first step is to smooth the intensity values of points within each cluster in P_cg using a moving average algorithm (as shown in Figure 4b). This smoothing stabilizes the intensity values without excessively altering them, making the gradient analysis of the intensity spike easier to perform.

Fig. 4.

Glass-points-intensity smoothing. (a) Raw data, before smoothing. (b) Smoothed data, after smoothing.

Following this, the minimum intensity value i_min and the maximum intensity value i_max (intensity range) are identified. Based on this range, a set of k bins are created to map points according to their intensity values (as shown in Figure 5a). Experimental results indicate that using k = 5 bins effectively separate the points within a cluster into two groups: those within the intensity spike and those outside it.

Once the 5 bins are created and the points in each clusters in p_cg are mapped into them, the bin containing the lowest values within the cluster is retained, while the remaining 4 bins are merged into one. This results in only two bins, B1 and B2, effectively separating the points within the intensity spike from those outside of it (as shown in Figure 5b).

Fig. 5.

Glass points mapping. (a) Points mapped into 10 bins. (b) Points mapped into 2 bins, where the first bin unchanged, and the other bins are merged into one.

Based on these two bins, the number of points in B1 is compared to the number of points in B2. If the number of points in B1 is greater than that in B2, the cluster will proceed to further processing. Conversely, if the number of points in B1 is less than that in B2, the points in the cluster will be classified as non-glass points and will not undergo further processing.

Next, the intensity values of points in B2 are checked for an increasing trend, with a consistently positive gradient before reaching the peak, followed by a decreasing trend with a consistently negative gradient after the peak. If this condition is met, the points in the cluster are categorized as glass points and added to P_g. If not, they are classified as non-glass pointsand will not undergo further processing. Thus, the glass points are gathered in P_g. P_gis then passed to the glass reconstruction module for the glass plane reconstruction process.

3.2 Glass Reconstruction

After obtaining the clusters of glass points in P_g from the glass detection module, the glass reconstruction module reconstructs the glass plane based on these clusters of glass points. This process begins by selecting the point with the highest intensity value, p_peak, in each cluster. Each point in the 2D LiDAR data inherently contains information on distance/range and azimuth angle, which represents a vector from the LiDAR origin point to the given point. Assuming that p_peak in each glass point cluster represents the most accurate measurement, glass plane reconstruction is based on p_peak.

For each cluster of glass points, a vector [TeX:] $$\begin{equation} \overrightarrow{v_{\text {peak }}} \end{equation}$$ is defined based on p_peak. Then, a perpendicular vector [TeX:] $$\begin{equation} \overrightarrow{v_{\text {perp }}} \end{equation}$$ is defined at p_peak, positioned exactly at this point. Thus, p_peak becomes the intersection point between [TeX:] $$\begin{equation} \overrightarrow{v_{\text {peak }}} \end{equation}$$ and [TeX:] $$\begin{equation} \overrightarrow{v_{\text {perp }}} \end{equation}$$.

The goal of the Glass Reconstruction module is to correct points that should lie on the glass plane but are instead positioned behind it or have infinite values (inf/ 0) due to the laser beam not returning to the LiDAR. To determine whether a LiDAR point belongs on the glass plane, a check is conducted starting from the outermost point of the glass point cluster and extending to neighboring points. This evaluation is based on the Cartesian coordinate system, although 2D LiDAR points are originally defined within a spherical coordinate system.

As illustrated in Figure 6, the point p_check, shown in blue, is the point under inspection, and the vector [TeX:] $$\begin{equation} \overrightarrow{v_{\text {perp }}} \end{equation}$$serves as an estimated profile of the glass plane. So next, based on the vector [TeX:] $$\begin{equation} \overrightarrow{v_{\text {check }}} \end{equation}$$, which is defined based on p_check, find the intersection point p_int between [TeX:] $$\begin{equation} \overrightarrow{v_{\text {check }}} \end{equation}$$ and [TeX:] $$\begin{equation} \overrightarrow{v_{\text {perp }}} \end{equation}$$. This intersection point p_int is assumed to be the position where p_check should ideally be located. In other words, p_int is the point on the LiDAR laser beam defined as [TeX:] $$\begin{equation} \overrightarrow{v_{\text {check }}} \end{equation}$$, representing the correct measurement from the laser beam [TeX:] $$\begin{equation} \overrightarrow{v_{\text {check }}} \end{equation}$$ and lying on the actual profile of the glass plane.

Fig. 6.

Glass Reconstruction: point correction mechanism.

Once p_int is identified, the distance from the origin to p_check, denoted as d_check, is compared with the distance from the origin to p_int, denoted as d_int. If d_check is greater than d_int, then p_check will be replaced withp_int. If d_check is less than d_int, it will be checked whether p_check belongs to another cluster fromthe DBSCAN clustering results, C. If so, p_check will be consideredas a glass boundary, and the checking will be stopped. If p_check belongs to N, which represents noise in the DBSCAN clustering results, then p_check will be replaced with p_int, and the process will continue until the glass boundary is found.

Ⅳ. Experiments and Results

4.1 Experiment Details

To implement and further test the proposed method, a 2D LiDAR, the Oradar MS500 (Figure 7), which includes intensity information, is mounted on a tripodto capture point cloud data of a campus building with numerous glass walls and glass doors. An Intel NUC 13 Pro minicomputer is used for processing, featuring an Intel Core i5 1360P processor, 64 GBof RAM, and 2 TB of SSD storage.

Fig. 7.

2D LiDAR: Oradar MS500.

To evaluate the capability of the glass detection module in identifying glass planes, several experimental scenarios were conducted and compared with the previous method^[11]. First, the LiDAR was moved parallel to the glass plane, creating a lateral movement, while detecting the presence of the glass plane in real time. This scenario was designed to highlight the potential for glass detection failures due to the robot moving over uneven floor contours, which may cause the LiDAR’s position to deviate from being perpendicular to the normal angle of the glass plane, as illustrated in Figure 8a.

Second, the LiDAR will be rotated multiple times in the same position, creating a rotational movement. This is intended to introduce the possibility of glass detection failure due to imperfections in the LiDAR installation on the robot or specific conditions affecting the robot, which may cause the LiDAR to deviate from a 0-degree vertical angle in the robot’ s body frame, as illustrated in Figure 8b.

Fig. 8.

LiDAR misalignment against glass planes. (a) The effect of uneven floor contour on laser beams position relative to the glass-plane normal angle. (b) The effect of the robot body’s tilt on the laser beams position relative to the glass-plane normal angle.

Third, the LiDAR will be positioned at varying distances from the glass plane, forming a longitudinal movement. This experiment aims to assess the tested method’s ability to handle changes in the distance between the LiDAR and the glass plane.

Next, to test the capability of the Glass Reconstruction module in reconstructing a glass plane, several trials will be conducted on different types and conditions of glass surfaces. First, the LiDAR will be positioned close to a wide glass wall, and glass detection will be performed in real time. Second, the LiDAR will be positioned close to a glass door. Following this, glass plane reconstruction will be carried out based on the detected glass points. Based on that, the accuracy of the glass plane reconstruction results will be checked against the actual glass plane. Then, the success rate of the reconstruction processon the obtained glass points will be calculated.

4.2 Results

Based on the experiments conducted, as shown in Table 1, GDR achieved a fairly high accuracy in glass detection under various LiDAR positioning conditions, often detecting glass points with intensity values that do not exceed the fixed thresholds set by previous method [11]: 78.46% accuracy in lateral movement, 83.48% in rotational movement, and 89.07% in longitudinal movement. Since GDR does not directly apply a threshold on intensity values when detecting glass points, it is able to maintain good accuracy across these varied conditions.

Next, regarding the Glass Reconstruction module, as shown in Figure 9, the input point cloud (Figure 9a) was corrected successfully by the module (Figure 9b), and the corrected point cloud matches the actual condition of the glass plane in the environment. In Table 2, it can be seen that for both the glass wall and glass door, the reconstruction process achieved a high success rate, with a small margin of error related to the estimation inaccuracies of the glass plane profile.

Fig. 9.

Glass Reconstruction result visualized on RViz. Some points marked with green box are the detected glass points. (a) Input point cloud. (b) Corrected point cloud.

Ⅴ. Conclusion and Future Work

This paper proposed GDR, a LiDAR-based glass detection and reconstruction for robotics perception. GDR enhances LiDAR perception in areas with numerous glass planes, generating a corrected point cloud applicable to various robotics applications, such as robot localization, environment mapping, SLAM, and navigation. Experimental results demonstrate that GDR effectively tackles challenges related to LiDAR misalignment with glass planes, particularly in environments with uneven floor contours, suboptimal LiDAR positioning, or specific robot conditions. These factors have previously reduced detection accuracy in methods that relied on direct thresholding of glass point intensity values. GDR’s Glass Detectiontests showed an overall accuracy of 83.67%, which outperforms the previous method^[11], while the Glass Reconstruction module achieved accuracy and success rates of 100% and 96.77%, respectively.

In the future, the proposed Glass Detection methodwill be further developed for full implementation in robotics applications using 3D LiDAR. This implementation will focus on robots requiring 3Dperception, such as quadcopters or robotics manipulators, and will aim to optimize the reconstruction method for greater computation efficiency.