Optimizing OFDM Modulation in OCC Systems: A Comparative Analysis of AI-Based Optical Channel Equalization Models

Ⅰ . Introduction

The demand for wireless communication continues to surge globally, driven by the rapid adoption of fifth-generation (5G) networks. Looking ahead, artificial intelligence (AI)-driven sixth-generation (6G) systems are expected to be deployed between 2027 and 2030^[1]. To accommodate increasing data traffic, wireless systems require expanded bandwidth and improved spectral efficiency, necessitating the utilization of the broader terahertz spectrum^[2].

Optical wireless communication (OWC) leverages the entire optical spectrum to enhance data transmission rates by expanding available bandwidth^[3]. With significant theoretical bandwidth potential, OWC serves as a key area of research in terrestrial optical wireless technologies, encompassing free-space optical (FSO) communication, visible light communication (VLC), light fidelity (Li-Fi), and optical camera communication (OCC)^[4]. OCC has gained traction in applications such as indoor networking, vehicular communication, and broadcasting. To improve data throughput and propagation range, OCC employs advanced modulation techniques, frequency diversity, luminance modulation of light-emitting diodes(LEDs), and multidimensional color encoding^[5].

Several studies have explored various OCC frameworks. The rolling light system is a line-of-sight (LoS) OCC framework designed for efficient data transmission using rolling shutter and global shutter cameras in commercially available devices^[6]. Another study investigated an OCC system utilizing a smartphone camera with a free-space on-off keying (FSOOK) modulation scheme, achieving a low data rate^[7]. An OCC system employing a low-speed rolling-shutter camera demonstrated data rates of 1.02 kbps with a single LED and 64 kbps with an 8 × 8 LED matrix. This system utilized a neural network for LED detection and bit error rate (BER) reduction; however, rolling shutter distortions and interference posed challenges in high-speed mobility scenarios^[8].

Further research introduced an RS-based multiple-input multiple-output (MIMO) OCC system, integrating a compact 8 × 8 LED array transmitter with a Raspberry Pi camera receiver. This system achieved flicker-free communication with data rates ranging from 120 bps to 13.44 kbps over link spans of up to 1.8 m, making it suitable for short-range, low-data-rate Internet of Things (IoT) applications. However, it faced limitations such as image saturation at higher shutter speeds, reduced reception success over longer distances, and increased system complexity in practical scenarios involving mobility, rotation, and multiuser environments[9]. Similarly, hybrid VLC/OCC systems have been studied for indoor applications, where performance is influenced by brightness levels and data rate, often utilizing smartphones or photodiode-equipped VLC receivers^[10].

Another study implemented a DC-biased optical orthogonal frequency division multiplexing (DCO-OFDM) OCC system using a smartphone camera. The results indicated that the D-ELFC algorithm outperformed the P-CVS algorithm in terms of BER and data transmission rate. However, system performance was affected by ambient lighting conditions and the rolling shutter effect, which can degrade signal detection accuracy in real-world environments^[11]. Additionally, a convolutional neural network (CNN)-based LED detection method was proposed, incorporating a deep neural network for OFDM frame synchronization in rolling shutter OFDM (RS-OFDM) systems^[12].

Many prior studies have overlooked synchronization challenges or relied on graph-based programming in LabVIEW, which is impractical for real-world deployment. In contrast, this study employs Python scripts for data transmission and reception at 16 kHz. The proposed work introduces a novel OCC system architecture, including a comparative analysis of AI models for signal equalization. Furthermore, a Hermitian mapping technique, inspired by mobile communication systems, is integrated to enhance OCC performance. The study also implements real-time OFDM modulation on an embedded system and introduces a zero-integration transmission method to mitigate synchronization issues.

This paper is structured as follows: Section 2 describes the proposed system architecture. Section 3 presents the model and dataset description. Section 4 details the experimental setup. Section 5 discusses theresults. Section 6 concludes the study.

Ⅱ. Proposed System Architecture

The fundamental principle of the OCC system is the modulation and demodulation of transmitted data using optical signal intensity. Effective modulation and demodulation techniques enhance communication efficiency. The performance of an OCC system largely depends on the camera type and image quality. Typically, two types of cameras―CMOS and CCD―are used for data collection in OCC systems. The entire communication process, from data input to transmission, is illustrated in Figure 1 within the transmitter section. Similarly, the steps from image capture to data output are outlined in the receiver section of Figure 1.

2.1 Transmitter:

2.1.1 FEC Encoding

The input data are first encoded in binary using ASCII code. Forward error correction (FEC) encoding is crucial for identifying and correcting errors during data transmission over noisy channels. By eliminating the need for retransmissions, FEC enhances resilience and reliability, particularly in environments prone to interference or signal degradation. Key advantages of FEC include improved error correction, increased transmission reliability, efficient bandwidth utilization, and real-time communication.

A widely used FEC technique for mitigating burst errors in optical communication is Reed-Solomon (RS) encoding, which is particularly effective in OCC systems. We have employed a Reed-Solomon code with parameters n = 15 and k = 11, enabling a 2-bit error correction capability optimized for QAM-4 modulation.

2.1.2 QAM-4

Quadrature amplitude modulation with four levels (QAM-4) encodes data by modulating both phase and amplitude, representing four distinct symbols, each carrying two bits. OCC systems often operate in challenging conditions with low signal-to-noise ratios (SNR) and variable illumination. QAM-4’s relatively large symbol spacing enables robust performance in such environments.

Compared to RF-based communication, OCC cameras typically have lower frame rates and sampling rates, limiting their capacity to handle complex modulation schemes. The simplicity of QAM-4 ensures that data can be efficiently modulated and demodulated with minimal computational overhead. Additionally, QAM-4 requires lower SNR to achieve a specific BER than higher-order modulation schemes such as QAM-16 or QAM-64.

2.1.3 Hermitian Symmetry

In OCC systems, Hermitian symmetry is applied to ensure that the inverse fast Fourier transform (IFFT) generates a real-valued signal suitable for optical intensity modulation. This concept, inspired by mobile communication systems, is crucial for OFDM. Since optical signals represent light intensity (which must be both real and positive), Algorithm 1 introduces a modified Hermitian symmetry approach, incorporating adjustments to the DC component and Nyquist frequency. These modifications are experimentally optimized to Ensure that the IFFT output remains real-valued, preserving signal integrityMaintain a positive intensity signal, essential for optical modulation. By carefully tuning these parameters, the proposed algorithm ensures reliable transmission while adhering to the physical constraintsof OCC systems.

2.1.4 OFDM Modulation

OFDM divides the available bandwidth into multiple narrowband subcarriers, each transmitting part of the data. This technique improves spectral efficiency and minimizes inter-carrier interference (ICI) by maintaining orthogonality between subcarriers. Unlike single-carrier modulation schemes such as on-off keying (OOK) or frequency shift keying (FSK), OFDM enables higher data rates by transmitting multiple data streams in parallel. In this implementation, the cyclic prefix is omitted as OCC systems do not experience significant multipath propagation, which typically necessitates a guard interval in RF-based OFDM systems.

2.1.5 Clipping and Scaling

To enhance signal transmission, a clipping method is applied to remove extreme signal values, ensuring a stable intensity range. The clipping threshold is set based on percentile calculations of the biased signal to balance signal integrity and noise suppression. Before clipping, scaling is applied to adjust signal intensity by: Adding a bias proportional to the signal’s standard deviation. Applying a scaling factor to optimize the intensity distribution.

Algorithm 2 details the clipping and scaling process. The processed signal is then transmitted through the LED source.

2.2 Receiver

At the receiver end, a rolling shutter camera captures optical signals as variations in light intensity. The system employs CMOS image sensors for data acquisition. To ensure accurate signal sampling, the frame rate and exposure time of the camera are synchronized with the transmitted signal. In this OCC implementation, rows of pixels serve as transmissionunits, meaning that synchronization between the camera’s exposure time and the LED's on/off frequency is crucial for accurate data extraction. After image capture, image processing techniques extract intensity variations corresponding to transmitted data. The processed signal is then passed through an AI-based equalizer, which mitigates noise and enhances signal clarity. The signal is subsequently transformed into the frequency domain using the FFT, preparing it for OFDM decoding. To maintain signal integrity, Hermitian symmetry is applied within the OFDM symbol block, ensuring that IFFT operations generate real-valued signals suitable for optical transmission. The QAM-4 demodulator then extracts the original bit sequences from received data symbols. Finally, FEC decoding is applied to correct transmission errors caused by noise and channel impairments. The recovered bit stream is then output as the final data.

Ⅲ. Model and Dataset Description

Traditional equalizers, such as least mean squares (LMS), minimum mean square error (MMSE), and zero-forcing (ZF), rely on heuristic, model-driven approaches that incorporate channel estimation at thetransmitter^[13] to mitigate distortion and noise in optical communication channels. These methods are based on statistical assumptions and often struggle with nonlinearities and time-varying conditions. In contrast, AI-driven models for signal denoising utilize data-driven paradigms, leveraging deep learning or reinforcement learning algorithms to adaptively learnand predict optimal filtering strategies. These models often outperform traditional equalizers by handling complex noise patterns, nonlinearities, and high-dimensional interference more effectively in OCC systems.

3.1 Bi-LSTM

A bidirectional long short-term memory (BiLSTM) network is an advanced recurrent neural network(RNN) architecture (Figure 2) designed to enhance sequential data processing by incorporating both forward and backward temporal dependencies^[14]. Unlike conventional LSTMs, which capture only past dependencies, BiLSTM employs two hidden states―one processing the input sequence in a forward direction and another in reverse―allowing for a more comprehensive contextual understanding

Mathematically, given an input sequence [TeX:] $$\begin{equation} X=x_1, x_2, \ldots, x_T \end{equation}$$, the forward LSTM computes hidden states as:

while the backward LSTM processes the sequence in reverse:

The final hidden representation is obtained by concatenating both states:

where [TeX:] $$\begin{equation} h_t^f \end{equation}$$and [TeX:] $$\begin{equation} h_t^b \end{equation}$$ are the hidden states from the forward and backward LSTMs, respectively. The final output o_t is computed as:

In signal denoising, BiLSTM is particularly effective in capturing bidirectional dependencies within noisy sequences, enabling robust feature extraction and temporal distortion mitigation. By leveraging both past and future contexts, BiLSTM effectively attenuates noise-induced anomalies while preserving essential signal structures, making it highly suitable for OCC equalization and real-time signal processing applications.

3.2 ResNet

Residual neural networks (ResNet) introduce skip connections to mitigate the vanishing gradient problem, facilitating the training of deeper architectures. The residual learning framework allows the network to approximate identity mappings, preserving critical signal components while suppressing noise artifacts. This capability makes ResNet particularly effective for signal denoising tasks, where the goal is to reconstruct a clean signal from a noisy counterpart. The architecture of ResNet is depicted in Figure 3. The fundamental building block of ResNet is the residual block, defined as:

where x_lis the input to the l^th residual block, [TeX:] $$\begin{equation} F\left(x_l, W_l\right) \end{equation}$$ is the transformation function consistingof convolutional layers, batch normalization, and activation functions, W_l represents the trainable parametersof the block, and y_l is the residual block output after the addition operation. In the absence of noise, ResNet approximates the identity function, ensuring signal fidelity. However, in the presence of noise, it learns an adaptive residual mapping that selectively suppresses noise components while preserving salient features. This self-regularizing property enables ResNet to achieve superior performance in denoising tasks, particularly in OCC and other wireless communication scenarios where signal degradation is a significant challenge.

3.3 Dataset

The dataset used in this study consists of empirically acquired real-world data that accurately represents OCC scenarios. The data collection process follows an image-based acquisition framework, wherein optical transmission signals are initially captured as images. These images undergo a decoding procedure to extract the received data (Rx data) while simultaneously reconstructing the transmitted data (Tx data) from the original messages. The dataset includes raw received optical signals, along with corresponding labels that facilitate the benchmarking of different equalization methodologies. To ensure a rigorous and unbiased evaluation of signal processing models, the dataset is systematically divided into training, validation, and test subsets. This structured partitioning enhances the robustness and generalizability of the equalization models under development. Furthermore, the dataset serves as a valuable resource for advancing research in channel equalization, adaptive channel tracking, and data detection techniques. The data is structured in a (16,8) format, where eight subcarriers correspond to the OFDM system configuration, and 16 symbols represent transmission units. The training dataset consists of 89,949 samples with dimensions (16,8), while the test dataset contains 22,347 samples with the same structural representation. This dataset serves as a foundational benchmark for evaluating and improving OCC equalization strategies, ultimately contributing to the development of more efficient and resilient optical wireless communication systems. The dataset is specifically curated to evaluate advanced neural network architectures for signal restoration, mapping, and equalization. Among the tested models, BiLSTM demonstrates a superior ability to capture long-range temporal dependencies due to its bidirectional nature, enabling it to denoise signals and track complex time-varying channel behaviors with high precision. BiGRU, while computationally lighter, offers comparable performance in modeling sequential patterns, making it suitable for low-latency embedded applications. On the other hand, ResNet leverages deep residual connections to effectively learn spatial features and denoise structured signal distortions, especially when input data is represented in matrix or image-like form. Together, these models benefit from the dataset’s structural richness and variability, facilitating the exploration of novel equalization frameworks and paving the way for more resilient and adaptive OCC systems.

3.4 Evaluation Matrix:

Evaluation metrics are essential for assessing the performance of signal denoising and equalization models, providing quantitative benchmarks for accuracy, reliability, and generalization capability. These metrics enable objective comparisons between competing models, facilitating the selection of the most effective approach for real-world deployment. Commonly used evaluation metrics include mean squared error (MSE), root mean squared error (RMSE), R-squared score (R²), and Pearson correlation coefficient (PCC). The MSE measures the average squared difference between actual and predicted values, emphasizing larger deviations by penalizing substantial errors more severely^[15]. RMSE, derived as the square root of MSE, restores interpretability by expressing errors in the same units as the original data, making it a more intuitive metric for predictive performance. Lower values in these metrics signifythat the model effectively mitigates distortions introduced by the channel, leading to fewer symbol misinterpretations and consequently a lower BER. The R² Score, or coefficient of determination, quantifies the proportion of variance in the dependent variable explained by the model, providing insight into its explanatory power and goodness-of-fit^[16]. A high R² score suggests that the model captures the structural features of the signal with high fidelity, which is essential for accurate demodulation and symbol decoding in OFDM systems. PCC evaluates the linear correlation between actual and predicted values, with values approaching ±1 indicating stronger alignment with the ground truth. A PCC value close to 1 implies strong coherence in signal shape and pattern, which is critical for preserving the integrity of phase and amplitude information in the demodulated output ultimately reducing symbol errors that contribute to BER. Together, these metrics ensure that the developed models are robust, precise, and suitable for practical applications, ultimately enhancing signal processing and equalization techniques in communication systems. The mathematical equations for these evaluation metrics are as follows:

where y_iis the actual value [TeX:] $$\begin{equation} \widehat{y}_l \end{equation}$$, is the predicted value, n is the total number of samples, [TeX:] $$\begin{equation} \bar{y} \end{equation}$$ is the mean of actual values.

Ⅳ. Experimental Setup

The OCC system was experimentally implemented using an Arduino Uno microcontroller as the primary controller for signal modulation and transmission. The experimental setup and system configuration are illustrated in Figure 4. A single 5V Light Emitting Diode (LED) functioned as the optical transmitter, with its driving circuit enhanced by an IRF520N MOSFET driver module. This module was necessary because the Arduino Uno’s output voltage and current were insufficient to drive the LED directly, requiring an external amplification stage. The system was operated through a Python-based framework, which managed message encoding and OFDM modulation. The modulated digital signal was transmitted to the Arduino Uno, where it was converted into an electrical signal within a 0 to 5 V range, making it suitable for optical transmission. The LED emitted modulated optical pulses at a carrier frequency of 16 kHz, enabling data transmission over a 3-m communication distance. On the receiver side, a Basler global shutter industrial camera (Basler 1920-25 μm) equipped with an automatically adjustable zoom lens captured the transmitted optical signals. The global shutter mechanism was essential for reducing motion blur and ensuring precise frame acquisition. Python (version 3.7.15) was employed for signal processing and decoding, with the OpenCV library playing a key role in video frame extraction, frame rate adjustment, and parameter optimization.

Ⅴ. Results and Discussions

The raw image captured by the camera in the OCC system represents intensity fluctuations detected by the image sensor row by row, as shown in Figure 5. To mitigate synchronization issues between the transmitted signal frequency and the camera frame rate which result in pixel intensity reductions in received image zero-padding was applied between successive broadcast data points. A Python script was then used to process and decode the captured raw data, extracting the transmitted information. However, due to factors such as channel noise, variations in performance among deep learning models were observed. Figure 6 presents a comprehensive visualization of the signal processing and analysis workflow within the optical camera communication (OCC) system. Subfigure (a) shows the decoded binary data extracted from the received signal, which is structured in packets with identifiers, reflecting successful demodulation and decoding. Subfigure (b) displays the QAM-4 constellation diagram, where the clustering of points indicates the quality of signal demodulation and the effectiveness of the applied equalization. Subfigure (c) depicts the AI model loading process, showing real-time inference progress, which highlights the deployment of deep learning models during signal decoding. Subfigure (d) illustrates the received intensity pattern captured by the camera, emphasizing the image-based nature of OCC data acquisition. Table 1 presents a comparative analysis of three models

ResNet, BiLSTM, and BiGRU based on four evaluation metrics: MSE, RMSE, R², and PCC.

Among them, BiLSTM demonstrated superior performance, achieving the lowest MSE (0.72%) and RMSE (7.24%), alongside the highest R² score (86.75%), indicating greater accuracy and predictive capability. Additionally, its PCC value (0.95) signified a strong correlation with the target signal. While ResNet outperformed BiGRU, it still lagged behind BiLSTM. BiGRU, in contrast, exhibited the highest error values, suggesting weaker performance in this specific task. Figure 7 compares the denoised OFDM signals produced by BiGRU, ResNet, and BiLSTM against the actual transmitted (Tx) signal and the raw received (Rx) signal. The BiLSTM model (dashed green line) closely follows the actual transmitted signal (solid blue line), demonstrating its superior noise reduction capability. While ResNet and BiGRU improve signal quality, they exhibit slightly greater deviations from the transmitted signal than BiLSTM. Incontrast, the raw received signal (dotted line) shows the most distortion, highlighting the necessity of deep-learning-based equalization techniques to enhance signal fidelity in optical communication systems. In terms of communication efficiency, the system achieved a data rate of 6.2 kbps at a modulation frequency of 16 kHz, with a corresponding bit error rate (BER) of 2.88 × 10^-3. These metrics were calculated based on the number of correctly decoded frames captured by the camera. While raw signal processing without equalization led to considerable bit loss due to channel noise, the incorporation of deep learning equalizers significantly mitigated these errors, enhancing both decoding accuracy and data throughput. Furthermore, the robustness of the proposed system was evaluated by collecting and testing data under various environmental conditions, including different lighting settings, transmission angles, and distances. The consistent performance of the BiLSTM model across these conditions demonstrates its adaptability and generalization capability, thereby substantiating its suitability for real-world OCC applications. The proposed AI-equalized OFDM OCC system not only achieved reliable data recovery under noisy conditions but also addressed key system limitations, such as synchronization mismatch and channel-induced degradation.

Ⅵ. Conclusion

This study successfully designed, implemented, and evaluated an OCC system integrating OFDM modulation with a BiLSTM-based equalization framework. The experimental results demonstrated a significant reduction in BER to 2.88 × 10−2, validating the effectiveness of the BiLSTM equalizer in mitigating noise and enhancing signal fidelity. Additionally, the system achieved a data transmission rate of 6.2 kbps at a modulation frequency of 16 kHz, confirming its robustness in optical wireless communication. To address synchronization discrepancies between the transmitted signal and the camera’s frame rate, a zero-padding mechanism was incorporated, improving intensity capture, signal reconstruction, and decoding accuracy. Comparative performance analysis revealed that BiLSTM achieved the lowest MSE (0.72%), RMSE (7.24%), and the highest R² (86.75%) and PCC (0.95), surpassing ResNet and BiGRU in denoising efficiency and signal recovery. These results underscore the potential of deep-learning-based equalization techniques in optimizing OCC systems by enhancing communication stability, OFDM modulation efficiency, and key system parameters such as exposure duration, sampling rate, and transmitter configurations. This research establishes a strong foundation for future advancements in high-fidelity optical wireless communication technologies, paving the way for more efficient and reliable data transmission in next-generation OCC systems.