Multi-User Augmented Reality-Based Aerospace System Design Review Platform

Ⅰ. Introduction

Augmented reality (AR) technology superimposes virtual objects or information on real-world environments, enabling users to interact with these elements as if they were part of the physical world. With the advent of the fourth industrial revolution and the acceleration toward a contactless society spurred by the COVID-19 pandemic, the demand for this technology has surged. Its applications have expanded beyond the IT sector, permeating diverse realms such as entertainment, gaming and film―service industries such as tourism and travel, and other sectors such as automotive and real estate^[1].

In the aerospace industry, AR has revolutionized system assembly and maintenance by enhancing efficiency and improving employee safety. For instance, Boeing employed AR technology in aircraft electrical wiring tasks, and Lockheed Martin integrated it into the assembly process of an F-35 aircraft. By overlaying work-related diagrams and component assembly instructions on AR glasses worn by workers, these companies facilitate a more intuitive understanding of tasks, thereby increasing both accuracy and efficiency^[2,3]. Another notable application exists in maintenance and repair: Klatt Works offers an AR device outfitted with a checklist for aircraft repairs, technical documentation, operational blueprints, and on-site recording capabilities^[4]. Furthermore, JAXA initiated a project utilizing AR devices to guide crew members in maintaining equipment on the International Space Station (ISS), particularly when communication with Earth was restricted^[5].

AR technology offers further innovation potential for aerospace system design^[6]. For example, the Korea Space Launch Vehicle-II (KSLV-II) successfully completed its flight test in June 2022. Since 2010, this project has undergone numerous design reviews and modifications, as illustrated in Figure 1. These reviews include the System Requirements Review (SRR) in 2010, the System Design Review (SDR) in 2012, the Preliminary Design Review (PDR) for individual stages in 2013, the Engine and Subsystem Preliminary Design Review in 2014, the Critical Design Review (CDR) in 2016, and the Engine and Subsystem CDR in 2017^[7]. Over this extended period, various stakeholders, particularly those new to the project, may have faced challenges in understanding complex aerospace system designs intuitively.

This study introduces a novel aerospace system design review platform that integrates AR technology to address these challenges. Within the proposed platform, users can visualize complex hardware structures, such as rockets, satellites, and aircraft, in three-dimensions (3D) via AR, thereby facilitating a more visually intuitive understanding. Moreover, multiple users can participate in a virtual meeting environment to share and review AR-based 3D data.

Ⅱ. AR-based Aerospace System Design Review Platform

Figure 2 illustrates the architecture of the proposed AR-based aerospace system design review platform. A virtual meeting room was set up in which multiple users equipped with AR devices such as head-mounted displays (HMDs) could join the session. The virtual meeting room was connected to a server that housed the 3D data related to aerospace systems. Within an AR-enabled virtual meeting environment, users could access 3D data from the server to physically disassemble and reassemble aerospace system hardware, thereby facilitating realistic 3D design review sessions.

· To conduct AR-based design review meetings in such a virtual environment, the following criteria must be met:

· The 3D data must be optimized for real-time rendering.

· The optimized 3D data should be stored on the server, which then interfaces with the AR devices and applications.

· Users are required to wear AR devices such as HMDs to interact within the virtual space.

· The virtual meeting room (or AR application) must offer features for the assembly and disassembly of 3D data in addition to supporting multi-user access and bidirectional communication.

In the following section, we describe the implementation of each requirement.

Ⅲ. Implementation Results

3.1 3D data optimization

Computational performance in virtual environments is significantly influenced by geometric features and intricacies inherent in 3D data^[8]. An overly detailed geometry can result in performance issues such as stuttering during screen transitions, owing to limitations in computational resources. This can lead to discomfort or symptoms similar to those of dizziness. Conversely, overly simplified representations can compromise the user's ability to comprehend the intricate structural details of an object. Therefore, achieving a balanced level of detail in 3D data is crucial. To achieve this balance, various techniques, such as tessellation, mesh simplification, and texture baking, are employed. As a practical example, 3D CAD data for lunar rovers and landers, which are currently under research and development at the Korea Aerospace Research Institute, have been used to demonstrate these techniques^[9,10]. Additionally, Pixyz Studio software was used for these optimization processes^[11].

3.1.1 Tessellation

Tessellation involves converting the surface of 3D data into a mesh structure consisting of polygons composed of several triangles. The primary objective of tessellation is to simplify the processing and rendering of 3D data while significantly reducing file size. Figure 3 presents an example of tessellation conversion applied to 3D data in the STEP format.

The use of a large number of polygons can produce a more realistic representation of an object; however, it also requires increased computational time and resources. Therefore, it is crucial to determine the optimal tradeoff between these competing factors. The appropriate number of polygons for tessellation can vary based on several criteria: the geometric features inherent in the 3D data, the computational capability of the hardware being used, and the desired quality level as determined by the user based on empirical results.

3.1.2 Mesh simplification

Mesh simplification can be achieved through the following techniques:

· Decimation: This involves reducing the number of polygons by altering or regenerating the mesh of an object through a retopology.

· Hidden Removal: This method eliminates parts that are not visible from a specific viewpoint, significantly reducing real-time computational requirements and, consequently, memory and power consumption.

· Face Repair: This technique addresses surface inconsistencies primarily by correcting rough and uneven 3D data.

· Hole Removal: This process eradicates gaps or holes in the 3D data either by adjusting the vertices of the existing polygons or by filling in the holes.

Figure 4 presents a comparative example illustrating the mesh structure of the 3D data for a lunar lander before after the application of the aforementioned simplification techniques. The figure shows various components of the lander, including the side ladder panel, head, and legs. The goal of simplification was to retain as much of the original shape as possible while minimizing the number of polygons used. Table 1 provides a comparative analysis of the vertex, edge, and triangle (tris) counts before and after mesh simplification, indicating reductions of 68.34%, 85.26%, and 69.89%, respectively.

3.1.3 Texture bake

After completing the mesh simplification process outlined in the previous section, textures were applied to the surface of the 3D data. The creation of a UV map is essential for this process. A UV map serves as a '2D unfolded representation' of a 3D mesh, facilitating the application of textures.

Based on this UV map, texture maps were generated using the physically based rendering (PBR) method. This approach involves creating texture maps that emulate the physical properties of a material, such as light diffusion, reflection, and glossiness, to produce realistic 3D objects. The key types of texture maps are as follows:

· Albedo map: Provides color information and measures how an object reflects and absorbs light.

· Metalness map: Represents the object's degree of metallicity.

· Roughness map: Depicts the surface roughness of the object.

· Normal map: A specialized type of bump map that uses light distortion to represent surface details, such as bumps, indentations, and scratches.

The process of consolidating multiple texture maps into a single image file is known as texture baking. Once texture baking was completed, it served as the basis for generating the 3D rendering models. Figure 5 illustrates the UV map, various texture maps, and the resulting 3D rendering model for the lunar rover and lander after texture baking has been applied.

3.2 3D data storage server

Given the limited memory capacity of AR devices, it is impractical to store all 3D data directly on the device. Therefore, a dedicated server is required to store and manage 3D data separately. This server can be either physical or cloud-based. Furthermore, the server must be integrated with an application that enables access to a virtual meeting room through an AR device. We selected a local hardware server and developed a web-based server user interface (UI) to enhance user accessibility. The UI has been designed in two distinct versions: one for administrators and the other for end-users.

3.2.1 Administrator UI

As shown in Figure 6, the administrator UI provides features for managing users, groups, and resources. Administrators can view all the user names, groups to which these users belong, and their registration and login histories. With administrative privileges, they can create, edit, or delete specific user information. Management options extend beyond the individual user-level to include group-level controls. For example, license periods can be set for specific groups to access platforms. Additionally, administrators can access 3D data files uploaded by users. With granted administrative authority, they can edit or delete specific files, thereby aiding in the efficient management of memory resources.

3.2.2 User UI

Figure 7 shows the initial screen of the user’s UI. After logging in, users can upload 3D data files to the server for use in the virtual meeting room. Because the AR device's virtual meeting room application is integrated with the server, it enables users to easily access and employ the uploaded 3D data files. Figure 8 shows the interface designed to upload these 3D data files. To simplify the upload process, a drag-and-drop function is incorporated to support a wide array of file formats, including FBX, GLTF, and OBJ.

3.3 AR device

To enable seamless access to a virtual meeting room and to facilitate effective design review meetings, users must wear AR devices with essential functionalities. For optimal user engagement, AR devices should feature capabilities such as data visualization, eye tracking, hand tracking, and voice recognition. The data visualization function allows users to perceive virtual 3D objects as if they were physically present. Eye tracking dynamically recognizes and follows the movement of the user's pupils, adjusting the display in real-time to offer an immersive experience. Hand tracking captures and interprets hand movements and facilitates interactions with virtual objects. Voice-enabled features serve as audio input-output interfaces, simplifying communication among multiple users in a virtual meeting room. In this study, Microsoft's HoloLens 2, which integrates all these functionalities, was employed^[12].

3.4 AR-based meeting application

This section outlines an application designed to enable users with HoloLens 2 devices to access a virtual meeting room and participate in multiparty meetings. Users can join a virtual meeting room using login credentials pre-approved by the administrator. Figure 9 shows how the meeting organizer can set up a meeting room that specifies both the title of the meeting and the number of participants. Furthermore, as illustrated in Figure 10, the organizer can grant certain users the right to speak or control a 3D model within a meeting space.

Although multiple users can possess speaking rights in a single meeting room, the authority to control a 3D model is restricted to one user at a time. This ensures a clear and focused experience akin to conventional online meetings where only one user's presentation screen is visible at a time. Allowing multiple users to control the 3D model simultaneously could lead to confusion and compromise the visual clarity.

When a user raises their left palm upward, the hand-tracking feature of HoloLens 2 detects the gesture and displays an editing mode window, as shown in Figure 11. Through the “Generate” tab, users can upload and augment 3D models stored on the server. In a virtual meeting room with a user count (n) of three or more, a 3D model is augmented at the geometric centroid based on the spatial distribution of the user locations. Specifically, if the coordinates of each user are arranged in either a clockwise or counterclockwise manner, as in [TeX:] $$\left(x_1, y_1\right),\left(x_2, y_2\right), \cdots \left(x_n, y_n\right),$$ the 3D model is augmented in the positive z-axis direction according to the centroid [TeX:] $$\left(C_x, C_y\right)$$ of the closed polygon in the plane formed by these coordinates. The geometric centroid was calculated as follows^[13]:

where A is the area of the polygon, calculated using Gauss’s Shoelace formula, as follows:

This approach enhances the spatial representation within the virtual meeting environment. Figure 12 shows an example in which both the lunar rover and lander are augmented. Users can move the augmented 3D model in any direction, rotate it 360°, resize it, or disassemble its components. Figure 13 shows instances of moving and disassembling the rover, enabling comprehensive hardware design reviews.

Ⅳ. Conclusions

In this study, we introduce a platform that allows multiple users to participate simultaneously in an AR-based virtual meeting room for aerospace system design reviews. To provide a seamless experience for sharing augmented 3D data within this virtual environment, we developed a comprehensive 3D data optimization process. The platform features distinct interfaces for both administrators and end-users on a server dedicated to 3D data storage. Moreover, we designed a specialized application for HoloLens 2 that was fully integrated with the server in question. This application equips users with tools to edit augmented 3D models and disassemble their components, thereby enabling a detailed design review. To demonstrate the end-to-end capabilities of the platform, we used the 3D data of a lunar rover and lander as test cases.

The proposed platform eliminates spatiotemporal constraints, thereby facilitating a seamless information exchange among users. It also supports the real-time 3D visualization of complex aerospace systems, which could potentially enhance users' spatial cognition. Consequently, the pace of design reviews and modifications can be substantially expedited. Furthermore, leveraging this platform for performance validation in a virtual setting before the real-world deployment of aerospace systems presents an opportunity to proactively identify and address potential challenges. This could, in turn, contribute to reducing the overall development costs.