The U.S. Marine Corps (USMC) has taken the initiative of introducing interactive learning experiences at its training centers as a cost-effective and timesaving means to augment classroom instructions and physical equipment training with immersive maintenance and safety training in a simulated environment. However, the techniques to create the 3D models for immersive environments, which use Computer Aided Design, graphics software, 3D-scanning, and Photogrammetry, require software skills, manual effort, time and financial investments. The USMC has the need to rapidly build a repository of ready, reusable and manipulable 3D models of their assets in a scalable manner. Recent advances in generative Artificial Intelligence (AI) can fill this need by rapidly generating approximate but realistic 3D models from available 2D pictures of equipment found in existing USMC training guides such as presentations and student handouts.

In this paper, we present a scalable and automated content generation process that uses an ensemble of vision-based generative AI techniques to convert 2D images into 3D models based on appropriate tradeoffs between desired levels of quality and computational complexity. We will leverage an existing foundational 2D-to-3D conversion model trained with large and diverse web-scale data for “few-shot” transfer learning with domain specific data. The 3D content-generation process will use open-source software, and incorporate intuitive user interfaces to minimize the need to learn machine learning (ML) or graphics programming. The resulting 3D objects can be directly imported into reusable libraries for use across various schoolhouse applications requiring immersive training content.

This paper documents the results from performance experiments that convert a wide array of images from a USMC schoolhouse course with varying degrees of complexity, and benchmarks various vision-based AI/ML techniques with respect to object fidelity and speed of conversion. The paper concludes with best practices and lessons learned from these content conversion experiments.

Keywords
3D;AI;AUGMENTED AND VIRTUAL REALITY (AR/VR);AUTHORING TOOLS;CONTENT GENERATION;DEEP LEARNING;EMERGING TECHNOLOGIES;IMMERSIVE;MACHINE LEARNING;MODELING;TRAINING

Additional Keywords