Real2Sim

As the robotics revolution accelerates, one of our key visions is to combine simulation with real world testing across all: from control systems to AI model development, to verification and validation (V&V)

The nice image comes straight from the actual research & development results. From right to left, you can see the Gaussian Splatting training process progressively improve the 3D reconstruction.

Bring Real data into Simulation (Real2Sim) and to... Movies.

Bridging the gap of Real and Simulation (Real2Sim - Sim2Real) toward engineering values is interesting and motivating to see how far we could push the boundary. Take a look at our 3D scene reconstruction demo in a parking scene, compare virtual and real side by side.

When reconstructing 3D traffic model from real world data (Real2Sim), one of the challenges is incorporating physical properties such as materials and textures.

Great discussions with Prof. Holger Caesar (Intelligent Vehicles Group, TU Delft) with his visiting to our R&D team in Siemens Leuven office last Friday.

Transform the vehicle sensor data into a large AI neural network model. The Gaussian Splatting model captures very well all details of the street from other cars, road terrains, trees, buildings… with high texture, color performance.

While simulation continues to play an important role in autonomous driving development and validation, one of the main challenges is to have a credible simulation environment and data.

Data is very valuable for autonomous driving/ADAS engineers, i.e. training & validation of ADAS functions such as perception, planning, control.

Autonomous vehicle dataset such as nuScenes, Waymo Motion, or your own collected one, is known valuable for algorithm development and testing. What's more?

Bring real collected data into simulation (Real2Sim) is essential activity in ADAS testing, allow engineers to test different sensor, vehicle dynamics models, as well as various what-if traffic scenarios.

ADAS or autonomous driving dataset is captured from vehicle sensors, and often used for perception algorithms or motion prediction. However, logged data is often passive, static, and open-loop. It is not able to actively adapt or evolve, for example with respect to the new sensor types. As the data is with human driver providing vehicle actions, closed-loop control testing is not possible.

Significant efforts have been also devoted to building and tuning a large and high quality 3D Munich city model (on realism, structure, texture, material). The model is then optimized into a real-time VR platform in Unity, and also Simcenter Prescan to exploit physics-based sensors (lidar, camera, radar) and perception-control algorithms.