Recent work from Meta shows there is a tradeoff between realism and diversity. “The more realistic a generative model becomes, the less diverse it becomes”, Yann LeCun: https://lnkd.in/eK2UrCb4. I suspect more similar tradeoffs will be explored when we keep pushing for realism vs engineering properties…
At Siemens Digital Industries Software, we keep exploring and exploiting GenAI methods for engineering applications, especially when combine with Simulation and Digital Twin at Siemens Simcenter. The combination adds extra physical dimension to potentially overcome above limitations, i.e. simulation could use for controlnet (next to text prompts) and other guidance.
This demo shows an example for autonomous driving video generation from our R&D ADAS team recently. Together with Simcenter Prescan, the trained model can create realistic diverse, safety-critical driving scenarios from an original nuScenes video. The interesting results (not perfect yet) are with contributions from two talented Master thesis students at Siemens, Jasper van Leuven from Delft University of Technology and Samuele Ruffino from ETH Zürich.