New Meta AI turns text prompts into explorable VR worlds

Another day, another leap in generative AI development from big tech. Meta Reality Labs has announced a research project that takes a step towards making text to world generation a reality.

WorldGen is designed to generate stylistically coherent explorable 3D worlds from a just one text prompt. The idea is that these could be used for VR development in engines like Unreal Engine 5 or Unity (see our guide to game development software), although the research paper reveals that there's a long way to go.

According to the WorldGen research paper, Meta's AI model can create scenes spanning up to 50 × 50 metres that users can walk around and explore, They can also be exported for use in real-time engines.

The pipeline combines procedural layout generation, image-based planning, diffusion-driven 3D reconstruction, navmesh extraction, scene decomposition, mesh refinement, and texturing. WorldGen can also break scenes down into objects via an accelerated AutoPartGen process, which is intended to make the environments generated more editable and reusable.

Image 1 of 3

But WorldGen currently has significant limitations. It can only generate a single reference view, limiting the scale of the scenes that can be produced.

It doesn't natively support larger open worlds spanning kilometres rather than metres. This would require generating and stitching multiple local regions, which could introduce non-smooth transitions or visual artifacts at the boundaries, the research paper notes.

The platform can't model multi-layered environments, such as multi-floor dungeons or interior-exterior transitions, either. And it can't reuse textures or geometry, which limits its efficiency and hence its usability for development.

Some devs have noted that they need textures to be shareable between objects. Meta says it's looking at how it could add reusable textures in future versions to make the platform more scalable.

Other developers have suggested that the paper makes it look like the generations tend to be repetitive with elements always arranged in a grid pattern.