Game engine creator Unity has a plan for AI that is "empowering" and "responsible"

Nico Perony: Unity is uniquely positioned to help creators succeed by injecting generative AI in their existing workflows. Unity AI (Unity Muse and Unity Sentis) is designed to work seamlessly with the ecosystem of Unity Editor, Runtime, data, and the Unity Network.

Going forward with Unity 6 (formerly LTS 2023), AI will be a major part of the techstream. Additionally, for Unity Muse, our AI models are built to allow game developers to create assets without worrying about challenges with IP and data ownership.

NP: Our goal with Unity Muse (AI tooling options) is to meet developers where they are, help them, and keep them in control. We want them to be able to iterate and experiment faster so they can find the fun in their games and experiences.

Muse Chat in the Editor lets you use natural language prompts to find exactly what you’re looking for (see our recent blog post). In seconds, you’ll have well-structured answers and instructions, and even project-ready code. Using the Muse Sprite capability, you can produce and modify 2D art directly in the Editor.

With the Muse Texture tool, you can also generate production-quality textures for 2D and 3D projects, in any style, all inside your project in the Editor. Just like our other tools, we expect to embed our AI tools – like Unity Muse – in the Unity workflow so it's there when you need it if a developer chooses to leverage AI in their workflow. 

Unity Sentis enables developers to bring complex AI data models into the Unity Runtime to help solve complicated tasks and create new functionality in a game using AI models. Just like a physics engine is one of the tools to bring games alive, we also believe that you‘ll need a neural engine to solve some complicated tasks and or create new functionality in your game, with AI.

Sentis is a cross-platform solution for embedding AI models into a real-time 3D application, so you can build once and deploy AI capabilities in your Unity app from mobile to PC and web to popular game consoles. Since you’re running your program on the user’s device, there’s no complexity, latency, or cost associated with model hosting in the cloud, and the data from your users stays safely within the app.

NP: We have built the suite of AI tools to fully integrate with existing capabilities of the Unity Editor. Our creators do not have to use AI, but they will feel empowered and noticeably faster when they create with the help of AI. For beginners, it can be as simple as asking questions to Muse Chat while building a project in the Editor.

For more advanced users, they can train their own models in their framework of choice (for example PyTorch) and embed them as ONNX files in their build with Unity Sentis. We cater to all creators, whether they are just starting on their AI journey or well-versed in machine learning development.

NP: With Unity Muse and Unity Sentis, we are meeting game developers where they work, and that’s in the Unity Editor. We are aware of the multiplicity of AI tools and platforms existing out there, and we are bringing a select and ethical curation to Unity developers.

For example, with the bridge we have built between Unity Sentis and the popular AI platform Hugging Face (blog post), Unity creators can easily embed existing models in their application without having to worry about model discovery, optimization, and deployment. 

NP: Unity’s AI tools, like the rest of the Unity ecosystem, are made to be used at all stages of the game development life cycle, and by teams of all sizes. Tools like Muse Sprite, Muse Texture, or Muse Animate will make prototyping easier, especially for smaller teams.

Tools like Muse Chat or Unity Sentis will enable production teams to efficiently write and audit functional and performant code, and embed AI in the runtime of their application.

We are focused on managing our development of AI capabilities in a transparent and responsible manner. You can see that in our blog post on responsible AI. We also released a blog post showing how we curate and create ethical datasets to train the models powering Muse Texture and Muse Sprite.

Our approach is to think through how we can deliver tools that are easy for creators to use, responsibly sourced, and with output that creators can feel confident in using in their projects. 

We are building AI tools to help creators when they are stuck or want more iterations and give them more tools they can use – if they choose. Our goal is to keep the creator in control and in the centre of creativity.

Just like our other tools like SpeedTree, which uses procedural generation to build plants and vegetation, we expect Unity Muse to be another tool in an artist or developers toolbelt.