Generative 3D creation with AI prompts

So, Sloyd is a 3D automation tool for instantly generating game-ready assets, and we have just gone live with AI prompting. We’ll tell you all about it, but first, let’s take a step back and look at the state of AI in gaming and specifically in modeling.

How AI is used in game development and in 3D modeling

I bet you’re already using AI to some degree in your game development workflow. According to a survey conducted by A16Z Games, 87% of studios are using an AI tool or model as of today. Yet, the use right now is mostly on the periphery, as the vast majority of studios are using horizontal, non-game-specific tools (ChatGPT/GPT-3/4; Midjourney; Copilot; etc).

To this point, AI has been adopted to varying degrees for concept art ideation, coding for games, creating textures, NPC scripts, sound effects, and motion capture. The area with the least adoption is 3D modeling. It’s no wonder; when it comes to 3D models for games, we have high standards. A game asset not only needs to look good but also needs to perform.

The key to a game-ready asset is good topology. The term “topology” refers to the distribution and structure of vertices, edges, and faces of a 3D model. The topology shows how well vertices are organized (cgiscience). A model with bad topology will be created with many more polygons than necessary to serve its artistic purpose. It would simply be needlessly heavy, limiting the amount of art you render in a single frame.

More often than not, when taking assets from another source (a marketplace, an older studio game, or a generated model), we need to make modifications and adjustments. Working with an asset with bad topology will also be costly. It will be much harder to work with the UVs and make changes to the textures. It would also be difficult to modify parts of the mesh and even harder to animate without stretching or shrinking parts of the asset.

Looking at the outputs of the best-in-class GenAI research, Get3D, and Dreamfusion, the topology is very far from optimal. They don’t look great, they gobble polygons, and are extremely painful to work with.

A third factor in using generative 3D models is the creation time. A result can take an hour. One might argue that an hour is a lot less than creating a model from scratch. But often you need several iterations before you get a satisfactory result, and if each iteration takes an hour, the overall tradeoff of time versus quality is a lot less compelling.

Crab 3D model taken from Dreamfusion inspected in Blender. (Left – UV, right – vertices)

A parametric approach is the key to instant, game ready assets

Sloyd adopts a ‘Lego pieces’ approach to 3D automation. It employs parametric 3D geometries as its base building blocks and assembles them into 3D assets. On the one hand, this approach might appear limiting as the system can only create from the building blocks it has been ‘fed’. However, it carries a significant advantage: the results are guaranteed to possess clean topology, easily manageable UVs, separable parts, and optimization for performance. The Sloyd team is actively expanding its library of building blocks, aiming to enable the creation of a wider range of assets. While Sloyd doesn’t cover everything yet, it ensures that every model you create is game ready.

Lamp 3D model taken from Sloyd inspected in Blender. (Left – UV, right – vertices)

Adding to its capabilities, it’s incredibly fast! Input your text, and you’ll receive a result in mere seconds. Following that, you can make adjustments to items using simple inputs like sliders, and the changes occur in real-time. Currently, each text prompt yields a new object. So, if you’re dissatisfied with the outcome, refining your text will yield a new model. However, in the near future, you’ll be able to iterate on a model using text descriptions, and all of this will be in real-time.

AI prompting of models, step-by-step

Signing up for Sloyd is free and you’ll get to creating in a few seconds. There are a few ways to start creating with Sloyd, and the newest way is AI prompt. Just describe what you want to create in simple words. Include the object’s name and one or two adjectives. To achieve optimal results, focus on props, weapons, furniture and buildings. Try something like: “Well with an oriental roof”, “spaceship with X-wings” or “big flying saucer”.

Sloyd, step 1: AI prompt

A model will spawn on the canvas and a new side menu with sliders, toggles and buttons, will open.

• Buttons: At the top, you’ll find selectable parts that change areas of the model.
• Sliders: Adjust standard parameters such as height, width, and curvature using sliders. These parameters allow you to manipulate the shape of your model.
• Toggle Parts: Toggle different parts of the model on or off to create unique variations. Experiment with different combinations to achieve the desired look.
• Advanced Options: Delve into the details to access more advanced options for fine-tuning your model’s appearance. You might encounter sliders that drastically alter specific features or parts of the model.

Sloyd, step 2: iteration with sliders and buttons

One of the standout features of Sloyd is the Randomizer. This tool provides an excellent starting point for your design. Click the Randomizer button multiple times to generate various design iterations.

• Explore unexpected and creative design possibilities that you may not have considered.
• Find a starting point that resonates with you. It could be a unique design element or an intriguing shape.

Once you’ve crafted your ideal 3D model, it’s time to add the finishing touches:

• Materials and Colors: Navigate to the Materials section to experiment with different colors and textures. Transform your model’s appearance by selecting vibrant, bold colors or more subdued tones.
• Preview Your Changes: As you make adjustments, the outline of your model will update in real-time. This feature allows you to visualize the changes you’re making and make informed decisions.
• Export Your Creation: When you’re satisfied with your masterpiece, select the object and click the Export button. Choose the desired format for your export, such as GLB or OBJ.

What’s coming up in Sloyd in terms of AI and 3D modeling

While our AI prompting is still in the experimental stage, it’s continually improving. Yet, this is just the initial phase. The upcoming stage involves introducing the capability to iterate on a model using text. Subsequently, our plan encompasses enabling the AI to comprehend color and material changes, as well as generating and integrating AI textures. Our big leap will be AI spatial assembly of 3D models. Once accomplished, the system will autonomously select ‘Lego pieces’ from a virtual ‘warehouse’ and meld them to craft a 3D model without external guidance. This, coupled with an extensive and perpetually expanding ‘warehouse,’ will render the potential to create pretty much everything. As we extend this capability from crafting singular models to multiple models simultaneously, we’ll have AI world creation from the ground up.