Stable Diffusion WebUI 3D Model&Pose Loader
A custom extension for AUTOMATIC1111/stable-diffusion-webui that allows you to load your local 3D model/animation inside webui, or edit pose as well, then send screenshot to txt2img or img2img as your ControlNet’s reference image.
Currently, it supports to load several types with two modes. Single File mode By the default, you can upload single file directly, support formats:
- glb – partly support, it fails on the model needs draco_decoder (like IridescentDishWithOlives.glb) or KTX2 textures (like coffeemat.glb), need to figure out how to fix later
- gltf – partly support, it fails on the model has other resources (like DamagedHelmet.gltf), will fix later in Multi Files mode
- vrm – more support see Multi File Mode
But it has potential to support more, since threejs does. So if you want other formats such as cmt, just feel free to open an issue, I will add it later. (I upload two models, male02.obj and Sanba Dancing.fbx, as your test, you can find them under /models folder)
Since I just finish refactor this extension by React recently, I will add pose edit support back later
Just like you install other extension of webui:
- go to Extensions -> Install from URL
- paste this repo link
- go to Installed, apply and restart UI
Do not use configurations under Settings -> 3D Model, it would not work. I will remove them later.
Settings from page
There are 4 panels and several buttons on the page:
- Model – allow you to load your local 3d model file.
- Scene – control the canvas properties, list the objects in the scene
- click Refresh Scene Tree if there is nothing
- click Scene – to show up background color picker
- visible checkbox is available on Directional Light, Ground, Grid, Axis and mainObject, you use it to show objects up or not
- Operate radio button is available on Directional Light, Hemisphere Light, and mainObject, you use it to translate or rotate object from scene (in fact, I will use this to rebuild pose editor later)
- click Preview Camera – show up Near, Far and FOV for camera
- Animation – control the FBX animation playing with progress bar here.
- Send To ControlNet – Send the current preview view of the 3D model on ControlNet in txt2img or img2img
On the main tab of 3D Model Loader, you can use:
- Mouse left button -> rotate the camera
- Mouse wheel -> zoom in or out
- Mouse right button -> move the camera
I know gradio has its own 3D model component called Gradio.Model3D, but it only supports three formats: obj, glb and gltf, I think it is not enough, at least it should support FBX animation, so I build my own extension.
Created by jtydhr88 and special thanks to other contributors:
My other extension for Stable diffusion webui
- Canvas Editor A custom extension for AUTOMATIC1111/stable-diffusion-webui that integrated a full capability canvas editor which you can use layer, text, image, elements and so on, then send to ControlNet, basing on Polotno.