“Rigging is the Art of Articulation. Not the Art of Naming Attributes or Setting Driven Keys” –Somebody, somewhere
Welcome to the ML Skinning Framework — where skinning gets a fresh twist!
What started as a small research project into the world of ML turned into something quite incredible.
Can ML skin a Pinky Finger?
This project explores using machine learning to automate the skin weighting process, for… the pinky finger. By training a model on existing skinned assets, we can teach a model to predict joint weights based on mesh and spatial data. The goal is to build an MLP (multilayer perceptron) model that can predict the proper skin weights for a new pinky finger mesh.First Model
Collecting Basic Data
I started by batching about 15 assets, gathering data about each vertex influenced by a list of joints. I gathered world position and skin influence. Unfortunately I did not capture a video fo my first test. But it was successful enough to show me the potential of this technology and move forward with a bigger dataset.Second Model
Improving Predictions by Adding Features
The initial result were not perfect but they showed there was potential. The initial model was limited to only vertex position as input. I began exporting additional features per vertex. Vertex normal, blend weights, joint matrices, etc. The model’s input vector expanded from 3 to 12 dimensions which allowed the model to better understand spatial relationships in the context of deformation. And here are the results:Third Model
Scaling the Dataset
With the new features added the model began learning complex relationships between geometry and deformation. Even with the same 15 character dataset the improvement was dramatic. Predictions became cleaner, smoother and closer to human painted weights. Now to see the results of a larger dataset: I think we can now answer the question. An ML can skin a pinky finger.Comparing Results
Ml vs Human
With the models now ouputting good skin weights, are the results better than human made? Additionally, are the results of 131 datasets more accurate than roughly 15?Can ML skin a Hand?
Is an MLP Model suffecient for a more complex mesh?
After training a model on the vertices of a hand, on about 50 assets. Here are the results. This shows a lot of promise! But I think we are running into the limitation of a simple MLP model. I will next explore using PointNet to give the model a better understanding of the geometry it is working on.
