A cursory assessment of consistent character image generation with AI

I'm currently at a bit of a crossroads with respect to where to go next with my New Media Project class. I could continue with the "AI-powered Scribblenauts" idea I proposed in the previous post, but I've also been wanting to look into ways that AI can assist with other game ideas I've been wanting to develop. Specifically, I'm early in the process of developing a prototype fighting game that implements some novel mechanics, and even though I'm not really too concerned with graphical fidelity at this stage, there's still a fairly heavy burden in terms of asset production required to get the game to a playable state.

Fighting game development in particular tends to be bottleneck pretty heavily at the animation stage. I believe there are two primary reasons for this. One, fighting games require a lot of animations per character, and all of those animations have to conform to the same model. And two, the expected standard of quality for those animations is very high. Fighting games are designed around replayability and long-term investment, so players will be seeing the same animations over and over again, and the animations also bear a lot of the burden of expressing each character's personality. 2D animation is especially difficult to realize for fighting games: I've personally seen a lot of projects struggle to get off the ground because the developers underestimated the difficulty of producing an entire animation set in 2D.

However, I believe that AI image generation can greatly help with accelerating the animation process. AI image generators are at the point where they can create images of the same character in multiple different poses while staying decently on-model.

Image generated by Ammar Alghamdi in Midjourney

In my opinion, the AI still isn't quite good enough at staying on-model to be perfectly convincing, and there's still a lot of uncanny-valley stuff going on with facial expressions and eyes. All of the AI-generated animations I've seen seem to struggle with the "animation" part: wobbly eyes and backgrounds are all over the place, as well as unnatural character movements. AI seems to have difficulty with maintaining continuity of fine details between frames, which means going beyond static image generation is hard to realize with the current tech.

However, I think this would be workable for an application like "Infinite Scribble," where there's more of an emphasis on individual poses, and a lot of the appeal would come from the novelty of being an AI-based game. I also think that this approach could at least assist in the process of generating 2D sprites for fighting games. Fighting games in particular tend to favor animation styles with a lower frame count, sudden jumps between strong key poses, and lots of model breaking in order to sell animations even when there's not a ton of frames to work with.

These two frames are actually consecutive with each other

In the context of a fighting game, off-model moments would be harder to notice and easier to pass off as intentional, and with a lower frame count, it would be a lot more reasonable to do manual sprite adjustments where necessary. 

Right now, I'm looking at two potential paths I could go down:

1. Continue implementation of my "Infinite Scribble" concept
2. Develop a workflow for 2D sprite animation using AI

Regardless of which I choose, both of them will heavily involve the use of AI image generation to create posed characters. In that vein, I've been doing a lot of exploration of a program called ComfyUI, a free, open-source tool that allows for the creation of customizable, node-based workflows.

In future posts, I'll discuss some of the ideas I've been trying out and what they mean for the viability of my prospective projects.