Agentic Animations

I had always wanted to make games, and the reality was that I couldn't. Over the years I got good at the parts programmers get good at. Engines, systems, physics, the machinery underneath. But so much of a game is the art and the assets that skin the things I could build, and I didn't know how to make any of it. Every programmer who has started a game knows this wall. You can make it run. You can't make it look like anything.

In the summer of 2026 I had some time, and I'd noticed that agentic workflows had gotten really good at things that weren't just text. Graphical things, more complex things. I knew they could do 2D image generation. So I went in with a modest question. What kind of simple animations could I get out of them, maybe something with a little squash and stretch, and how far could I push it?

The 2D work went well fast. Image models could draw a character, draw it again in a new pose without losing who it was, and draw it from the side and the back. I set up a funnel that generated a pile of enemy concepts and kept the good ones, and about seven in eight were keepers. Then I tried something I didn't expect to work. Animating a flat drawing frame by frame is brutal, so game artists rig sprites instead: they put a simple skeleton under the drawing so a bone can swing a wing or bend a tentacle without anyone redrawing it. I asked Opus to do that, and it could. It looked at a bat, decided where the bones went, and bound the pixels to them. The bugs it hit were real rigging bugs, like a wing flap that dragged the ear along because the ear was close to the wing on the page, and it fixed them with real rigging answers, measuring closeness along the body instead of straight across the image. A jellyfish got one bone chain per tentacle straight on the flat drawing and swam.

Then I hit the ceiling. To make really satisfying 2D animation, the kind that isn't just a strip of frames, you take a finished character drawing and pull it apart into its limbs, and then you have to paint in the parts that were hidden behind other parts, the shoulder under the arm and the torso under the cloak, so that when a limb swings away it doesn't leave a hole. This is where the models really struggled. They weren't good at making the cuts where they should be, and they were worse at painting in what had been hidden, which is a particular kind of hallucination they don't seem to be good at yet. I tried pixel art next, and the pixel art itself came out well, but rigging it produced strange anomalies. The results looked pretty good, and they needed a lot of hand tweaking, and the agents couldn't clean them up without me. Faster than doing it by hand, but still a manual process.

That got me wondering. Rigging in 3D isn't much harder than rigging in 2D. You look at the bone structure from a given view, decide whether it looks right, and move what doesn't. So I decided to see whether I could get a 3D model and have an agent work on that instead.

I remembered hearing about TRELLIS, Microsoft's open-source image-to-3D project, and sent a research agent out to see what the state of that field was. The field calls it a lift: hand a model one drawing and it hands back a full 3D mesh. I'm glad I did the research, because TRELLIS turned out to be quite bad, and the two other models the research surfaced were quite good. The first mesh that came back from one of them blew me away. It looked better than I could have imagined. Then I started playing. Could Opus look at the mesh and move the bones around? Yes. Could it build an animation? Yes. Could it put a Mixamo-style rig on it and retarget existing clips? Yes. Everything I asked it to do, it did. I had gone in thinking the answer was 2D, and it turns out that with modern models, 3D is easier than 2D for a lot of use cases, in some real ways that matter.

The numbers are what make that real. For about forty cents you get an amazing textured model. A rig costs about fifty cents in Claude processing, and even relatively low-powered models are decent at it. I built tooling that lets me rig a model myself in about a minute. Then you have Claude create the animations, or if you have existing animations, even better, you're off to the races. I've made dogs and elephants walk. I've rigged lots of humanoids. Everything seems to just work. Granted, none of it is anywhere near what a professional animator could do with a lot of time, but it is absolutely passable for 99 percent of games.

I went into this with a highly experimental framework, because I'd learned on past projects that when you don't know what the right solution is going to be, you don't guess. You test. Have the agent make twenty different 2D animations and see which squash and stretch looks alive. Have it draw the same character in a dozen art styles and see which one survives the trip to 3D. Have it fit a piece of equipment onto a model a hundred different ways and see which ones actually sit right. The project is organized as dozens of those experiments, each one asking a single question and getting a single verdict, and each one gets its own page on a site I built alongside the pipeline so I can watch the results land as they're generated. The pages double as the memory. Without a record of what's been tried and what the verdict was, the agent drifts back into work that's already been done, and I pay for it twice.

One of the biggest problems I ran into was that the agents wanted to use their own judgment about whether the results were good. They are still really bad at that. An agent can't reliably tell whether a walk cycle looks right, or which of two methods produces the more convincing animation. Left alone, it would run six experiments, reach a conclusion that was wildly off base, and build its next six on top of it, a self-reinforcing set of falsities. So I wrote an entire rule set into the project to keep it from hallucinating success, and the heart of it is four words: you generate, I judge. The agent generates. It puts the results on the page, says what it ran and what it cost, and stops. I look, I decide, and it writes down what I said. My eye is the instrument. Its eye isn't calibrated, and it has no way to become so. Yet.

Looking at hundreds of results that way taught me a lot about what these models can and can't do, and the lessons are consistent. The geometry they produce is fantastic. What they're still bad at is particular kinds of detail, and faces most of all. Anything where you care about the quality of the face, I have yet to find a tool that does well. The texture that comes back on a face is always low-detail and slightly wrong, and generating at a higher resolution doesn't fix it, so I've learned to pick art styles that don't put the character's identity in the face. One of the best models I made the whole time was a goblin that happened to be the very first one I did. By pure luck it had a forgiving level of detail, in a sort of cartoony style that translated really well through these models.

Hands are the other weak spot. The models tend to hallucinate three fingers unless the hand faces the camera, so I generate my source drawings with the palms forward, where the detail is visible, and then repose the finished model into the T-pose that animation tools expect.

One thing you learn pretty quickly working with these models is that they put most of their effort into what they can see in the source drawing, which means they underdevelop the textures facing away from the camera. The back of a model just looks kind of generic. The workaround a lot of the models offer now is that you can feed them multiple views, and that works pretty well. What's interesting about the way I'm using them is that I'm going from AI-generated 2D images to 3D, and when a model creates a 2D image it hallucinates the lighting, so the light is always inconsistent from one view to the next. Feed it several views and those inconsistent lighting artifacts show up on the model. There are diminishing returns. Too many views gets you more detail, but also lots of little lighting artifacts that just look weird. So it tends to be better to give the model one or two angles, enough to get the detail it needs but not so many that it starts inventing lighting.

And the biggest lesson is that everything depends on the drawing you start from. The lift adds a little inconsistency of its own, so if your 2D characters weren't consistent to begin with, the 3D versions come out looking like they're from different games, and you spend the money you saved on cleanup.

The other thing the experiments taught me is that an agent working in 3D needs tools, and the way to invent them is to keep asking one question. How would a human do this, and what would they use? It isn't that an agent can only understand a 3D model through flat pictures of it. It can reason over the 3D data directly and write scripts to analyze it, things no human could do by eye. But the most useful tool turned out to be the human one, because these models were trained on so much 2D data. A person can look at bones drawn over a picture of a model and say the knee is out of place. Give the agent the same rendered views with the skeleton laid over them, and it can do the same thing. Once I asked it to work that way, it started inventing its own tools. It put a reference grid in the background. It started labeling things, then noticed its labels were long enough to hide parts of the image, so it encoded them as smaller marks that kept the information while covering less of the picture. That's how a slow, cheap model can place a full skeleton on its own.

It's been wild how well it rigs. I had it rig an elephant purely by prompting, no hand rigging at all, and then build a walk cycle for it. The first cycle looked okay, but it swung its legs in a way that looked weird, so I had it go research what's known about walk cycles and the kinematics that drive a leg through different gaits, and it kept coming back with better results until it landed on a walk that looks pretty darn good. Better than 90 percent of what I see on the Unity Asset Store. I also built tooling for myself, and the reason is simple. The agent can do a lot of this, but when I want to make a single change, it's faster to just make the edit. Every time I'd tried to learn editing through the big, complicated 3D tools, I'd bounced off, so I built a small modeling tool that works the way my brain thinks about it. Look at one view and move points to where they need to be, look at another view and move points again, and that's it. With that simple process I can rig a model in about a minute. The current line of work is fitting equipment onto characters, a sword into a hand or a helmet onto a head, and there's no shortage of problems behind it. This project keeps evolving, and some of these threads will probably become projects of their own.

I haven't put a character in a game yet. Months in, I'm still experimenting, with no end of things left to try. But this project matters to me for reasons that go back a lot further than this summer. I first tried to learn to program when I was ten, because I wanted to make a game. What I could make were little text games, and they weren't the games that excited me. Ira Glass has the best description of that feeling I know: "All of us who do creative work, we get into it because we have good taste. But there is this gap. For the first couple years you make stuff, it's just not that good. It's trying to be good, it has potential, but it's not. But your taste, the thing that got you into the game, is still killer. And your taste is why your work disappoints you. A lot of people never get past this phase. They quit... It is only by going through a volume of work that you will close that gap, and your work will be as good as your ambitions." I had the taste for twenty-five years and never the volume, because in a game the volume is art, and my hands don't make art.

The volume doesn't have to come from my hands anymore. It can come from the agent, while the taste stays mine. That's all "you generate, I judge" ever meant. For the first time since I was ten, what I can build is bounded by my imagination instead of by what I can draw. The engine is waiting.