Designing Characters with Deep Learning: Spellbrush (W18) - YC Gaming Tech Talks 2020

My name is Corey; I'm the CEO at Spell Rush, and I'm here to talk to you today about designing characters with deep learning.

So, um, we're Spell Rush. We're a YC company as well; we're building deep learning tools for art and artists. What exactly does this mean? Art is hard, so my co-founder is a professional artist. But we're a fairly small team, so a lot of the question is often how do we scale up her ability to create more content without becoming a massive studio? Drawing things takes time, illustrating things takes time, and budgets on a lot of AAA titles and kind of the studio pipeline format—budgets are often 50, 60, 70 percent of the total production budget.

So, obviously, like art is becoming increasingly more expensive and increasingly difficult to scale. So, um, what if we could use AI in the asset pipeline? This was kind of the question we asked ourselves originally when we started our company, and we've been kind of building tools to sort of help in this direction.

So, a quick quiz for the chat: we're building AI tools. Which one of the following images was not actually drawn by a human? So, I'll give 10 seconds or so. So, these are three illustrations in the anime style of character portraits, and, um, it turns out the right one is actually drawn entirely by our AI. The left two are from popular Twitter artists.

And the key thing here is we can actually create images that are on par with what an illustrator would be able to draw. So, we can, um, basically in the amount of time—the left two images would have probably taken a professional illustrator anywhere from two to fifteen hours in order to draw—and our tool can actually draw a character in sub two seconds. And that's not all; we can not only draw a character, but we can also draw hundreds of characters in the same amount of time that it would take to generate one character. So, the possibilities are kind of endless in that respect.

So here, here's an example: we actually have one of our older models online. That one of this model is called on wifilabs.com, and what you see here is you can actually interact with one of our earlier models. You have the ability to pick any character and then customize this character using various steps through the AI through the online flow.

So yes, yes, I am reading the chat. We were the ones behind…

So, um, how does this work? So I'll give a brief introduction to how this technology works. The ability—the way we are able to kind of create characters from scratch—the general idea is we're using a technology called GANs, or Generative Adversarial Networks. The way this works is we have a network, a neural network called a generator, and we have a second neural network called a discriminator.

The generator's job is to learn how to draw art, and the discriminator's job is to learn how to tell good art from fake art. So, we can take these two agents, and we can actually construct a network in the following way: we take the generator and we take a corpus of real art that we want the generator to learn how to draw like. In this example, we have a bunch of classical paintings, so we actually want the generator to learn how to draw classical paintings.

We can take these two images and feed them at random to the discriminator. So, the discriminator's job is now to decide whether an image came from the real corpus or from the generator's learned drawings. What happens is this is then fed; we can then evaluate whether the discriminator was correct in his assessment, and then we back propagate and update the weights for both the generator and the discriminator so that both of them now can learn from whether they made a mistake or not.

This then gets propagated, and this is how the cycle learns. We run this millions upon millions of times in order to train both the generator and the discriminator. One small note, but this isn't quite important, is that both of the generator and the discriminator are computer programmed, so they are idempotent; they will always give back the same result every time. So, we actu...