Paint Along With Frida #4: Models and such | Artificial Intelligence | Articles | Ant Smith

I concluded my 'Conversations with Sparky' generative text AI exploration by getting it to write a 3000-word short story, since the proof of the pudding is in the eating... It's pretty easy to get these AIs to make trivial things that seem astounding, but it's a different matter to prove they can be genuinely useful. So I wanted to put Frida through her paces by using her to draw not just random pretty pictures, but a whole graphic novel.

At first my intention was to create an anthropomorphic ferret character. I chose to use the 'Absolute Reality' model which has been trained on top of Stable Diffusion v1.5 - so it benefits from a wide range of known concepts with some really good photorealistic detailing.

But it has no idea what an anthropomorphic ferret is - managing only to draw a polecat taking a walk with, I think, a young lady; a later attempt drew a young man with a ferret on his head...

I had a search about on civitai.com for models that had anthropomorphic training and found the YiffyMix, created to make furry-porn images; but it did make me a not so bad ferret/human hybrid; using exactly the same prompt.

The YiffyMix model, though, lacked the 'realistic' detailing so I decided to try a merge of the two models and managed to create Street Bob!, as shown above. Which was fun really, but I do recognise he's a tad creepy; in fact downright disturbing. I didn't use him at the end of the day, but the idea of mixing models was great. It meant I could create my own personal blend of commonly available models and thereby further personalise my work; using tools that most likely others are not using - a great boon when looking to create images I can call my own in a world where nobody really owns anything.

To fully appreciate how merged models work I created 19 of them with the mix ranging from 5% Absolute Reality (AR), 95% YiffyMix (YM) through to 95%AR/5%YM; allowing me to create a range of 21 images starting from 100%AR cross blending with YM in 5% steps:

The precise blend that works best will undoubtedly change from prompt to prompt, so I have kept all of the merged models for future use; even though they're about 2GB a piece. I currently have over 100GB of models on my disk, so you'll want plenty of disk capacity while exploring models.

The choice of model has the greatest impact on what gets drawn. All other settings discussed so far (apart from prompt and seed) are concerned with fine tuning an image. You should choose your model first (or create the blend you want) since if you change it, you pretty much have to start from scratch finding a seed and building the prompt. Here's an example of how radically the scene changes depending on model, all of these images used exactly the same settings, seed and prompt:

There are thousands of models available that have been based on SD1.5 because training 512px images isn't too onerous in terms of hardware specs and processing time. Lots of these trainings overcome some of the biases in AI image generation - LifeLikeDiffusionEthnicities, for example, improves the ethnic diversity of the AI's imaginings.

SD XL, using 1024px images and also being very new, has very few layered variants. I think it will be a long time before it catches up with the huge range of community created models that SD1.5 enjoys. It does create much better results, less mangled hands and feet and fewer wonky eyes, but I think comes with quite a bland and idealised view of humanity based on western ideals.

Beyond base models

The model file contains all of the visualisations of all of the known concepts that you can work with through prompting - which is why they're so big (typically 2GB, or more). But it's highly likely that there wil lbe concepts you want to cook up which are just not known - e.g. maybe yourself.

There's a great video over on YouTube by Koiboi explaining these; and I'll look later, in another article, at how I taught Frida to draw our ferrets.

Dream Booth training updates the (2GB) model itself and that can be simply selected as the checkpoint to be used.

The other methods create extra files that can be used with a base model to extend its range of known concepts. These files are typically much smaller, perhaps 100MB for a textual inversion; or a little more for a LoRA. Lots of folk use these techniques to stick their own faces int scenes, and why the heck not, huh?

But at some point, one bright spark had the idea of training their AI on pictures it had made that included mangled hands... look for the badhandv4 textual inversion. I said previously (when discussing prompting) that you can't give the AI instructions on how to draw, only prompts of what to draw (or not to draw). But, if you train it with examples of it's own poor drawing then you can ask it not to draw like that!