Koobdoo - Learning Equilibrium Mascot
The Learning Equilibrium
Subscribe
← Back to Blog

DALL·E, MidJourney, Stability: How Diffusion, GAN, and other ML Models Generate AI Images

By Sanjay Mukherjee•October 8, 2024

When looking for an image from a Generative AI platform, what’s essential: a good prompt, multiple prompts, or prompting skill? To answer this, let’s compare it to how I’d communicate an image requirement to a human designer.

For a general task, my process involves:

  • Defining the desired outcome (what I need and the goal).

  • Establishing evaluation criteria (quality standards, subjective vs. objective factors).

  • Communicating the brief (clearly outlining the task and necessary skills).

  • Setting the process for evaluation and iteration (discussing the approach, reviewing drafts, and deciding how to move forward).

The resulting summary might look like this:

  • Project Overview

  • Goal of Task

  • Output Required

  • End-Use

  • Context

  • References/Examples

  • Specifications

  • Process

  • Acceptance Criteria

  • Deadline

After sharing this summary, I’d have a discussion to ensure the team understands, sometimes reviewing sketches or early drafts. Based on this interaction and past work, I gauge risks (e.g., multiple iterations or misalignment) and decide whether to proceed or explore alternatives.

When evaluating AI platforms for image generation, what changes? Like with human designers, I must understand the AI’s capabilities, model type, and how to effectively communicate my needs. AI platforms are powered by machine learning models (like artists learning techniques). Models such as GAN, VAE, Diffusion, or Transformers all generate images differently, and each model is best suited for particular types of visuals. My prompts, therefore, should adapt to the platform’s underlying model to get the best results.

For example:

  • DALL·E 2 uses Diffusion and CLIP models—Diffusion generates the image, while CLIP matches text to images, allowing for more conversational-style iterations.

  • Leonardo, built on Stable Diffusion, offers various fine-tuned models but lacks text transformers, meaning prompts need to be clear and concise rather than conversational.

  • MidJourney, like Leonardo, is based on a Diffusion model and requires more specific, one-shot prompts.

Here are the images generated by DALL.E, MidJourney and Stability for the following prompt:

Please generate an image that depicts different ML models for AI such as GANs, VAEs, Autoregression, Transformers, CLIPS etc.

DALL.E output

Mid-Journey output

Stable Assistant output

Note the difference and think about how the underlying model interprets the prompt and why that led to its output. In summary, while all three platforms deliver high-quality images, their approach to prompting and iteration differs. On DALL·E 2, you can engage in iterative dialogue, whereas Leonardo and MidJourney call for more precise briefs. Understanding these distinctions ensures a smoother workflow and better output.

(Psst: I loved the MidJourney version and went with that from Feature Image on this post due to its 'x' factor - it was symbolic but clear enough, it was cool but most importantly the images within the image represented outputs from the different models. DALL.E version was a good illustration to go within the article. The Stable Assistant version was accurate but within a limited context, I would probably go for that if the article was for a technical audience.)