Become A Stable Diffusion Prompt Master By Using DAAM - Attention Heatmap For Each Used Token - Word
Our Discord : https://discord.gg/HbqgGaZVmr. In this video, I am explaining how to use the DAAM extension of Automatic1111 to understand how input words influence the parts of a generated image. If I have been of assistance to you and you would like to show your support for my work, please consider becoming a patron on 🥰 https://www.patreon.com/SECourses
Published Feb 05, 2023 · 17,057 views · Watch on YouTube
Chapters
- 0:00 Introduction to becoming a Stable Diffusion prompt master
- 0:25 What the DAAM: Interpreting Stable Diffusion Using Cross Attention Paper
- 0:55 How to install DAAM extension on Automatic1111 Web UI
- 1:10 How and from where to use DAAM extension on Automatic1111
- 1:50 How to get prompt heatmap of the generated image
- 3:46 How to give different attention / emphasis / strength to different words used in a prompt
- 4:22 Heatmap blend alpha
- 5:43 Outro
About this video
Our Discord : https://discord.gg/HbqgGaZVmr. In this video, I am explaining how to use the DAAM extension of Automatic1111 to understand how input words influence the parts of a generated image. If I have been of assistance to you and you would like to show your support for my work, please consider becoming a patron on 🥰 https://www.patreon.com/SECourses
Playlist of #StableDiffusion Tutorials, Automatic1111 and Google Colab Guides, DreamBooth, Textual Inversion / Embedding, LoRA, AI Upscaling, Pix2Pix, Img2Img:
https://www.youtube.com/playlist?list=PL_pbwdIyffsmclLl0O144nQRnezKlNdx3
What the #DAAM: Interpreting Stable Diffusion Using Cross #Attention Paper :
https://arxiv.org/pdf/2210.04885.pdf
DAAM Script Repo : https://github.com/castorini/daam
DAAM Extension of Automatic1111 : https://github.com/toriato/stable-diffusion-webui-daam
Notepad++ : https://notepad-plus-plus.org/downloads/
0:00 Introduction to becoming a Stable Diffusion prompt master
0:25 What the DAAM: Interpreting Stable Diffusion Using Cross Attention Paper
0:55 How to install DAAM extension on Automatic1111 Web UI
1:10 How and from where to use DAAM extension on Automatic1111
1:50 How to get prompt heatmap of the generated image
3:46 How to give different attention / emphasis / strength to different words used in a prompt
4:22 Heatmap blend alpha
5:43 Outro
Some info from paper
Abstract
Large-scale diffusion neural networks represent a substantial milestone in text-to-image generation, but they remain poorly understood, lacking interpretability analyses. In this paper, we perform a text–image attribution analysis on Stable Diffusion, a recently open- sourced model. To produce pixel-level attribution maps, we upscale and aggregate cross- attention word–pixel scores in the denoising subnetwork, naming our method DAAM. We evaluate its correctness by testing its semantic segmentation ability on nouns, as well as its generalized attribution quality on all parts of speech, rated by humans. We then apply DAAM to study the role of syntax in the pixel space, characterizing head–dependent heat map interaction patterns for ten common dependency relations. Finally, we study several semantic phenomena using DAAM, with a focus on feature entanglement, where we find that co-hyponyms worsen generation quality and descriptive adjectives attend too broadly. To our knowledge, we are the first to interpret large diffusion models from a visuo linguistic perspective, which enables future lines of research.
6 Related Work and Future Directions
The primary area of this work is in understanding neural networks from the perspective of computational linguistics, with the goal of better in- forming future research. A large body of relevant papers exists, where researchers apply textual perturbation (Wallace et al.,2019), attention visualization (Vig,2019;Kovaleva et al.,2019;Shimaoka et al.,2016), and information bottlenecks (Jiang et al.,2020) to relate important input tokens to the outputs of large language models. Others explicitly test for linguistic constructs within models, such as Hendricks and Nematzadeh’s (2021) probing of vision transformers for verb understanding and Ilinykh and Dobnik’s (2022) examination of visual grounding in image-to-text transformers. Our distinction is that we carry out an attributive anal- ysis in the space of generative diffusion models, as the pixel output relates to syntax and semantics. As a future extension, we plan to assess the unsupervised parsing ability of Stable Diffusion with syntactic–geometric probes, similar to Hewitt and Manning’s (2019) work in BERT. The intersection of text-to-image generation and natural language processing is certainly substantial. In the context of enhancing diffusion models using prompt engineering,Hertz et al.(2022) cement cross-attention maps for the purpose of precision-editing generated images using text, and Woolf(2022) proposes negative prompts for removing undesirable, scene-wide attributes. Related as well are works for generative adversarial networks, whereKarras et al.(2019) andMaterzy n´ska et al. (2022) disentangle various features such as style and spelling. Along this vein, our work exposes more entanglement in co-hyponyms and adjectives. A future line of work is to disentangle such concepts and improve generative quality.
7 Conclusions
In this paper, we study visuo-linguistic phenomena in diffusion models by interpreting word–pixel cross-attention maps. We prove the correctness of our attribution method, DAAM, through a quantitative semantic segmentation task and a qualitative generalized attribution study. We apply DAAM to assess how syntactic relations translate to visual interactions, finding that certain maps of heads in- appropriately subsume their dependents’. We use these findings to form hypotheses about feature entanglement, showing that co-hyponyms are jumbled and adjectives attend too broadly.