DALL·E: An “Intelligent” Illustrator for your Language Classroom

By Shannon Donnally Quinn and Frederick Poole, Michigan State University

   

Introduction and Overview

You may have heard of ChatGPT, the chatbot created by OpenAI that is creating a lot of buzz about artificial intelligence (AI). The same company has also created DALL·E, an artificial intelligence interface that allows users to automatically generate images from a text description, called a “prompt.” You can use it to generate fun images to illustrate things from your class, make images that will inspire your students to create with language, or have your students illustrate narratives themselves.

Name of tool	DALL·E
URL	https://openai.com/dall-e-2/
Primary purpose of the tool	Automatically generate images from a text description
Cost	Generating images requires users to create an account. One “prompt,” which generates 4 images, costs 1 credit. You get 50 free credits during your first month, and 15 free credits per month after that. Credits can also be purchased for $15 for 115 credits.
Ease of use	Easy to use, but generating good prompts can take some practice. However, OpenAI does offer a guide for improving the quality of prompts.

The benefits of using visuals as a part of language learning are well-known, and DALL·E makes it much easier than before to create visuals that can supplement your language learning materials. Further, the tool connects pictures with language in a way that can be leveraged as a part of a language lesson. Finally, we as educators have a responsibility to help students grapple with the emergence of these new tools and their affordances and drawbacks. Many of us are the main point of contact between our students and the target culture, and learning to use these tools for their language learning in helpful (rather than detrimental) ways is part of a strategy for making them lifelong learners.

In this article we first give some examples of visuals that we generated using DALL·E and how to use it yourself. After that, we will give you some of our ideas about how DALL·E could be used in your language class.

How to Generate Images with DALL·E

On the main DALL·E page, it asks you to give the description of a picture that you would like to generate. It also gives you examples of pictures that have been generated by members of the community. You can hover over each picture to see the prompt that produced it.

Once you enter in your prompt, it takes a few moments for the AI tool to work, and then four images appear. These are different interpretations of your prompt. One of the really useful things about the tool is that you can ask it to create images on otherworldly or surreal topics if you desire. For example, we asked DALL·E to give us pictures of “a dog eating cake on the moon.” Whether the dog is on the moon or is looking at the moon from another planet may be debatable, but it certainly gave us something related to what we asked for.

When you select one of the images that you generated, you can do several things with it. You can share it with the community, download it, generate variations of it, or edit it. Generating variations will give you several pictures that are slightly different, like these dogs, which were generated from the first image in the group above.

One of the useful features of DALL·E is that it allows you to generate images in particular styles. When we generated the dogs above, we did not specify the style. Then we repeated the prompt, but specified that we wanted a photograph. Likewise, we could have specified that we wanted it to be in Pixar style, hand-drawn, pixel art, or steampunk, among many other options.

One of the most exciting things that you can do in DALL·E is that you can alter images with new prompts. For example, we thought the second dog in the previous photos was eating a strange-looking cake, so we edited the image, removing the cake.

Then we put in a new prompt, asking DALL·E to insert a birthday cake into the missing area. You can also upload images and add them into what DALL·E has generated. This allows us to create multiple images with controlled variation, which might be ideal for creating assessments, or spot the difference exercises. 

You can also upload your own images and alter them. I took images of myself and cropped out everything except my face and then asked the AI to put in a background.

While uploading personal images can be fun, it raises questions and concerns around privacy. Certainly, no one should be required to upload images of themselves, and minors in particular should not have their photographs used in these tools. As the tool warns users, students should be taught that uploading images without permission is strictly prohibited.

The prompts that we have shown above should get you started with using DALL·E. However, the best way to explore the capabilities of this tool is to simply try it out and experiment with it. There are also several guides online for improving the prompts that you provide to DALL·E. While it is fun to simply add random prompts to the tool, having a goal or intended outcome that you can work towards will result in understanding the tool better. In the next section, we look at how DALL·E handles other languages.

Does DALL·E speak Russian (Chinese, Spanish…)?

We have not been able to ascertain from the online information about the tool how it understands languages besides English. We experimented with some prompts in other languages and saw some interesting things.

When we asked it in Russian to generate “человек который танцует на столе” (“a person who is dancing on a table”), it gave us the following:

Only one of the images fits the prompt well, but the reason for the inaccuracy is not clear. The dog on the moon prompt also showed a similar discrepancy, so it is not clear whether the source of the problem is the language of the prompt.

Next, we tried the same prompt but in Spanish (Picture 10) and Chinese (Picture 11). While there may be some minor differences in how the original prompt was translated into these languages, the results were quite different. In fact, simply adding in the English prompt creates four more unique images. Even in the same language, trying the prompt a second time can create very different results.

When we asked DALL·E to give us images that were very culturally-specific to Russia, we got mixed results. When we asked it to generate a man standing on Red Square, probably the most well-known place in Russia, it did a relatively good job. A generalized prompt asking for two people on a street in Russia gave a plausible-looking result. But more specific and less well-known prompts had worse results. When we asked it to show tourists on Mamaev hill in Volgograd, it gave something only somewhat resembling the actual landmark. And when we asked it to show a person playing a Russian folk instrument, the results were inaccurate and a bit eerie. Asking it to show Baba Yaga (a Russian witch) resulted in some strange and seemingly random images. While it knew who the Smurfs were, it did not seem to know who some well-known Russian cartoon characters were. 

Thus, it bears keeping in mind that DALL·E is very likely to have inaccuracies, especially with culturally-specific content. This may change as the AI tool receives further information and training. There are, however, some ways that you can use this in activities with your students, as we will outline below.

One of the interesting things about using the target language for generating images is that it provides the learners with immediate feedback. While DALL·E and ChatGPT are admittedly powerful tools, they do take the prompts quite literally. When adding prompts in Chinese, we noticed that our prompts may have been unclear. As a result we changed it three or four times until we got our desired result. This is very similar to negotiation of meaning that occurs when communication breaks down between two speakers. Constantly re-thinking how we are describing a picture to the AI tool provides an opportunity to focus on how we are leveraging language skills to deliver the prompt. Because we rarely get the image we want first, it forces the learner to think of another way to express the same idea. This is similar to what happens when communication breaks down with real people. A reformulation of the original comment is made, and this reformulation is quite beneficial for the learner to not only develop communicative competence (e.g. developing circumlocution skills), but also to recognize potential gaps in their own linguistic ability. Having students keep track of the prompts they used as well as the outcomes of those prompts not only draws attention to their own language use (which facilitates learning) but it also helps students learn about how the tool works.    

Ideas for DALL·E in your Class

Here are some ideas for how you could use DALL·E in your class. If you have more ideas, we would love to hear about them in the comments.

• Illustrate your own stories or have students illustrate their own stories.
• Give students a series of images and have them narrate a story according to the pictures.
• Create prompts using target vocabulary and then have students try to guess what the prompt was based on the image generated.
  • Have students work in pairs creating prompts, and then have their partner guess what the prompt was that created the image.
• Give the students the four similar pictures that were generated by DALL·E and have them describe the differences between them.
• Give different students different photos that were generated by the same prompt and have students write descriptions of the photos, seeing if other students can figure out which description goes with which picture.
• Give two students similar but distinct pictures and have them compare them with each other in the target language without showing each other the images.
• Have students try to produce a particular image with an explanatory prompt and see whether DALL·E can produce a photograph of the intended word. (For example, if the word is “library,” a student’s prompt could be “a place where books are kept”.) Then other students have to try to guess the original word.
• Give the students the four similar pictures that were generated by DALL·E and have them vote on which picture is the best rendering of the prompt and explain why they think so.
• Create variations of the same image and have students put the images in order according to a story that you tell them about the images, or have them create their own story that goes along with the variations.
• Have students give prompts to DALL·E and then revise the prompts, analyzing why one prompt may have worked better than another or given a different result.
• Ask students to translate prompts suggested by other artists into their target language and compare the images that were generated in different languages.
• Ask DALL·E to generate culturally-specific images and then have students research and explain why the image does or does not fit the reality in the context of the target culture.
• Ask DALL·E to generate images based on idioms or proverbs as part of a lesson on them.
• Have students try to explain some culturally-specific situations in prompts to see whether DALL·E accurately renders them.
• Play a game similar to telephone – one student creates an image, the next student tries to guess the prompt that the first student used and creates the next image. After a number of turns, see how the final image turns out and compare the prompts used by each student.
• Ask students of literature to generate potential cover images for books or stories that they have read and then debate which cover captures the book the best.

DALL·E and Copyright

The copyright implications of AI tools can be very tricky to wade through, but in the case of DALL·E, OpenAI (the company behind it) has made it clear that as long as users are abiding by the content policy and terms of the tool, the images are owned by the user who generated them and can be reused, even for commercial purposes. 

Conclusion

DALL·E is an interesting new tool that presents language instructors and students with another way of generating and using visuals in language learning.