Text vs Visual Learning Tools. What does this research paper uncover?
Note: This is part 2 of a series of posts reviewing some of the origins of scientific claims that compare text vs visuals. Access part 1 here.
If you would like to follow along, you can find the original article that I’m reviewing from University of Minnesota website.
The paper is titled “Persuasion and the Role of Visual Presentation Support: The UM/3M Study” prepared by Douglas R. Vogel, Gary W. Dickson, and John A. Lehman.
Last time we talked about various quantitative claims that many use to validate power of visual communication. While I don’t disagree about the power and potential of using visuals to convey a message, some of the cited references had some hazy origins. Since my work depends on scientific and medical accuracy, I constantly read journal articles and attend scientific talks. I am very familiar with the general setup of a research paper.
Now that we went over the abstract, a curious introduction that started with a conclusive statement, and materials and methods, let’s dive in to the results that claim to have discovered that using visuals will make a presentation 43% more persuasive. Make sure to check out the previous article so you know how the experiment was set up.
It’s time for Results!
Little note here about results. Usually, the result section is where the readers see all the relevant data in a clear concise manner that is used to support the question/hypothesis stated in the introduction.
The paper states that the students who had some style of visual aid “planned to spend 16.4% more time and 26.4% more money” compared to the students who didn’t have any visual aids. Great! …what?
Fig. 2. Vogel et al, 1986.
I REALLY HOPE they didn’t get 43% from adding 16.4+26.4=42.8……
Usually data for a group test will show how many participants in each group (n=35 for example), how they answered the questions, and then what percent said what. A table showing how the 9 groups answers changed before and after the presentation would have been great. Unfortunately, there’s nothing like that. We first see this graph.
The next section is pretty funny: The authors claim that the students’ willingness to spend time is a better indicator than money because…they’re full-time undergrad students and they have roughly the same amount of time available in the schedule, but they might have varying financial resources. Then why did they ask about money?
Don’t some full-time students work and/or participate in various extra-curricular activities? Don’t some students live nearby, and some students drive a good distance to get to school? Some students are single, some have significant others, some might even have families.
I like how they immediately begin discussing the results in the second paragraph of the results section. Usually the discussion of the results is found in the discussion section.
Another “quick” comment is that the authors add is that it’s easy for the students to lose interest during the presentation because that’s “obviously” (they use that word in the paper, no joke) how the students who didn’t get any visuals responded when asked about time commitment.
If you read the paper and are following along, the initial graph (figure 2) shows that compared to pre-presentation questionnaire (control), the students who didn’t get the visuals said they planned to spend 23.8% LESS on the course where the students who did get the visuals said they planned to spend 16.4% MORE.
I hope to find out more about what these percentages actually mean. Are we comparing results from 35 students against 280 (35×8) students? Is this all right? How did the 35 non-visually treated students respond to the pre-presentation questionnaire?
Anyway, where is 43% from? -23.8+16.4=40.2.
First of all, in proper science studies, one can’t just add some numbers together and claim that as a result. Measuring the negative results by control and positive results by the variable in manner is not how numbers are calculated. I can’t even show you how it should be done, because we have no access to raw data…which should have been included somewhere on this study.
Authors describe, “Note that the (-23.8) to 16.4 does not quite add up to 43%. This is due to the fact that the initial positions of the groups differed slightly (not statistically significant)”. It’s like they’re saying, “Don’t worry! We’ll get to that magical number from the introduction eventually!” But this is all kinds of wrong. This is not how percentages work. At least I was wrong in the above assumption that they just added the positive results.
This initial result itself is enough to determine that this “study” is unsubstantiated. For some reason they can’t even hit 43%. But, I wanted to be fair. I wanted to see if the 43% was explained better later in the study.
fig. 3, Vogel et al, 1986.
Next comes figure 3. I don’t understand this at all. They somehow measured positive changes in perception, attention, comprehension, agreement, retention, and action. Besides action (either you do it, or you don’t), comprehension, and retention (they did mention post-presentation asked about the presentation itself), how do you measure perception, agreement, and attention? Apparently these results are also statistically significant and the graph also compares the students without visuals and students with visuals. How do we discern this? I might have forgotten how to comprehend graphs. Please let me know in the comments.
Update 5/7: My mind was trying to wrap around this graph so hard that I didn’t notice that Action change in percentage is 43%. It’s interesting, because the paper doesn’t mention that at all in the text. Usually it’s not ok to just put up a graph and not talk about THE MAIN FINDINGS in the text. Even I don’t just throw some artwork up without text, and I’m an illustrator.
By the way, there’s no figure caption at all in the graphs. The titles are really plain and non-descriptive. The axis don’t really explain anything.
fig 4. Vogel et al. 1986
Then the figure 4. All of a sudden, we jump to how the students perceived the speaker. I thought the speaker was “average” so they can concentrate on the effects of visuals. They compared the non-visual support students against the visual-support students. The authors were nice enough to describe what p-values mean. I hang out with people whose sole mission in life is to get p<0.05. Actually, the authors go above and beyond and provide the readers with little arrows that correspond to different levels of p-values. More arrows you see next to the word, more likely that the students will have particular impressions about the speaker…sort of. This is wrong on so many levels.
This figure suggests that now, we are also evaluating the speaker him/herself. Where was that question in the intro? Do visuals make presenters look sexy? …well not according to this figure. But according to this data, it does help the speaker be more clear and concise….if you still trust these numbers.
I’m trying my best to figure this out. If you have a better idea of what’s going on in the paper than me, please let me know in the comments. Next up, Results, part 2! They just keep going!
To be continued to Part 3.
Part 1: Abstract and Introduction
Part 3: Results, part 2
Part 4: Discussion
Part 5: New experiment begins and wrap-up