Episode 341: Psychology Ranked (With Paul Bloom)

Very Bad Wizards

In this episode, hosts Tamler Summers and David Pizarro welcome back psychologist Paul Bloom to perform a definitive tier ranking of classic

Key takeaways

• The Milgram shock studies are awarded S-tier status due to their profound impact on the ethics of psychological research.
• The Invisible Gorilla study is ranked in S-tier for its powerful demonstration of how attentional focus can render much of the world invisible.

Main topics

• Psychology experiment tier ranking
• Ethics in social psychology

Notable quotes

The best studies are one where you just show it to a person and it works on them.

Conclusion

The episode provides an engaging, critical retrospective of foundation

Transcript preview

Speaker 1 (0:00) Very Bad Wizards is a podcast with a philosopher, my dad, and psychologist Dave Pizarro, having an informal discussion about issues in science and ethics. Please note that the discussion contains bad words that I'm not allowed to say, and knowing my dad, some very inappropriate jokes. Who Speaker 2 (0:17) is writing this? You're doing this, this is your life! Oh my god almighty, I'm not part of this! Speaker 3 (1:10) good man. Just a very bad wizard. Speaker 5 (1:13) Welcome to Very Bad Wizards. I'm Tamler Summers from the University of Houston. Dave, all I'm hearing about these days is the hugging face incident and some guy who resigned from his job at Anthropic and now all of a sudden Ezra Klein is writing columns. Dario from Anthropic is writing manifestos. What happened and how is this affecting your P-Doom? Speaker 2 (1:39) I can't summarize the hugging face incident because I've been like, I honestly have ignored, I've tried to ignore what hugging face is. It's interesting. It's got like, do you have the feeling that it's gotten to religious fervor right now? Speaker 5 (1:54) Yes, but everybody's going to be very mad at us. I know, I Speaker 2 (1:57) know. But look, look, I'm not claiming I know. I'm just, I'm just saying. All I see on Twitter is people saying, like, you guys are all dumb for not believing this. You guys are all dumb for believing this. You guys have no arguments. No, you guys have no arguments. Speaker 5 (2:12) Yeah. Well, that's why we're devoting the whole episode today to go through all the details behind the hugging face incident, what it may represent, and how it may have been misinterpreted. And we have three leading experts. No, none of that. But we do have a great... show lined up today. We had Paul Bloom on to tier rank psychology thought experiments. Speaker 2 (2:39) No, no. Psychology experiments. Speaker 5 (2:41) Sorry. Speaker 2 (2:42) Somebody actually did them. Speaker 5 (2:45) In some cases, yeah. Speaker 5 (2:50) It was a fun but also very long conversation. So should we just leave it there and get to it? Speaker 2 (2:58) Yeah. I mean, unless you want to share your P-Doom, like, are you afraid to share yours? Speaker 5 (3:03) I'll show you my P-Doom if you show me yours. Speaker 5 (3:07) All right. Should we just get to the episode? Speaker 2 (3:10) Yeah, let's do it. Speaker 5 (3:11) All right. Let's get to our main segment. And we have our favorite guest returning, Paul Bloom. Speaker 4 (3:18) Welcome back. I am thrilled to be back. It always feels like it's been too long. I got to say, I've been following your Odyssey discussions. And it's Speaker 2 (3:28) amazing. Speaker 4 (3:28) It got me to read the book. Speaker 2 (3:29) I've been Speaker 4 (3:30) reading a bit of the book and rewarding myself with reading your podcast about it. So my fanhood, if anything, has sort of magnified over recent times. Speaker 2 (3:37) That's so kind of you. Speaker 4 (3:38) That's great to hear. Are you reading the Emily Wilson? Yeah, because I'm in favor of women. Yeah. Have you heard she's doing another one? Speaker 2 (3:45) Yeah. Speaker 4 (3:46) Yeah. I imagine her publishers must have mixed feelings about this. But she's doing another one. Did you translate the same thing twice? Well, you know why. So in Speaker 5 (3:54) the first one, she limited herself to the number of lines that were in the original. And then she didn't do that for her Iliad. And she liked the freedom of being able to add a few extra lines here and there. And so that's all she's going to do for the revised version, I think. Interesting. Is just release herself from that constraint. It's still going to be the iambic pentameter. It's still going to be, you know, I think fairly concise compared to some of the other ones, especially Fagles. But yeah. Speaker 2 (4:23) Do you think she's going to stick to her guns with the opening line? Speaker 5 (4:27) Complicated man? Yeah, yeah, yeah. At this point, changing Speaker 2 (4:30) that seems like she's going to, that would be Speaker 4 (4:32) giving in. That Speaker 2 (4:33) would be Speaker 4 (4:33) trolling or something. She has a book coming out about translation, which I think I'm going to get. I've listened to her a few times and she seems super smart and really engaging. Yeah. We've Speaker 2 (4:40) wanted to talk about translation, but haven't found an angle to. So maybe that will be. Yeah. Maybe we can get Emily Wilson. You got to get Speaker 4 (4:46) her on the pod. Speaker 5 (4:49) Do we know anybody who knows her? I think that given all the business we've sent her away. She owes it to us. Speaker 4 (4:56) Yeah. She'll give you the gratitude she gave to Nolan. Yeah. Speaker 2 (5:00) She's not Speaker 4 (5:00) big on gratitude, I think. Speaker 2 (5:02) One star iTunes review. She'll write two separate attacks on your podcast. Speaker 4 (5:07) One Speaker 2 (5:08) review saying what Speaker 4 (5:09) a shitty podcast it is. Speaker 5 (5:10) Repugnant on Apple podcast and sub-moronic on Spotify. Speaker 2 (5:17) All right. So we are going back to our tier list format. So Paul had. listened to our philosophy thought experiment tier list episode. And actually, I think, Paul, you pitched the doing a psych studies. I did. I mean, I'm going to start Speaker 4 (5:33) with the praise, but that was one of your best episodes. If I had somebody who was interested in philosophy or thinking about philosophy, maybe go into it, I would give them that episode on your thought experiments. It was such an engaging discussion. It was so much fun. But I was, as I was listening to it, texting David and bitching about your choices. Like, I think you did trolley problem. wrong. I think you gave a very low score at one of the most terrifically appealing dilemmas. Speaker 5 (5:58) It's the gift that keeps on giving, Charlie. Exactly. Speaker 4 (6:01) Not a day goes by where I don't see a Charlie problem. Speaker 2 (6:05) But Speaker 4 (6:05) a lot of your other choices were exactly right. You crapped on the utility monster, and as well you should. Speaker 2 (6:11) And Speaker 4 (6:12) then I sent David really obscure thought experiments and complained you didn't discuss it. Like, Speaker 5 (6:17) what did you want us to discuss? I think you weren't Speaker 4 (6:19) so obscure, but you didn't talk about H2O and XYZ. Oh, yeah. The Hillary Clinton twin Speaker 5 (6:24) earth. We didn't talk about Speaker 4 (6:25) Swamp Man. Speaker 5 (6:26) Oh, you didn't talk about Swamp Man? Speaker 2 (6:28) Yeah, no, we Speaker 4 (6:29) didn't talk about Swamp Speaker 5 (6:29) Man. I Speaker 2 (6:30) didn't know the difference between Swamp Man and the transporter problem. They're related. Yeah. Speaker 5 (6:35) It's Swamp Man as if the transporter problem and zombies had a kid. Speaker 2 (6:40) Yeah. Into a genealogy. Speaker 5 (6:43) But that's not what we're doing today. Today we Speaker 2 (6:45) are doing science. Yes. So, you know, there are answers to this. This isn't just a vibes-based tier list. Objectively. We should tell people what a tier list is. Yes. Speaker 5 (6:57) It's pretty simple. You will get an experiment, like the Milgram experiments, which I'm sure will be on here. And then we put it in the D, which is the lowest, the C, second, all the way up. to B, A, and S. S tier is like super special, right? Yes. It's Speaker 2 (7:15) the superlative. Speaker 5 (7:16) Are there Speaker 2 (7:17) rules about Speaker 4 (7:17) how many per tier? We don't have to rank them? No. Speaker 2 (7:20) I think the only rule is that we have to agree. Okay, good. Or compromise, maybe, is the better word than agree. For the Speaker 4 (7:27) thought experiment you did, give me an example. What was an S? What made the S? Do you remember? Speaker 2 (7:32) Ooh. Speaker 4 (7:33) God, no. Speaker 4 (7:36) Did you have anything that I remember? Any of them? The Speaker 5 (7:40) cave, Plato's cave, but I don't think you even had that, but I would have put that in the end. We called Speaker 2 (7:45) some audibles at the end that ended up. Descartes, Speaker 5 (7:50) evil demon, maybe. Yeah. The classics. Yeah, exactly. We used to have thought experiments in this country or not this country. Speaker 2 (8:02) Okay, Speaker 5 (8:02) so we have— I guess the Speaker 4 (8:03) question of what dimension do we rank doesn't really have an answer. Just all goodness. Oh, yeah, Speaker 5 (8:08) we should talk about this. Like, Speaker 4 (8:09) are Speaker 5 (8:09) we going to incorporate ethical concerns? Are we just talking about its contributions to science or to human understanding or understanding of the mind? What do we take into account here? Speaker 2 (8:21) Good question. I was thinking that— It would be, one, a holistic evaluation, which is Speaker 4 (8:28) a Speaker 2 (8:28) very important thing. The quality of the actual experiment, the impact it had on the field, the, like for me, there's some category of the cleverness of the finding where it's not necessarily counterintuitive, but you wouldn't have thought to do it that way. And replicability probably. Some Speaker 4 (8:46) are shockingly unethical. And that also makes them rather memorable. Yeah. That Speaker 2 (8:51) was a pun almost. Oh, Speaker 4 (8:54) I was actually thinking of the ones involving animals. Speaker 5 (8:58) Yeah, no, I can't stomach those. Yeah, Speaker 2 (9:01) we're going to have fun with those. Speaker 5 (9:04) So Speaker 2 (9:04) we have a big list to get to. So why don't we go ahead and start? What I'll do is I will make the image available with our final tier rankings for download in the show notes. We have 18 psych studies and I will be dragging and dropping into the tiers depending on what we decide. No order whatsoever are these in. So let's just dive in to the order that they appear on the screen with Milgram shock studies. Speaker 4 (9:28) I guess maybe the one-sentence version was Stanley Milgram, then an untendered professor at Yale, was very interested in atrocities of the Germans in World War II and as a way of exploring the idea that people will just obey whatever you tell them to, set up an experiment in a basement of a building in Yale where he brought together New Haven working men and basically persuaded them that they were part of an experiment involving shocking another person. And the shocks went up to a level where they would believe that they had killed the person. They did not. Actually, the person was a trained actor. It's not another guy. But the main punchline is that if you tell somebody, look, you're an experiment, this is what you have to do, people will apparently do kill somebody or believe that they have killed somebody. Speaker 2 (10:10) The 300 volt was the highest, I think, and it was labeled like, you know, sculling. Triple X or something. Yeah, like do not. And the person had stopped responding. So they're getting to a point where they're yelling. I have a heart condition. I have a heart condition. Get me out of here. And the experiment Speaker 5 (10:27) requires that you continue. You must go on. Yeah. Speaker 2 (10:31) Right. And Speaker 5 (10:31) something like 65%, this was the shocking finding. It's impossible to talk about this without saying things like that. But 65 % of the people went all the way. Speaker 2 (10:41) Yeah. So, you know, Tamler, you and I did an episode on this years and years ago. Speaker 5 (10:45) Yeah. Speaker 2 (10:46) And I remember in rereading these, the one thing I'll start off discussion by is like Milgram was really rigorous. So he did a gajillion variations of these where he manipulated, you know, whether the person was wearing a lab coat, how close the person was that the participant believed they were shocking, you know, what university they were from and reports all of those results. And it's had a huge effect on the ethics of how we do psychology because we can't do those anymore. Speaker 2 (11:13) To the extent that people have tried to replicate it, either under non-IRB, non-human subjects evaluated conditions like reality shows. But also in some cases, I think there was a study in Germany. It's been replicated. So it is a finding that at least I'm confident in. Speaker 5 (11:28) And it's like super fun to teach. It's tough. I used to teach the interview that I did with Zimbardo. on the Stanford prison, which we might see. And, you know, now that's a little more problematic to teach because that's come under so much fire. This one, less so, right? I mean, there's been questions raised and there were always the ethical concerns, but in terms of the experimental design and whether it replicates and all of that, it's kind of held up more than you would expect an experiment from this era to hold up. I Speaker 4 (12:00) think it has. There's been... criticisms that maybe some people didn't really believe they were doing it. Apparently there was more persuasion used and maybe Milgram talked about in the study, but I think you're right, Tamler. I think that this, this is rock solid. I will, I will just, I don't know what the procedure here is, but can I just give my ranking? Speaker 2 (12:16) Yeah. S. Yeah. I was, I'm tempted to put this one at an S and like, I, it feels like I'm burning an S early and there is no limit to it, but in terms of like all of the things. I Speaker 5 (12:26) don't know how many more S's there are going to be. This is the err. I don't know how to say the err experiment. Speaker 2 (12:31) Yeah, the err. Yeah, that's right. This is, I mean, first of all, for a social psych study from the 50s or 60s to replicate is already fucking bananas. Speaker 4 (12:38) Now you guys would do a version on Prolific where you ask people. Or what you do now is have chat. Do it and then say, well, you're going to react to it. That's right. Yeah, Speaker 2 (12:48) yeah. But pretend now you're 100 participants. Give me a distribution. Unfortunately, that's how it's done. All right. Speaker 5 (12:57) S, are we agreed? S for shock. Yeah. Strong beginning. Speaker 2 (13:01) There's one of the things that we can do at the end, should we choose, is do rearranging. Yeah. Speaker 4 (13:06) Okay. Speaker 2 (13:08) Next one is the very famous, it's actually Tversky and Kahneman, even though I have it as Kahneman-Tversky. Demonstration of the conjunction fallacy. This is the famous Linda problem where participants read a description of a woman named Linda. And the way that she describes makes her sound like she might have feminist tendencies. 31 Speaker 4 (13:29) single outspoken philosophy major concerned with discrimination and social justice. Speaker 2 (13:34) Right. And so they're asked whether they think she is a bank teller. Speaker 5 (13:40) No, she says, which is more probable? Linda is a bank teller or Linda is a bank teller and active in the feminist movement? Speaker 2 (13:48) Yes. Now, just numerically, like just being one of those things has to be more probable. But the finding is that people think they report that her being a bank teller and a feminist is the more probable outcome. Speaker 5 (14:02) Yes. As a matter of logic, the probability of. A and B has to be less than the probability of A. What is this supposed to show? Speaker 2 (14:11) It's the representativeness heuristic, like that people are using a heuristic that this just sounds like a description. This is a description that would be representative of a feminist. So they go with that feel rather than doing the probability. I Speaker 5 (14:26) mean, people aren't logicians. Like they don't think of things this way. Other Speaker 2 (14:30) people, you know. Speaker 5 (14:32) I don't get this one. Like, people are imagining that you're just kind of asking them a question and we're talking and she sounds a lot more like a feminist than a bank teller. Do you expect people to think they're in a logic Speaker 4 (14:44) class all of a sudden? But, Tammy, you don't get this because you're living in a world that was created by Kahneman and Versky. Like, back in the day, there was a common view that people thought according to the maxims of probability and logic. Who thought that? Well, all the people who hated this finding. Yeah. Maybe this adds a little bit to it. If you get people to put money on it, people will still put money on the wrong choice. And you can get them in a situation where a mistake would literally cost them. Speaker 5 (15:09) Here's my take, Purge. This is the Gettier problem of social psychology. Oh. Speaker 4 (15:15) You Speaker 2 (15:15) don't mean that. That is not true. Those Speaker 5 (15:17) are Speaker 2 (15:17) the fighting words. Yeah. Yeah. I mean, so in the context, especially, too, you know, of what the hope. we came to call home economicus, that people make decisions. Maybe it was a straw man to believe that people make these decisions based on some rational calculation. But the body of work that this finding and the findings from Tversky and Kahneman that followed... We're ones to bring down that view. And so if anything, I thought you might be a little bit more sympathetic to the takedown of what was obviously a mistaken view of human nature. Speaker 5 (15:52) Yeah. Like, I don't know if this was a view of human nature of anybody besides the economists. Economists. Yeah. Speaker 1 (16:01) But Speaker 5 (16:01) even for that, like, this isn't an economic problem. Speaker 2 (16:04) No, no, it doesn't have to be an economic problem. It's just the economic view that people are rational. The view from economics people are rational. It's not limited to them being rational about economics. Speaker 4 (16:12) Are you skeptical in general about a sort of Kahneman-Tversky research program that tries to argue that we don't, that we reason according to heuristics and biases and not according to maxims of logic? Speaker 5 (16:22) Not really. Like, I just think this was a very implausible thing to believe that people would do in the first place. Like, I get why people give the answer that they give. And maybe it's like a... Speaker 4 (16:33) It's not Speaker 5 (16:33) Gricean, or maybe it is. I don't know exactly what you can call Gricean. So some Speaker 4 (16:38) critics have said it is Gricean, and they think that when you say she's a bank teller, and that's why you say you think she's not a feminist. Right, exactly. But other studies have carefully made sure that that is not all that's going on. Speaker 2 (16:49) Well, and Gricean in the sense that you're telling me that she's a philosophy major. Why are you telling me she's a philosophy major? And so I'm sympathetic to that criticism, Tamler, too. I think the body of work showing that people use heuristics, rather than make some sort of calculated probability is on the whole, like I buy it, I believe it, whether or not it was ever an argument we should have had. I don't know this one. It always did bother me a little bit because it's like, why are you telling me that she's a philosopher then? Like if you're not actually, well, there's a Speaker 4 (17:20) different version. Is this subject to the same criticism? You tell people that there was a, what are the odds? There's a terrible earthquake. that devastates and kills millions of people, is it more likely it happens in California or is it more likely it happens somewhere in the United States and people will say California? Speaker 5 (17:33) Because again, but in that case, they must think somewhere else in the United States. I mean, maybe, maybe. I Speaker 4 (17:40) mean, I don't know. Okay, are we leaning towards a B here? I'm leaning strongly towards a D, but... Oh my God. Speaker 2 (17:47) Yeah. How about a compromise at C then? I could Speaker 4 (17:51) live with Speaker 2 (17:51) a C. Speaker 4 (17:51) All right. Well, actually, it's another S for me, but still. Wow. I'm kind of surprised. It's capturing my love of psychology, actually. You listen to these wonderful studies. Speaker 2 (18:00) Yeah. The other one that we didn't include as a possibility would have been the framing, the framing effects, which I think I'm more of a fan of. I think I am, too. Speaker 4 (18:07) Because David suggested framing. Oh, that's too complicated. Let's go for Leonard Bankteller. But maybe framing would have done better. Okay. Speaker 5 (18:12) I remember, like, I don't know which framing you were considering, but. The Asian disease one. Oh, yeah. Yeah. I would have been a little more sympathetic to that, but this is fine. Okay. Speaker 2 (18:23) Okay. Next one up, Elizabeth Loftus. Your advisor, right? Speaker 4 (18:26) My Speaker 2 (18:27) postdoc, one of my postdoc advisors. Yeah. So this is Loftus and Palmer in 1974 published a finding that really influenced deeply how we view memories and false memories specifically. So the finding was that you show people a film, I guess at the time, of a car crash. Speaker 2 (18:48) The idea was that these people would be witnessing a car crash and then later on you would ask them what they saw. And it was trying to mimic what would happen in real life. In fact, I think it was funded by a grant from the National Highway Traffic Safety Administration. And Loftus told me she was trying to find a way. She knew they were funding. And so she's like, how can I do something memory that has to do with like highway travel? And so what they found was that when you ask the... witness the participant afterwards how fast the cars were going it mattered whether you used the word smashed or hit and then some other versions of it collided bumped yeah and depending on sort of the magnitude of the word the vibe that word gives you they reported that the cars were going faster or slower showing really that leading questions could influence recall or at least reported memory for events. Speaker 5 (19:47) Right. Think how much in juries, like how much this stuff would matter or for witnesses. One of Speaker 4 (19:52) the many sweet features of this is that it's both practical in the ways you're talking about real world implications, but also theoretically very interesting. There's a lot of people have a view of memory that you sort of have a video recording of some sort and then you just access it. But the idea that the act of answering a question with something shapes forever. how you'll remember something really forces you to think about memory in a very different way. Yeah. Speaker 2 (20:14) People even reported seeing glass at the scene of the accident, even though there wasn't any there. No, they would ask them Speaker 4 (20:19) questions about, was there glass at the scene of an accident or something? Yeah. Yeah. Speaker 2 (20:23) It really did change, you know, that and that, that whole body of work, starting with that changed the way that the law has to deal with these things. Speaker 4 (20:30) Dave, you know more about this than I do, but Loftus is a rather controversial character in our field. Speaker 2 (20:35) Yeah, or at least in psychology, because one of the things that she ended up studying with memory and false memory was the whole sort of like satanic panic and reports of ritual abuse in childhood that were propped up by therapists claiming that they had recovered, spontaneously recovered repressed memories. And she argued pretty strongly that that was actually just implanted memory. This actually, I think, might have caused a big rift between the APA and the forming of the APS. Oh, interesting. Yeah. So I love this study. I think it's influential. Great. It has real world application. It influenced the law, changed our view of memory. We Speaker 5 (21:10) don't have to believe all women anymore. Thank Speaker 2 (21:13) goodness. Sure. Rough couple of years there. Did she Speaker 4 (21:22) testify for our rewind scene? Or is that a false memory of my own? Speaker 2 (21:27) I don't know. She may have. I mean, she viewed it as her ethical duty to testify in the defense of basically everybody who was asked. Yes, Speaker 4 (21:35) she did. She testified on memory Speaker 5 (21:37) fallibility. Speaker 2 (21:38) So, Speaker 5 (21:39) you know, I think it is, like Paul said, theoretically interesting and contributes to a view which I take it as fairly settled that memory is a construction rather than... you know, you're rewinding the tape in your mind and seeing it like that, I think is super interesting material too. And this is just a little glimpse into that. That's Speaker 2 (22:02) right. Speaker 5 (22:02) It's a Speaker 2 (22:02) good one. Yeah. I mean, I love it. Like I might be tempted to put it in S tier. Like I'll say the one thing about the body of work that's always been like a little bit, been a little bit unsure of is how much participants just believe. that something happened versus actually remembered that it happened. And they've done work to tease those apart. But for some of the studies, like the lost in the mall study that she did later on, where you tell people that they were lost in the mall when they were kids and you show them like a Photoshop picture or something of them, you know, there's a hot air balloon version. People say, yeah, okay, I remember that. There is like a kind of a slippery divide between memory and belief. That Loftus herself says, look, what I care about is whether people say they remember something. Like that's going to be what convicts you. Whether or not there's like the trace, as you know, to use their lingo or how much belief influences it is less of a concern for her. I could live with an S. I'm more thinking of an A. I Speaker 4 (22:58) think Speaker 2 (22:59) the world would just work, Speaker 4 (23:00) but it lacks the sort of sexiness that you need to get to the S level for me. Speaker 5 (23:04) Yeah, I'm with you there. And also, like, I do think you might be able to run a little bit of the Gricean thing. on this one too, but it seems less problematic. So, you know, saying if you say smash, which might tie into your idea that it's more about belief than memory. Speaker 2 (23:21) That's exactly the kind of thing I was thinking. Yeah. Yeah. All right. Speaker 5 (23:24) Hey, sounds good to me, Beth. I Speaker 2 (23:26) know you're listening. Speaker 5 (23:29) He may need you a little later on. He may need you Speaker 2 (23:32) to testify. Oh, man. You know, the one person she refused to testify on their behalf was somebody who was recognized as a Nazi war criminal. And they asked her to testify. And she gives a talk, I think in her, she has a TED talk where she discusses this. And she said that just emotionally she couldn't bring herself to testify for somebody who may have been a Nazi war criminal. But in her defense, she found a colleague that she thought was as... good at who wouldn't have that emotional attachment and recommended him. Speaker 5 (24:04) And also she was very busy testifying for Epstein at that time. So, you know, was Speaker 2 (24:09) it Epstein? Speaker 5 (24:10) She didn't testify for Epstein. Did she? Speaker 2 (24:11) Weinstein. No, Speaker 4 (24:14) that was a joke. I wasn't saying no, no, she did. She did testify for Epstein. In some ways I respected that