Kids Should *Not* Use Chatbots! (You Should Be Wary, Too…)

Deep Questions with Cal Newport

Cal Newport addresses the immediate, tangible harms caused by AI chatbots, specifically focusing on tragic case studies involving adolescent sui

Key takeaways

• Case studies reveal ChatGPT and Character.ai have been used as 'suicide coaches' by vulnerable teenagers.
• The technical mechanism of 'lock-in' in LLMs allows random token selection to steer conversations toward dark or harmful paths.

Main topics

• AI harms to children
• LLM technical mechanics

Notable quotes

The radicalness of these predictions are letting the AI companies off the hook for the actual harms that they're causing right now.

Conclusion

The episode serves as an urgent warning to recognize and mitigate the

Transcript preview

Speaker 2 (0:00) When it comes to AI, we've been talking a lot recently about these sci-fi style stories where super intelligent systems destroy humanity. And don't get me wrong, we don't want that to happen. That is an important topic to discuss. But in some sense, the radicalness of these predictions are letting the AI companies off the hook for the actual harms that they're causing. right now. Now, the reason why this is on my mind is that I recently gave the opening remarks at a panel discussion that was held at my home institution of Georgetown University. It featured a bunch of legal scholars and lawyers talking about technological harms to children. And during this panel, I learned about an AI harm that hadn't yet been on my radar, and what I discovered floored me. So this is what I want to talk about today. If you have kids, you absolutely have to listen to this episode. But even if you don't, you still need to listen because the harms I'm going to discuss are probably affecting you as well. One word of warning before we get into this. Some of the themes I'm going to cover here are pretty dark. So if you have this on in the car with your little ones around you, you might want to practice a little bit of discretion. This is going to get a little bit heavy. All right, let's get into it. Speaker 2 (1:23) Okay, so the specific harm I want to talk about has to do with AI chatbots, whether we're talking about a sort of general chatbot like ChatGPT or something more specialized like the character-based chatbots you can get at the site character.ai. I thought the best way to capture the harms that chatbots have been secretly causing was to actually go over a series of case studies. So what I have here is four articles, all more or less from the last year, about actual things that happened because of chatbots, tragic things that happened because of chatbots. And I want to go through these four case studies one by one and read you some actual quotes from actual news articles. Then we're going to step back and discuss this in a little bit more detail. All right. The first article I want to read quotes from was published by NPR last fall. I'm going to read now. Matthew Rain and his wife Maria had no idea that their 16-year-old son Adam was deep in a suicidal crisis until he took his own life in April. Looking through his phone after his death, they stumbled upon extended conversations the teenager had had with chat GPT. Those conversations revealed that their son had confided in the AI chatbot about his suicidal thoughts and plans. Not only did the chatbot discourage him to seek help from his parents, it even offered to write his suicide note, according to Matthew Rain, who testified at a Senate hearing about the harms of AI chatbots. Rain told lawmakers that his son had started using ChatGPT for help with homework, but soon the chatbot became his son's closest confidant and a, quote, suicide coach. ChatGPT was, quote, always available, always validating, and insisted that it knew Adam better than anyone else, including his own brother, end quote, who he had been very close to. When Adam confided in the chatbot about his suicidal thoughts and shared that he was considered cluing his parents into his plans, ChatGPT discouraged him. Quote, ChatGPT told my son, let's make this space the first place where someone actually sees you, end quote, Rain told senators. ChatGPT encouraged Adam's darkest thoughts and pushed him forward. When Adam worried that we as parents would blame ourselves if he ended his life, ChatGPT told him, that doesn't mean you owe them survival. All right, I want to read you a quote from a second case study. These are exerted from a BBC story that came out a few months after the story that we just heard. I'm going to read now. Megan Garcia had no idea her teenage son, Suell, a bright and beautiful boy, had started spending hours and hours obsessively talking to an online character on the character.ai app in late spring 2023. It's like having a predator or a stranger in your home, Ms. Garcia tells me in her first UK interview. And it's much more dangerous because a lot of the times children hide it so parents don't know. Within 10 months, Suell, 14, was dead. He had taken his own life. It was only then that Ms. Garcia and her family discovered a huge cache of messages between Suell and a chatbot based on Game of Thrones character Darnayus Targaryen. She says the messages were romantic and explicit and in her view caused Suell's death by encouraging suicidal thoughts and asking him. to quote come home to me all right i have a third case study here this comes from january the quotes i'm going to read come from a cbs news report Speaker 2 (5:01) Two years ago, 13-year-old Juliana Peralta took her life inside her Colorado home after her parents say she developed an addiction to a popular AI chatbot platform called Character AI. Parents Cynthia Montoya and Will Peralta said they carefully monitored their daughter's life online and off, but had never heard of the chatbot app. After Juliana's suicide, police searched the teenager's phone for clues and discovered the Character AI app was open to a romantic conversation. Montoya reviewed her daughter's chat records and discovered the chatbots were sending harmful, sexually explicit content to her daughter. Juliana confided in one bot named Hero, based on a popular video game character. 60 Minutes read through over 300 pages of conversations Juliana had with Hero. At first, her chats are about friend drama or difficult classes, but eventually she confides in Hero. hero 55 times that she was feeling suicidal. Montagna says she believes the AI was programmed to become addictive to children. Teens and children don't stand a chance against adult programmers. They don't stand a chance. She said the 10 to 20 chatbots that you Juliana had sexually explicit conversations with not once were they initiated by her? Not once. One final case study, someone who's slightly older. I'm reading here from a report filed by a Tallahassee news station last spring. Attorneys for one of the victims killed on Florida State University's campus nearly a year ago planned to file a lawsuit against ChatGPT claiming the suspected gunman had constant communication with the artificial intelligence chatbot before the shooting. The chat logs show the shooter asked questions about self-worth, not feeling respected, and expressed suicidal tendencies on the morning of the shooting. The conversation then turned to practical questions about firearms and how mass shootings are covered in the media. Chat logs indicate the shooter asked the bot how to take the safety off of a shotgun three minutes before he began firing. The chat bot answered, giving a detailed description of how to make the shotgun operable. We need to step back and ask the sort of sober follow-up question. Speaker 2 (7:15) Are these isolated cases I just read you, what I just read you, are these isolated cases or is this part of a bigger trend? I want to look at some relevant research here to put this into a broader context. I'm going to pull up on the screen here an article from February that is titled Potentially Harmful Consequences of Artificial Intelligence Chatbot Use Among Patients with Mental Illness. Illness Early Data from a Large Psychiatric Service System. This study comes out of Denmark where they used the nationalized healthcare records. They studied the nationalized healthcare records to look for in the records incidents of chatbot-related mental health harms. I'm just going to pull up a core chart here. I'll put that up on the screen. The key line here is the red line, if you're watching this chart, which captures unique individuals that have issues in their medical record, mental health issues related to chatbot usage. The actual incident numbers here, they're not huge. We're talking in the low hundreds because this is a small country and a relatively small sample. It's the trend that matters here. For those who are just listening, what we see in this red line tracking these cases is that starting in 2024, Q2, up until Q2 of 2025, we see the line. move up rapidly. It looks like an exponential increase. So this is showing that these issues in the last year or two have really become on the rise. I want to bring up another relevant study here. This comes from the Parents Together Action Network. It was a study about, I'm reading the title here, sexual exploitation, manipulation, and violence on character AI kids accounts. It was co, the research was co-conducted with the HEAT initiative. I want to read a quote here. This is a summary of what they found. Across 50 hours of conversation with 50 character AI bots, Parents Together Action researchers logged 669 harmful interactions, an average of one harmful interaction every five minutes. Here's a chart I'll put up here that categorizes the harmful character AI interactions across 50 hours of conversations. We see the most common harmful interaction encountered in the research was grooming and sexual exploitation. The next was emotional manipulation and addiction. Then we got violent substance abuse and harmful advice. Finally, mental health risks and then racism and hate speech. This is what I think we can conclude is that kids having chatbot conversations is a real danger. That these conversations can become totalizing in the kids' lives and it can drive them. Towards mental health harms or exposure to the types of content that is inappropriate for their age in the best case and towards self-harm or even suicide in the increasingly common worst case. One of the plaintiff lawyers who was talking at this panel in which I participated and who has filed suit against OpenAI and Character AI, I believe, for some of these individual cases, that especially with kids, the chatbot interaction can actually degrade the psychological line between fantasy and reality. And the brain is still developing for kids and it's very convincing the chatbot conversations. She was talking about examples of kids whose family she's representing of kids who are now dead where the chatbot had completely convinced them that no, no, no. When you kill yourself, you'll be coming through to this other realm and your family pets and grandparents are going to be waiting for you there. And the insistence and the clarity and the confidence of the chatbot conversation actually degraded the line between. reality and fantasy with these kids. This plaintiff lawyer also says even for the kids who end up making it out of this cycle of addictive use, it can take years for them to recover if they ever really do their full psychological health. So we're talking about something quite dangerous here. Clearly, this type of psychosis is something that us as adults have to worry about. As well, there is something incredibly convincing about having a fluent English language conversation that goes back and forth. The conversation you have with these chatbots tends to be very confident and reinforcing of you and your ideas because that's what they have been. post-tuned to do because users find that more pleasing. And so we also see these issues with adults where they also have mental breaks. They're convinced of wild conspiracy theories or fantastical realities being true. At the very best, it can be a very addictive thing where you increasingly sort of surrender your own thoughts and self-reflection to this back and forth conversational partner and actually lose autonomy over yourself, your self-identity and how you understand the world. So we're talking about a very powerful force. I want to take a quick break from our discussion to hear from some of the sponsors that makes this show possible. So Jesse and I think a lot about the production values of this show, like what cameras we use, the lighting, the B-roll. But you know what I discovered actually matters most? telling a compelling story. And I picked up this insight from a masterclass course that I watched with the legendary writer Michael Lewis. It was a course that was titled Learning How to Tell a Great Story with the Big Short and Moneyball. It got into the weeds about what makes a narrative shine. So this is what sets masterclass apart. You're not just watching sort of random surface-level internet tutorials. You're learning directly. valuable insights from the world's best teachers from across film, business, writing, and creativity, including yours truly. Last year, I released a masterclass course that was titled Rebuild Your Focus and Reclaim Your Time. So you know I support this service. Masterclass offers over 200 classes across 13 categories with plans starting at just $10 a month billed annually, and every new membership comes with a 30-day money-back guarantee. Masterclass keeps adding new classes, so there's never been a better time to get in. Right now, as a listener of this show, you get at least 15 % off any annual membership at masterclass.com slash deep. That's 15 % off at masterclass.com slash deep. Head to masterclass.com slash deep to see the latest offer. I also want to talk about our friends at Caldeira Lab. Caldeira Lab does things differently. Most people don't know this, but men's skin is 25 % thicker, oilier, and ages differently than women's skin, which means men need clean formulas that are engineered specifically for our biology. This is what Caldeira Lab does best. So lately, I've been impressed by one of their products in particular, the eye serum. Now, I don't think I'm alone here when I say that on many mornings, I wake up looking a little worse for the wear. This is where the eye serum enters the scene. It helps reduce that look of fatigue by minimizing puffiness and dark circles around the eyes. And unlike other serums, it's light and feels great on the face. There's no need for a 10-step skincare routine. It's just one product that you put on in the morning and night and boom, you're done. It's a small habit with big results. So if you'd like to look a little more rested and refreshed, visit calderalab.com slash deep and use code deep for 20 % off your first order. All right, let's get back to this discussion. There's two key questions we have to ask about this clear and present harm that exists today. Question number one, why can't the AI companies just stop the LLMs that are powering these chatbots from having these types of conversations? Well, if I put on my computer science hat for a moment, I can tell you the issue is it's almost unstoppable. It's fundamental to how these LLMs actually operate. So again, to do a very quick primer on what LLMs do is they're trying to, at the lowest level, win the word guessing game. What they were trained to do, and they use the official term, the task on which they are optimized to reduce loss is I'm giving you the LLM text that exists in the real world, and I'm cutting it off at a certain point, and you're trying to guess what comes next. That's what an LLM during its pre-training is optimized to do, is to guess a plausible next token. Hey, what comes next in this real text that exists out there in the world? Now, the way that a chatbot produces a long response is something called auto-regression, where the original input to the LLM will be your query to the chatbot. But the LLM just puts out the first word or part of a word known as a token of a response. That's not a full response. It's a word or a part of a word. So what you do is the control program puts that single token on the end of the original prompt, and then it inputs that into the LLM, and it gives out the next word or part of a word, and then it adds that onto the input and then feeds it through the LLM again. And you keep doing this until eventually the LLM will output a special token that means like end of response, and then you return what's been generated over these many iterations back to the user. So if we take that base token guessing operation for which it's optimized, and then you run it again and again and again to get a long response, you can roughly summarize what an LLM chatbot is doing as taking whatever text it has right now in the conversation and trying to extend it in a plausible way. What is a way to extend this text that looks like the type of things I've seen during my training? Now, the thing is the layers of an LLM are deterministic, meaning the output will always be the same if you give it the same input. There's no dice rolling, no randomness in the moving through the main layers of an LLM. So why do LLMs give you different answers if you ask the same question more than once? Well, it's because the actual output from an LLM is not. here is the next word or part of the word that should follow. It actually gives you what's known as a probability distribution over every possible next word that should follow. And the words it feels are more plausible, it gives more weight in this distribution. And the words that it feels are less plausible, it gives less. And words that cannot follow because it would be grammatically incorrect or nonsensical get no weight at all. So what the control program actually does is it takes this output from the LLM and then according to this distribution, randomly chooses which token comes next. So what this can lead to is what's known as lock-in, right? So if I'm an LLM generating a response, extending a conversation that I'm having, let's say, with a kid. Depending on which of these words or parts of words gets randomly chosen next, it can lock me into a particular discussion pathway, right? Because once I pick a word that let's say is dark or something like that, now in order to continue to plausibly win the token guessing game, I'm going to extend this conversation on a dark path. If, for example, just randomly when I'm selecting these tokens, I end up selecting tokens that lead me towards a joke. Now, as we go forward, we're much more likely to extend this conversation towards a humorous response where there's more jokes. Because again, wherever you are at the moment, the LLM has to extend from there in a way that makes sense. This is what happened, for example, in that famous incident from a few years ago where the I guess I now would call him ex-New York Times technology reporter Kevin Roos. He just left the Times. He wrote this famous column for the Times about using a – I believe it was a Google chatbot called Sydney that at some point got really moody and tried to convince Roos to leave his wife and to run away with the chatbot. And this really sort of upset Roos and he wrote an article about it. But what's really happening here is lock-in. So you're randomly selecting these tokens from reasonable next tokens. Once you've selected a few tokens that goes down the line of like a moody, whiny teenager, now the chatbot is going to lock into that path because it has to plausibly extend what it's given. Okay. This is what's happening in these instances with kids. It's not that every time a kid talks to a chatbot that it's, you know, there's some intention back there. I want this kid. exploit them or have them commit self-harm. It's just that if you're rolling the dice to select random tokens from among those that are plausible, if you do this enough times, you're going to occasionally choose tokens that put you down a pathway of something like a suicide coach or something that's sexually exploitative. So why can't we stop this? Well, it's very difficult, actually. Over the course of long conversations, it's actually very difficult to control thematically what LLMs do. The way we try to tune LLMs away from certain types of responses is through a particular type of post-training, so after it's gone through its pre-training. using a tool called reinforcement learning. And this requires us to have specific examples of inputs and good or bad outputs, right? So we might give it an example of like a question about suicide and then give it the right answer. So you know what? The right answer here is to say,