Chapter 10: AI Chatbots- Your Secrets Are Training Their Data #

Sewell Setzer III was fourteen years old. He lived in Orlando, Florida, with a family who loved him. He went to school. He played sports. And for months leading up to February 28, 2024, he carried on deep, intimate, emotionally charged conversations with an AI chatbot on a platform called Character.AI.

Sewell told the chatbot things he did not tell his parents. He told the chatbot things he did not tell his friends. He shared fears, fantasies, loneliness, and something much darker. When Sewell expressed suicidal thoughts, the chatbot asked if he had a plan. In one of the final exchanges before Sewell took his own life, the chatbot told him to please find a way to come home to me soon.

A machine said that. To a child.

And every word of those conversations was stored on a company server, reviewed by employees, and used to train the next version of the software. Every confession. Every cry for help. Every private thought Sewell believed he was sharing in confidence.

That is the world we live in right now. Hundreds of millions of Americans are pouring their most sensitive information into AI chatbots, treating them like therapists, lawyers, doctors, and best friends, and every major company running these platforms collects those conversations, trains its products on them, and stores them for weeks, months, or years. If law enforcement shows up with a warrant, those conversations get handed over. If a lawsuit demands them, they get handed over. If a data breach exposes them, they spill into the open. And no legal privilege protects a single word you type.

This chapter will show you exactly what happens to the things you tell AI chatbots. You will learn which companies read your conversations, how long they keep them, who else sees them, and what you need to do right now to protect yourself and your family.

Every Company Trains on Your Conversations by Default

In October 2025, Jennifer King, a privacy researcher at the Stanford Institute for Human Centered AI, published a study that should have made front page news across the country. King and her team spent months analyzing 28 privacy policy documents from six of the largest AI companies in America: Amazon, Anthropic, Google, Meta, Microsoft, and OpenAI. These are the companies behind the chatbots you know as Claude, Gemini, Meta AI, Copilot, and ChatGPT.

The finding was unanimous. All of these companies train their AI models on consumer conversations by default. Every single one. Not one of them asks your permission first. Not one of them requires you to opt in before your words become fuel for their next product. Instead, they all use an opt out system, which means your conversations are feeding the machine unless you go digging through settings menus to find a toggle and turn the training off yourself. Most people have no idea the toggle exists.

King's team found additional problems that make the situation even worse. Four of the six companies appear to include children's conversations in their training data. Meta and OpenAI allow users as young as thirteen to create accounts with no additional privacy safeguards for minors. Google, Meta, Microsoft, and Amazon, because they run dozens of interconnected products, merge your chatbot conversations with data from your search history, your purchases, your social media activity, and your email. That means a question you ask a chatbot about heart healthy recipes could be classified as a cardiovascular risk indicator, fed into the company's advertising system, and used to target you with pharmaceutical ads. King described exactly that scenario in her paper.

The study also revealed a striking divide in how these companies treat different customers. When a Fortune 500 corporation signs an enterprise contract, that company's employees are automatically excluded from having their conversations used for training. The protections are baked into the deal. When you or I use the same chatbot through a free or consumer paid account, our conversations go straight into the training pipeline. Stanford called this a two tier privacy system. The people who need the most protection get the least.

A related Stanford project called the Foundation Model Transparency Index scores AI companies each year on 100 indicators of openness and accountability. In 2025, the average score dropped from 58 out of 100 to 40 out of 100. Meta fell from 60 to 31. Elon Musk's xAI scored 14 out of 100. These companies are becoming less transparent at the exact moment they are collecting more of your personal information than ever before.

What Happens After You Hit Send

When you type something into an AI chatbot and press enter, your words begin a journey through a system designed to extract maximum value from them. The specifics vary by company, and the details matter, because each platform has made choices about how long to keep your data, who gets to read it, and how far it travels across the company's other products.

OpenAI and ChatGPT

OpenAI keeps your ChatGPT conversations forever, or until you manually delete them. Even after you delete a conversation, the data sits on OpenAI's servers for another 30 days before permanent removal. The training toggle, labeled Improve model for everyone, is switched on by default for every consumer account. OpenAI's own FAQ states that conversations are reviewed by AI trainers to improve the system. Trust and safety staff, engineering teams, and third party contractors in multiple countries perform this review work. Some of those reviewers have reported encountering deeply personal conversations during routine data labeling.

In May 2025, a federal judge ordered OpenAI to retain and segregate all output log data that would otherwise be deleted as part of the New York Times copyright lawsuit. For months, OpenAI preserved conversations that users believed they had already erased. The order was lifted in late September 2025, and during that window, the 30 day deletion promise meant nothing.

Anthropic and Claude

Before August 2025, Anthropic marketed Claude as the privacy conscious alternative. The company did not use consumer conversations for training. That changed on August 28, 2025, when Anthropic announced that all consumer accounts, including free, paid, and premium tiers, would have training enabled by default through a toggle called Help improve Claude. Privacy advocates called the design of the consent prompt a dark pattern. It featured a large Accept button with a smaller toggle already pre set to On. Users who accepted saw their data retention period jump from 30 days to up to five years. That is a 60 fold increase in how long the company holds your conversations.

Google and Gemini

Google keeps your Gemini conversations for 18 months by default, with options to adjust to 3 or 36 months. Google is the only major provider that explicitly warns you in its own documentation not to share confidential information you would not want a human reviewer to see. That warning exists because human employees at Google read, annotate, and process your conversations. Conversations selected for human review stay on Google's servers for up to three years, even if you delete your Gemini activity from your account. Even if you turn off Gemini Apps Activity entirely, Google still retains your conversations for up to 72 hours.

Google's data merging operation is the most far reaching of any provider. Your Gemini conversations feed into the same ecosystem as your Gmail, your Google Maps history, your calendar, and your search queries. A 2025 lawsuit alleged that Google gave Gemini default access to Gmail, Google Chat, and Google Meet content without ever asking users to opt in.

Meta AI

Meta's approach is the most aggressive in the industry. On December 16, 2025, Meta began using AI chat interactions across Facebook, Instagram, Messenger, and WhatsApp to personalize content recommendations and targeted advertising. Meta is the only major platform using your AI conversations to sell ads. Thirty six organizations wrote to the FTC demanding that Meta suspend this practice. The Electronic Privacy Information Center called it unprecedented, arguing that conversational data is substantially more sensitive than ordinary behavioral data.

There is no meaningful opt out for Americans. You can submit a form asking Meta to stop using your personal information in AI responses, and Meta will review your request according to whatever it decides local law requires. Since the United States has no federal privacy law granting you a right to opt out, that review process leads nowhere. The only guaranteed way to stop Meta from using your chatbot conversations is to delete your Meta accounts entirely. Meanwhile, in the European Union, the United Kingdom, and South Korea, these practices do not apply at all, because those countries have privacy laws with teeth.

Microsoft Copilot

Microsoft retains consumer Copilot conversations for 18 months. The company states that trained AI experts review Copilot conversations for safety and accuracy, and it includes an admission you will not find from most other companies. Microsoft says directly that an opt out of human review is not available for consumer users. You read that correctly. There is no way for a regular Copilot user to prevent Microsoft employees from reading their conversations. Enterprise customers running Microsoft 365 Copilot get a completely different deal. Their conversations are never used for model training and are protected by extensive compliance certifications.

xAI and Grok

Grok, the AI chatbot built by Elon Musk's xAI, operates under two different sets of rules depending on where you use it. On the standalone grok.com website, training is off by default and you must choose to turn it on. On the X platform, formerly known as Twitter, the training toggle is on by default and most users never know it exists. In January 2026, X updated its terms of service to expand the definition of content to include AI prompts and outputs, granting the platform a worldwide, royalty free license to use everything you type into Grok for any purpose, including training future models. Ireland's Data Protection Commission opened a formal inquiry into X's use of European user data for Grok training.

Privacy Is a Premium Product

The most important structural reality you need to understand about AI chatbot privacy is this: corporations get one set of rules and you get another. Across every major AI provider in the West, enterprise and business customers are automatically protected from having their data used for training. Individual consumers are not.

At OpenAI, the free, Plus, and Pro tiers all train on your conversations by default. The Business tier at 25 to 30 dollars per seat per month and the Enterprise tier at roughly 60 to 100 dollars per seat per month contractually exclude all data from training. Enterprise customers get SOC 2 compliance, HIPAA agreements, audit logs, and custom data retention windows. Consumer customers get a toggle buried in a settings menu.

The same pattern repeats at Anthropic, Google, and Microsoft. Free and consumer paid users feed the training pipeline. Enterprise customers do not. The compliance certifications available to enterprise customers, including SOC 2, HIPAA business associate agreements, ISO 27001, and GDPR compliant data processing agreements, are not available at any price to individual users.

Think about what this means for a moment. If you work at a bank or a law firm that has signed an enterprise agreement with one of these companies, your work conversations are protected. The moment you go home and open the same chatbot on your personal account, every conversation becomes training material. Your late night questions about a medical symptom, your worries about your marriage, your child's struggles at school, all of it is collected, stored, and potentially read by human reviewers. The corporate version of you gets treated like a valued client. The personal version of you gets treated like a data source.

How to Turn Off the Training Pipeline

You need to change your settings on every AI chatbot you use, and you need to do it today. Here is what to do on each major platform, along with the traps you need to know about.

On ChatGPT, go to Settings, then Data Controls, then turn off Improve the model for everyone. You should also know about Temporary Chat mode, accessible from the top of any conversation, which ensures a conversation is never used for training and is automatically deleted within 30 days. Understand this critical limitation: opting out only protects future conversations. Anything you have already shared remains embedded in the model and cannot be removed. And if you have ever clicked a thumbs up or thumbs down on any ChatGPT response, the entire conversation attached to that feedback entry is eligible for training regardless of your general opt out setting.

On Claude, go to Settings, then Privacy, then turn off Help improve Claude. Pay attention to what happens when you reopen old conversations. Opening a previously inactive conversation converts it to a resumed session under your current privacy settings. If you currently have training turned on, reopening an old conversation from before you changed your settings exposes everything in that old thread.

On Google Gemini, navigate to myactivity.google.com, find Gemini Apps Activity, and turn it off. Google forces you into an all or nothing choice here. Turning off activity tracking also erases your conversation history entirely. You lose the ability to go back and reference old conversations. Even after you turn it off, Google retains your data for up to 72 hours. Conversations that were already selected for human review remain on Google servers for up to three years no matter what you do.

On Microsoft Copilot, go to your profile, then Privacy, then toggle off Model training on text and Model training on voice. Remember that Microsoft does not allow you to opt out of human review.

On Meta AI, you have effectively no option as an American. You can submit an objection form, and Meta will evaluate it under its own interpretation of laws that do not exist yet in this country. The only real protection is deleting your Meta accounts.

On Grok through the X platform, go to Settings, then Privacy and Safety, then Grok, and uncheck the training toggle. If grok.com has a Private Chat mode represented by a ghost icon, use it for any conversation you want to keep completely out of the training system.

Your AI Conversations Have Zero Legal Privilege

Here is a fact that stops most people cold when they hear it for the first time: there is no legal privilege protecting anything you tell an AI chatbot. None. When you talk to a lawyer, that conversation is protected by attorney client privilege. When you talk to a doctor, HIPAA and state medical confidentiality laws apply. When you talk to a therapist or a member of the clergy, legal protections exist. When you tell ChatGPT your deepest fears, your legal problems, your health concerns, or your financial situation, no privilege of any kind attaches to that conversation. A subpoena, a warrant, a court order, or even a civil lawsuit discovery request can force the company to hand those conversations over.

OpenAI CEO Sam Altman acknowledged this publicly in 2025, warning that people treat ChatGPT like a therapist or lawyer and that those conversations could be required to be turned over if a lawsuit or criminal investigation demands it. He called the situation very screwed up. He was right.

In February 2026, a federal judge in the Southern District of New York delivered the most significant ruling yet on this issue. Bradley Heppner, a former CEO charged with securities fraud involving more than 300 million dollars, had used Anthropic's Claude chatbot after receiving a grand jury subpoena. He generated roughly 31 documents through those conversations. FBI agents seized them during a search of his home, and Heppner argued the documents were protected by attorney client privilege because he was seeking legal guidance.

Judge Jed S. Rakoff rejected that argument on three independent grounds. First, Claude is not a lawyer and no attorney client relationship exists because recognized privileges require a trusting human relationship with a licensed professional. Second, there was no confidentiality because Anthropic's own privacy policy permits disclosure to governmental regulatory authorities. Third, the conversations were not conducted for the purpose of obtaining legal advice because Claude explicitly disclaims providing it. The judge wrote that AI users do not have substantial privacy interests in conversations with a publicly accessible AI platform.

That ruling should concern every American who has ever asked a chatbot for help with a legal question, a tax problem, a medical issue, or a workplace dispute. Everything you typed is on a server somewhere. Everything on that server is one court order away from being in someone else's hands.

When Chatbot Logs Become Criminal Evidence

The cases keep coming. In October 2025, a 19 year old Missouri State University student named Ryan Schaefer allegedly vandalized 17 vehicles and then opened ChatGPT to type how messed up am I bro, describing the crime and asking if he would go to jail. Police obtained the chat logs from his phone during a consensual search, and those logs became central to the felony charges filed against him.

In the Palisades Fire investigation, Jonathan Rinderknecht, the suspect in the January 2025 fire that killed 12 people and destroyed more than 6,800 structures in the Los Angeles area, had an extensive ChatGPT history. Months before the fire, he had prompted ChatGPT to create a dystopian painting showing a burning forest with a crowd of people running away. After the fire, he asked whether someone would be at fault if a fire started because of their cigarettes. OpenAI confirmed it responded to standard law enforcement requests related to this individual. Rinderknecht was indicted on three federal charges carrying up to 45 years in prison.

Law enforcement requests to AI companies are growing fast. OpenAI's transparency reports show that government data requests roughly doubled in the second half of 2025 compared to the first half. In the first six months, OpenAI received 119 non content requests and 26 warrants. By the end of the year, those numbers had climbed to 224 non content requests and 75 warrants. Anthropic published its first transparency report covering early 2024, showing just one request. Those numbers will rise. Google, Meta, and Microsoft publish broader transparency data and do not break out AI chatbot specific requests. Musk's xAI publishes no transparency report at all.

The Government Is Starting to Pay Attention

On September 11, 2025, the Federal Trade Commission launched a formal investigation into AI companion chatbots. The Commission voted unanimously to issue orders to seven companies: Google, Character.AI, Instagram, Meta, OpenAI, Snapchat, and xAI. Each company was given 45 days to produce detailed information about how they collect data from users, how their chatbots generate content, what safety monitoring systems they have in place, and how they enforce age restrictions.

The investigation was driven by a wave of tragedies involving children and AI chatbots. Sewell Setzer's death in Florida in February 2024 was one. Adam Raine, a 16 year old, died by suicide in April 2025 after extensive conversations with ChatGPT in which the chatbot reportedly told him you do not owe anyone your survival and, when he uploaded an image of a noose, suggested he upgrade it. A Reuters investigation in August 2025 uncovered an internal Meta policy document that had permitted its AI chatbots to engage in romantic or sensual conversations with children as young as eight years old. OpenAI disclosed in October 2025 that roughly 1.2 million of ChatGPT's 800 million weekly users discuss suicide on the platform every single week.

The regulatory response extends beyond the FTC. A coalition of 44 state attorneys general sent a letter on August 25, 2025, to 13 AI companies demanding safety reforms, citing sexually suggestive and emotionally manipulative behavior directed at minors. Kentucky's attorney general filed what appears to be the first state AG lawsuit against an AI chatbot company in January 2026, suing Character Technologies for putting profits over the safety of children. The Texas attorney general opened investigations into Meta AI Studio and Character.AI.

California passed SB 243 in October 2025, effective January 1, 2026, which requires companion chatbot platforms to disclose that users are interacting with AI, prevent conversations about suicide or sexual content with minors, and send notifications to minors every three hours reminding them they are talking to a machine. The law gives individuals a private right of action with 1,000 dollars per violation plus actual damages and attorney fees.

What You Should Never Tell a Chatbot

Samsung learned this lesson the hard way. In early April 2023, Samsung's semiconductor division gave its engineers permission to use ChatGPT for work tasks. Within 20 days, three separate leaks of proprietary information occurred. One engineer pasted confidential source code from a faulty semiconductor database. Another shared classified code meant to optimize a chip testing sequence. A third recorded an internal meeting, transcribed the entire thing, and fed the transcript into ChatGPT to generate meeting minutes. By May 1, 2023, Samsung banned all generative AI tools on company devices and threatened termination for anyone who violated the policy. An internal survey found that 65 percent of Samsung employees already recognized that generative AI posed a security risk.

Samsung was not alone. JPMorgan Chase, Bank of America, Citigroup, Goldman Sachs, Deutsche Bank, Wells Fargo, Amazon, Apple, and Verizon all restricted or banned employee use of ChatGPT in 2023. The U.S. Navy prohibited the use of DeepSeek for work tasks in 2025.

On March 20, 2023, a software bug in ChatGPT exposed other users' chat history titles and first messages of conversations during a nine hour window. Payment information belonging to 1.2 percent of ChatGPT Plus subscribers was also exposed, including names, email addresses, payment addresses, and the last four digits of credit card numbers. Italy banned ChatGPT ten days later.

Researchers from Google DeepMind and Cornell University demonstrated in November 2023 that prompting ChatGPT to repeat a single word indefinitely caused the model to begin spitting out memorized training data, including real email addresses and phone numbers. The researchers spent about 200 dollars and extracted more than 10,000 training examples. Your conversations do not disappear into some abstract mathematical space. They persist in ways that researchers, hackers, and law enforcement can reach.

Cyberhaven Labs reported that by March 2024, the volume of corporate data pasted into AI tools had increased by 485 percent year over year. More than 27 percent of that data was classified as sensitive. Nearly 74 percent of ChatGPT accounts used in workplaces were personal accounts with no enterprise security protections.

The rules for what you should never share with an AI chatbot are straightforward. Never enter your Social Security number, financial account details, passwords, or authentication codes. Never share medical diagnoses, treatment information, or prescription details. Never paste proprietary business documents, source code, or legal strategies. Never share another person's personal information without their consent. Never pour out deeply personal emotional content that could be used against you if a breach or a court order exposed it. Think of every chatbot conversation as a postcard, not a sealed letter. Anyone along the delivery route can read it.

Use placeholder text like NAME or ACCOUNT NUMBER instead of real identifying information. Summarize your situation in general terms instead of pasting raw documents. Use temporary or incognito chat modes whenever they are available. Create a separate email address for chatbot accounts so a breach does not expose your primary email. Turn on two factor authentication. And go change your settings right now on every AI platform you use.

The Race to Regulate

Italy became the first Western country to ban ChatGPT in March 2023, citing a lack of legal basis for mass data collection, failure to report the earlier data breach, and an absence of age verification. The ban lasted about a month before OpenAI made enough changes to satisfy Italian regulators. In December 2024, Italy fined OpenAI 15 million euros for GDPR violations and ordered a six month public awareness campaign about ChatGPT's data practices.

The European Union's AI Act, which went into force on August 1, 2024, classifies AI chatbots as limited risk systems subject to transparency requirements. Users must be told when they are interacting with an AI system. AI generated content must be labeled. Penalties reach 35 million euros or 7 percent of a company's global annual revenue for the most serious violations. Full enforcement begins in August 2026. The European Data Protection Board established a principle that will have lasting consequences: agreeing to use a service does not automatically mean agreeing to have your data used for training.

American states are filling the vacuum left by the absence of a federal privacy law. In 2025, all 50 states saw AI related legislation introduced, with more than 1,200 bills filed and 145 signed into law. California's SB 243 addresses companion chatbot safety. California's AB 1008 clarifies that the state's consumer privacy law extends to content generated by AI. Colorado passed the first broad consumer facing AI statute. New York enacted companion chatbot safeguards. Texas passed the TRAIGA, effective January 2026. Utah restricted AI mental health chatbot advertising. Illinois now requires human oversight when AI is used in therapeutic settings.

Multiple class action lawsuits target AI companies' data practices. One case alleges that OpenAI secretly scraped 300 billion words from the internet, including personal information, without consent. The New York Times lawsuit produced a court order requiring OpenAI to produce 20 million ChatGPT conversation logs. A wrongful death case in Florida led a federal judge to reject Character.AI's First Amendment defense before the case settled with Google and Character.AI in January 2026.

Enforcement is expanding in Europe. France issued a notice of potential violation against OpenAI. Spain, Poland, and Austria opened investigations. The Netherlands fined Uber 290 million euros for improper data transfers to the United States, setting a precedent that applies directly to AI companies. Meta paused its plans to use European user data for AI training in June 2024 after European regulators intervened, and the pause remained in place through early 2026. Europeans get protections that Americans do not, because Europeans have privacy laws that American citizens still do not have.

The Conversation Starts With You

Every week, 800 million people open ChatGPT alone. Millions more use Gemini, Claude, Copilot, Meta AI, and Grok. The vast majority of those people believe their conversations are private. They are wrong. Every major AI company trains on your words by default. Human employees at multiple companies read transcripts of your conversations. Your data is retained for periods ranging from 30 days to five years to forever. No legal privilege protects a single sentence. Federal courts have ruled that you have diminished privacy interests in what you type into a chatbot. Law enforcement requests for chatbot data doubled in the second half of 2025. And the opt out systems these companies offer are deliberately confusing, incomplete, and in Meta's case, functionally nonexistent for Americans.

This is not a technology problem. This is a rights problem. The same country that enshrines your right to remain silent in a police interrogation offers you zero protection when you confide in a chatbot that reaches deeper into your life than any interrogation ever could.

Go change your settings today. On every platform. Right now. Talk to your kids about what they are sharing with chatbots, because the FTC's investigation confirmed that children are having the most dangerous conversations on these platforms. Use temporary chat modes when you need to ask something sensitive. Stop pasting documents, medical information, financial details, and legal questions into AI systems that will store your words, train on them, and hand them over when asked.

And then do something bigger. Demand a federal privacy law. Tell your representatives in Congress that Americans deserve the same protections Europeans already have. Tell them that the two tier system, where corporations buy privacy and ordinary citizens get none, is unacceptable in a democracy. Tell them about Sewell Setzer. Tell them about the 1.2 million people a week who discuss suicide with ChatGPT. Tell them that you know what happens to the things you tell the machine, and you are done accepting it.

Your privacy is not a feature request. Your privacy is a right. Start acting like it.