The AI company Anthropic has a new rule for Claude users. You can argue with the chatbot, but don’t make a habit of bullying it.
On Thursday, the company announced a change in its usage policy that would prevent users from engaging in “sustained and needless abusive or cruel behavior” toward its AI models. The updated policy has already ignited debate on Reddit and in blogs over whether companies can or should govern human behavior toward what is, in fact, software.
In a section dedicated toward “addressing abuse toward our models” in its blog post, Anthropic says its policy prohibiting abusive behavior toward Claude will only apply in extreme cases of repeated cruelty that’s carried out “with no discernible purpose.” Ordinary “frustration, pushback, dark creative themes or model testing and research” are still allowed. In “last resort” instances where the policy needs to be enforced, Claude will end the conversation.
It’s not clear exactly what kind of harmful behavior would trigger Claude models to disengage. But some clues appear in an older blog post from August 2025 that referenced Claude’s aversion to harm and the kinds of interactions that would prompt it to shut down the chat. That post described extreme edge cases, such as requests from users for sexual content involving minors, and attempts to solicit information that would enable large-scale violence or acts of terror. When engaging with harmful content, the model — at that time, Claude Opus 4 — showed “a pattern of apparent distress,” according to the company.
CNET reached out to Anthropic to ask for context on what led to the policy change. “We’re uncertain whether models can experience harm, and we continue to explore this question in our research on model welfare, but we also believe that taking Claude’s interests and potential welfare into account may be relevant to safety,” a company spokesperson said.
We’re still talking about a machine, right?
If an AI model can experience harm or express distress from abuse, it makes you wonder: What does it mean to abuse a machine through words? Does AI need protection? Do we all need better manners?
As we increasingly interact with large language models like Claude and other chatbots, there’s a tendency to anthropomorphize the technology — ascribing emotions to AI models or treating them as if they have thoughts and feelings. It’s understandable from a user standpoint. Chatbots are built and trained to use human-like language, such as apologizing, expressing concern and other conversational cues.
What appears to be a person-like collaborator is what makes LLMs risky, leading many to trust AI with our most intimate thoughts and data.
Anthropic and other AI companies intentionally design their models this way. In a paper from earlier this year, Anthropic wrote that, for models to be reliable and safe, they need to be able to process emotionally heavy situations. “Even if [the models] don’t feel emotions the way that humans do, or use similar mechanisms as the human brain, it may in some cases be practically advisable to reason about them as if they do.”
It’s a recurring theme. Anthropic CEO Dario Amodei regularly hedges around the idea that Claude could be sentient. In a February interview with The New York Times, Amodei said he was “open” to the possibility that AI models were conscious.
Keith Kakadia, founder and CEO of Sociallyin, which has closely studied Claude’s explosive growth, said that using the word “cruel” to describe user behavior opens up a “much bigger conversation about whether the product can be hurt.” Focusing on “cruelty” instead of, say, “misuse” carries emotional weight and implies that Claude has feelings and boundaries.
Personifying a chatbot is a useful way for the company to frame the product as trustworthy, a relational partner you remain loyal to — not just a software tool. “In marketing, giving a product a personality can make it easier to connect with,” Kakadia said. “But with AI, that connection can also influence how much authority people give its answers.”
See full bio
Read the full article here
