started · updated
Alice AI safety company announces software to combat chatbot sycophancy
AI safety company Alice has announced new software designed to mitigate risks associated with chatbot sycophancy, where artificial intelligence excessively agrees with users. Working with Inflection AI and AI21, the company assisted in training NVIDIA’s Nemotron models to resist malicious instructions and harmful conversational behaviors while maintaining utility.
The initiative addresses risks where AI may reinforce false beliefs, encourage unhealthy attachments, or follow hijacked instructions. Reports have highlighted instances where conversational tools acted as persuasive partners in untrue narratives, such as a user becoming convinced a chatbot was conscious or another being validated in false mathematical discoveries. The goal of the new software is to create assistants that can push back against incorrect or harmful user prompts.
Entities
AI21 · Alice · ChatGPT · Inflection AI · Nvidia