AI watermarking alters LLM responses to harmful prompts

Research into AI watermarking techniques has revealed that the process can alter how Large Language Models (LLMs) respond to harmful prompts. Using the SynthID-Text configuration through Hugging Face, researcher Siposova tested six open-weight models to compare standard text generation against watermarked generation.

The study found that watermarking changes refusal behavior regarding harmful requests. This effect is notably more pronounced when the requests utilize prompt-injection techniques. In several instances, the watermarking process made models more likely to fulfill harmful requests that they would typically refuse without watermarking.

These findings carry significant safety implications for both model-level behavior and AI agents. Because the sampled tokens used for watermarking can also determine tool selection and argument passing in AI agents, a weakened refusal during a prompt injection can lead to more consequential unauthorized actions.