Prompt injection exploits role confusion in LLMs

Researchers Charles Ye, Jasmine Cui, and Dylan Hadfield-Menell found that LLMs cannot reliably distinguish system text from user input, enabling prompt injection. Role tags like <system> and <user> do not guarantee security. Destyling text reduced attack success from 61% to 10%, showing role confusion is a key vulnerability.