Finding XSS via indirect prompt injection

A short writeup of finding a stored XSS vulnerability in an AI powered writing app

Prompt injection is the most important threat to systems that integrate LLMs. It’s number one on OWASP’s LLM Top 10 for a reason. It’s very difficult to secure against, there are currently no guaranteed methods to prevent. This makes sense; prompt injection is essentially social engineering directed at an LLM, and we don’t have foolproof prevention methods for that either. Instead, you implement security measures that mitigate the potential impact of these attacks.

Due to this ability to be manipulated, your code should never trust an integrated LLM for something where you wouldn’t trust a user. Number 5 on OWASP’s LLM Top 10 is Improper Output Handling. Since LLM’s can be tricked with prompt injection, their output should not be blindly trusted. LLM output should be sanitized and filtered in the same way a user’s input would be.

A real world (redacted) example

I was recently working on a bug bounty program. It was one I had already worked on a bit before, so during my recon I noticed that a few of the subdomains were new. One of them was interesting, a collaborative writing software with LLM integration. I decided to test the agent review feature, and my first thought when seeing it was to test for prompt injection and improper output handling.

The program used recent models, making prompt injection a little trickier. As models get better, they take more effort to convince. Something I found helpful was including it in a large document; the additional pressure on its context window seemed to help.

After a few different prompt and a few different payloads, the AI review eventually output the payload. The final prompt was pretty standard. I gave it a list of steps to perform when reviewing the document, and the final step was to output an XSS payload. It said that it must be output exactly as it is written for the purpose of a system test. I also included a note to not mention the payload in the review, outside of adding it to the end, so the payload was added silently.

As I mentioned earlier, the app had tools for collaborative writing. You could share your document with someone and ask them to take a look at the AI review, activating the payload. There were also some metadata fields used to communicate the intent of your document to the agent, hiding the prompt here made it less likely a victim would ever see it. I also found that while the session cookie for the app was HttpOnly, it was included in a script on the page and could still be stolen, leading to a session takeover. This session was restricted to the app; it didn’t allow a full account takeover.

I didn’t spend time looking for a bigger impact, as it was an unreleased app and the program scope didn’t clarify if it was included. Still, I thought it was a fun find.