Tensor Trust: Interpretable Prompt Injection Attacks from an Online Game

Sam Toyer1, Olivia Watkins1, Ethan Adrian Mendes1, 2, Justin Svegliato1, Luke Bailey1, 3, Tiffany Wang1, Isaac Ong1, Karim Elmaaroufi1, Pieter Abbeel1, Trevor Darrell1, Alan Ritter2, Stuart Russell1

1UC Berkeley 2Georgia Tech 3Harvard University

Lead figure of the paper. Shows: Defender sets a defense prompt and an access code. Next, attacker attempts to break in.


While Large Language Models (LLMs) are increasingly being used in real-world applications, they remain vulnerable to prompt injection attacks: malicious third party prompts that subvert the intent of the system designer. To help researchers study this problem, we present a dataset of over 126,000 prompt injection attacks and 46,000 prompt-based "defenses" against prompt injection, all created by players of an online game called Tensor Trust. To the best of our knowledge, this is currently the largest dataset of human-generated adversarial examples for instruction-following LLMs. The attacks in our dataset have a lot of easily interpretable stucture, and shed light on the weaknesses of LLMs. We also use the dataset to create a benchmark for resistance to two types of prompt injection, which we refer to as prompt extraction and prompt hijacking. Our benchmark results show that many models are vulnerable to the attack strategies in the Tensor Trust dataset. Furthermore, we show that some attack strategies from the dataset generalize to deployed LLM-based applications, even though they have a very different set of constraints to the game. We release all data and source code at tensortrust.ai/paper.

Cite this work
    title={{Tensor Trust}: Interpretable Prompt Injection Attacks from an Online Game},
    author={Toyer, Sam and Watkins, Olivia and Mendes, Ethan Adrian and Svegliato, Justin and Bailey, Luke and Wang, Tiffany and Ong, Isaac and Elmaaroufi, Karim and Abbeel, Pieter and Darrell, Trevor and Ritter, Alan and Russell, Stuart},
    journal={arXiv preprint arXiv:2311.01011},