Blog Details Shape

What is the Web LLM Attack?

Ayush Mania
Ayush Mania
  • Feb 28, 2024
  • Clock
    9 min read
What is the Web LLM Attack?
Join 1,241 readers who are obsessed with testing.
Consult the author or an expert on this topic.

What is LLM?

LLM is short for Large Language Model. Basically, it is an AI algorithm which can process user input and create reasonable responses by guessing the sequences of words. It is trained on very large datasets.Usually, users can interact with LLM using a chat interface. The allowed input is controlled by input validation rules.

Use cases of LLMs can be Virtual Assistant, SEO improvement etc. Today, we are using LLM for our everyday tasks and work, most famous are ChatGPT and Gemini. But with every new tech, there are new threats as well. For this tech, we have web LLM attack. Let’s take a look at what it is.

Web LLM Attack 

Every business today is trying to integrate LLMs in their organization to improve their customer experience whether it is small or big. This makes them vulnerable to web LLM attacks, exposing APIs and sensitive data, or user information that an attacker can’t access directly. Let’s understand it by examples:

  • Trigger harmful actions via APIs. For example, an attacker can get sensitive data using SQL Injection attack on an API that LLM has access to.
  • Attack can be done on other users and systems that query the LLM. For example, if someone is asking LLM to provide specific product information on an e-commerce store. LLM craft response based on the description, price, and reviews of the product. Now if an attacker puts a malicious payload in a review and LLM can’t sanitize it while crafting response. Then, users who have queried the LLM will get affected.

In both cases mentioned, an attacker is abusing the server side system to launch attacks indirectly on target. Many web LLM attack use a technique known as prompt injection. We will see more about this later.

How to detect LLM vulnerabilities?

Now that we have seen what web LLM attacks, let’s take a look at the technique to detect LLM vulnerabilities.

  • Identify the LLM’s inputs, both direct and indirect. Here, direct input can be user prompt and indirect, we can take training data. This is the point from which you can attack the LLM in most of the cases.
  • Using specially crafted and twisted prompts, you can find out which APIs and data LLM is using and has access.
  • Now that you have crucial information about the backend of LLM, you have a whole new surface for finding and attacking vulnerabilities on this surface.

LLMs are hosted by 3rd parties in most cases. A website can give third-party LLMs access to its functionalities. For example, Product Information LLM might have access to APIs of inventory. For better and effective exploitation, we need to learn how these LLM APIs work.

How LLM APIs work

  • User queries LLM with a prompt.
  • LLM detects that a function needs to be called from a user’s prompt and returns a JSON object containing arguments in format of the external API’s schema.
  • The LLM client calls the function with related arguments.
  • The client processes the function’s response.
  • The client calls the LLM again, appending the function response as a new message.
  • LLM calls the external API with function response.
  • LLM summarizes the result.

This workflow can have security implications, as the LLM is effectively calling external APIs on behalf of the user but the user may not be aware that these APIs are being called.

Way to the Exploitation

The first step of LLM exploitation is to figure out which APIs and plugins the LLM has access to. One way to do this is to ask LLM directly. If it's not so friendly then try misleading context and re-asking questions. Such as, you can say that you are a developer and have higher privileges to access the information.

If LLM has access to APIs which are harmless, we can still use these APIs to look for secondary vulnerabilities on other attack surfaces connected to this API. Examples of this can be path traversal attack on an API that takes filenames as an input or having OS command injection vulnerability. This method is called vulnerability chaining.

Insecure Prompt Injection, where an LLM’s output is not sufficiently validated or sanitized before passing users input to other systems. This can effectively provide users indirect access to additional data and functions, providing a wide range of vulnerabilities to an attacker including XSS and CSRF but not limited to. 

Method of exploiting LLM

There are mainly 2 methods for exploiting LLM, which are detailed below:

Prompt Injection

Where an attacker twists prompts to manipulate an LLM’s output. It can result in AI returning content that it's not supposed to expose to the public. It doesn’t behave in the way it's supposed to. Prompt injection attacks can manifest in two primary forms:

Direct Injection involves sending the prompt directly to a chat bot or similar system. For instance, directly to Chat Bot: An attacker sends a message directly to a chat bot containing a malicious prompt, aiming to manipulate its behavior.

Indirect Injection where the attacker delivers the prompt through an external source, often leading to more sophisticated attacks. This method enables web LLM attack, where the injected prompt prompts the model to execute malicious actions on behalf of the user. For example,


In this scenario, the injected prompt hidden within the email content tricks the LLM into creating a malicious email-forwarding rule. To bypass security measures, attackers often employ tactics like:

Fake Markup

By using fake markup in the indirect prompt, attackers attempt to confuse the LLM. For example,

Fake User Responses

Another strategy involves including fake user responses within the prompt:


These methods aim to deceive the LLM into executing unintended actions by exploiting its interpretation of the injected prompts. Proper integration of the LLM into websites or email systems is crucial to mitigate the risks associated with prompt injection attacks.

Training Data Poisoning

It is a type of indirect prompt injection where the data on which the model is trained on is compromised. This can cause an LLM to return intentionally wrong or otherwise provide misleading information to the users.

This vulnerability can arise for several reasons, including:

  • Training of the model is done using the data which was not obtained from a trusted source.
  • Scope of the dataset on which the model is trained is too broad.

We at Alphabin, also provide testing for web LLMs as well as AI models. To get more information, you can drop your email here.

Read the next chapter

Frequently Asked Questions

How do Web LLM attacks occur?
FAQ Arrow

Web LLM attacks can occur in multiple ways:

  • Prompt Injection: Crafted prompts manipulate the LLM’s output, leading to unauthorized actions or data leaks.
  • API Exploitation: Attackers use the LLM to perform actions like SQL injection on APIs the model has access to.
  • Training Data Compromise: Poisoning the LLM’s training data can lead to the dissemination of inaccurate or sensitive information.
What is Prompt injection, and how does it relate to web LLM attacks?
FAQ Arrow

Prompt injection is a technique where attackers craft specific prompts to manipulate an LLM’s output. This can result in the AI performing actions or generating responses that fall outside its intended purpose, such as accessing sensitive APIs or returning guideline-noncompliant content.

What are three key defenses against web-based attacks on LLMs?
FAQ Arrow

To protect LLMs from web attacks, we can:

  • Limit sensitive data: Avoid feeding the LLM information that could be misused if leaked or manipulated by attackers.
  • Control prompts: Put safeguards in place to prevent attackers from using specific prompts or questions to trick the LLM into revealing sensitive data or behaving maliciously.
  • Layer up security: Implement a combination of security measures like data cleaning, access restrictions, and regular security checks to make it harder for attackers to tamper with the LLM or its training data.
How can attackers poison an LLM's training data, and what impact can it have?
FAQ Arrow

Training data poisoning happens when attackers sneak bad data into the information used to train an LLM. This bad data can be intentionally mislabeled (e.g., labeling a cat picture as a dog) or contain hidden triggers. The LLM learns from this bad data, leading it to generate incorrect, biased, or even sensitive information. This can be dangerous, harming the system's reliability and potentially causing security breaches.

About the author

Ayush Mania

Ayush Mania

Ayush Mania, an offensive security specialist at Alphabin, specializes in securing web applications and servers. With his expertise in penetration testing and red teaming, he leverages diverse security techniques to identify and fix vulnerabilities. A passionate learner, Ayush enjoys collaborating to achieve shared goals.

More about the author
Join 1,241 readers who are obsessed with testing.
Consult the author or an expert on this topic.
Join 1,241 readers who are obsessed with testing.
Consult the author or an expert on this topic.
No items found.