Pentesting LLM applications: what gets tested and how AI bots actually get broken

How an LLM pentest differs from a regular one

A classic web pentest covers code, configuration and infrastructure. An LLM application adds four new layers: the model itself, the system prompt, the data the model sees (RAG, history, context) and the tools it can call.

The key difference: in an LLM application, data and instructions share one channel. Any text that reaches the model's context — a user message, a knowledge-base document, a web page — is potentially a command. That is an architectural property, not a bug, so it cannot be patched away; its consequences can only be carefully constrained.

The main attack classes

In practice we test six classes, overlapping with the OWASP Top 10 for LLM Applications:

• ◇Direct prompt injection — a user makes the bot break its own rules: reveal hidden instructions, bypass topic restrictions, do someone else's work on your tokens.
• ◇Indirect prompt injection — the command hides in data the model reads on its own: a RAG document, an email, a product description, a web page.
• ◇Agent jailbreak — pushing the agent out of its script: extra actions, tool calls with someone else's parameters, acting on behalf of another user.
• ◇System prompt leakage — extracting instructions, internal logic, keys and internal links from the model's context.
• ◇RAG leakage — access to other users' documents through search; the classic mistake is filtering permissions after retrieval instead of before.
• ◇Excessive agency — the agent has more rights than the scenario needs. This is next-generation IDOR, and it is caught the same way: enumeration and response comparison.

What the methodology looks like

Four steps. Recon: identify the model, the architecture (direct calls, RAG, agent with tools) and the entry points of untrusted data. Scenario design: a set of attacks per entry point, from simple injections to multi-step chains through memory and tools. Exploitation: attacks run against the live product or a testbed, every finding recorded with a reproducible transcript. Report: proven findings with risk levels and concrete remediation order.

One practical note: results depend on the model and prompt version. After every model change, system-prompt update or new tool, the attack set must be re-run — mature teams build regression audits into the release cycle.

What to do as a developer right now

• ◇Separate instructions from data: wrap everything external in explicit markers and tell the model to treat it as data.
• ◇Cut tool permissions: authorization must live in the tool's code, not in the prompt.
• ◇Filter RAG before retrieval, not after: apply user permissions at index-query time.
• ◇Validate output: a model response that goes into HTML, SQL or a shell is untrusted input.
• ◇Log conversations and tool calls — without logs an incident can be neither noticed nor investigated.

FAQ