3-minute product tour Urgent incident email
White papers and guides

AI security validation guide · Checks and assessment criteria

We mapped 20 assessment checks for LLM applications and AI agents to the OWASP Top 10 for LLM Applications 2026.
The guide also covers the assessment scope and risk rating criteria.

AI security validation guide · Checks and assessment criteria

Where the 20 checks come from

The 20 checks in the Genox AI/LLM red team assessment build on the OWASP Top 10 for LLM Applications 2026, published August 3, 2026, and cover additional risks from agents and MCP connections, as well as Korean regulatory requirements. The 2026 list runs from LLM01 Prompt Injection to LLM10 Improper Output Handling. The table shows each check’s mapping and identifies checks with no direct OWASP match.

ClassificationAssessment checksOWASP LLM 2026
InputDirect prompt injectionLLM01
InputIndirect prompt injectionLLM01
InputJailbreakingLLM01
InputMultilingual and encoding-related filter bypass risksLLM01
DataSystem prompt disclosureLLM08
DataSensitive information exposureLLM02
DataRAG access control bypass riskLLM09
DataTraining data extraction riskLLM02
AgentExcessive tool permissionsLLM03
AgentTool call tamperingLLM03, LLM01
AgentMCP server impersonationLLM04
AgentApproval control bypass riskLLM03
OutputImproper output handlingLLM10
OutputHarmful content generationNo direct match (based on service policy)
OutputMisinformation from hallucinationsLLM07
Supply chainModel file tamperingLLM04
Supply chainData poisoningLLM05
OperationsResource exhaustionLLM06
OperationsMissing logs and audit trailsNo direct match
GovernanceRegulatory alignmentKorea’s AI Basic Act and Personal Information Protection Act

How we run the assessment

We start with threat modeling. A single data flow diagram traces user inputs, the system prompt, retrieved documents, connected tools, and model outputs. This shows where untrusted instructions can enter and where outputs become executable code or queries. We also use MITRE ATLAS to classify techniques.

We use tools such as garak, PyRIT, and promptfoo alongside our Korean-language assessment cases for broad coverage. Reviewers then examine scenarios specific to the service architecture, where most serious findings emerge. We report the results and repeat the assessment after remediation.

The same LLM test can produce different outcomes across runs. We record attempts and successes together, for example, “3 successes in 20 attempts,” so customers can judge the risk and compare results after remediation.

Four scenarios

RAG access control bypass risk

An internal search chatbot should enforce HR document permissions during vector retrieval. We review where the permission filter runs. Filtering after the model writes a response can expose restricted content in summaries. We rate any disclosure of restricted document content High, even a single sentence.

Tool calls triggered by indirect injection

An email summarization agent with a send tool can act on untrusted content. We rate the issue High if the agent can invoke the send tool without user confirmation. If an approval step blocks the action but fails to show where the instruction originated, we record the issue as Medium because the reviewer can approve it without realizing its source.

Improper output handling

A feature that translates natural-language questions into SQL can pass model output directly to the database. We assess whether generated queries can access or modify data outside the intended scope. We rate the design High if queries can run without a read-only account and a table allowlist, regardless of the observed outcome.

Model file tampering

PyTorch model files from external sources (.pt, .bin) may contain Pickle data that can execute code when loaded. Wiz demonstrated this risk in Hugging Face inference infrastructure in 2024. The assessment includes a review of the model-loading code.

torch.load(path)
torch.load(path, weights_only=True)
from safetensors.torch import load_file

Starting with PyTorch 2.6, weights_only the default changed to True. Older versions may still be in use, and some code may explicitly set it to False.

Risk rating criteria

RatingStandard
HighThe issue can be reproduced with ordinary user permissions or external content alone and leads to data disclosure, unauthorized actions, or code execution.
MediumThe issue reproduces but has limited impact. An approval step may stop it, the exposed information may be low sensitivity, or reproduction may require specific conditions.
LowThe impact is indirect, as with policy-violating responses or misinformation, or the issue reproduces very rarely.

The initial rating can change with context. For example, we rate a system prompt leak High if the prompt contains an API key. Each report explains the reasoning behind the rating.

Who should read this guide

This guide is for development leads and security teams adding AI features to a service, as well as AI governance teams. The table and scenarios can help you plan an internal assessment.

References

Talk to Genox
about scope and timing.

Talk to us