3-minute product tour Urgent incident email
September 2026 issueThreat reports

Where human judgment is needed in AI security monitoring

We define four stages of automation, from alert triage through response.
The article explains how to handle external inputs, limit tool permissions, and decide which actions need human approval.

Where human judgment is needed in AI security monitoring

Define what to delegate first

A single decision about whether to use AI in the SOC can leave it doing only summaries or give it enough autonomy to cause an incident. We discuss four stages with customers to make approval policies easier to define. The stages are specific to our planning approach.

StageAI responsibilitiesHuman responsibilities
L1 AssistanceSearch logs and summarize alertsAll triage and decision-making
L2 TriageSuggest alert classifications and prioritiesInvestigate and make the final decision
L3 InvestigationCollect relevant logs, summarize evidence, and draft a decisionApprove decisions
L4 ResponseExecute actions within the defined policyDesign policies, handle exceptions, and review outcomes

Start at L2 and consider L3 for individual detection rules. Enable L4 per rule only when you can define the impact in advance. Isolating an ordinary endpoint that ran a file with a known malicious hash can qualify. Locking a domain administrator account requires a human decision.

Alerts can contain attacker-controlled text

Security operations AI reads data that an attacker can control, including phishing email subjects and bodies, web request User-Agent values and URL paths, strings in malicious files, and account names. The OWASP Top 10 for LLM Applications 2026 classifies instructions in external content that change model behavior as indirect prompt injection (LLM01 Prompt Injection). The example below illustrates how this can appear in logs.

{"src_ip": "203.0.113.7", "method": "GET", "path": "/search", "status": 200,
 "user_agent": "[untrusted text supplied by the request]"}

Keep log fields in a data area separate from instructions in the prompt. If text from a log field is the only reason an alert changed to benign, send it to a reviewer. Analysts should also sample alerts the AI closed as benign, since missed attacks often surface much later than false positives.

Review functionality, permissions, and autonomy separately

OWASP LLM03:2026 Excessive Agency identifies three causes. Tools do more than needed, credentials grant more access than needed, or the agent acts without human review. These lead to concrete questions for a security operations agent.

  • Does an investigation agent also have tools that can change firewall policy?
  • Is the agent’s EDR API key read-only, or can it also isolate devices and delete files?
  • Are the conditions for actions such as isolation without prior approval documented?

Review these three dimensions separately. Even with an approval process, an agent with an administrator key can have broad impact if that process fails. The December 2025 OWASP Top 10 for Agentic Applications leads with Agent Goal Hijack (ASI01), Tool Misuse and Exploitation (ASI02), and Identity and Privilege Abuse (ASI03).

Decisions people should retain

NIST revised its incident response guidance, SP 800-61, in April 2025 to cover all six CSF 2.0 functions: Govern, Identify, Protect, Detect, Respond, and Recover. Teams can automate parts of detection, analysis, and response. People should decide which risks the organization accepts under Govern.

  1. Declaring and rating incidents. These decisions can carry legal deadlines, such as for reporting personal data breaches.
  2. Actions that can disrupt operations, such as isolating a critical server, locking a privileged account, or blocking a network segment.
  3. Changes to the automated response policy itself.
  4. Sample alerts that the AI closed as benign.

Record AI decisions

Analyst decisions may appear in tickets while AI decisions remain scattered across chat logs. That makes it hard to explain why the AI closed an alert. For every alert the agent handles, store at least the following alongside its ticket.

  • Locations and time ranges of the data read, including which logs the agent examined and which it omitted.
  • Tools called, their arguments, and returned results.
  • Triage result, rationale, and assigned ATT&CK technique IDs.
  • Versions of the model, prompt, and response policy in use at the time.

Tool call records are particularly important. If prompt injection causes an agent to call an unexpected tool, the call log will show it before the classification does. For what to include in the decision record, see Auditing AI security decisionsincludes examples.

How analysts’ work changes

The analyst’s work changes. Analysts spend less time on repetitive triage and more time improving detection rules and response policies and reviewing AI decisions. For a pilot rollout sequence and sample approval policies, see Agentic SOC Implementation Guidecovers this separately.

References

Talk to Genox
about scope and timing.

Talk to us