How we plan to measure
whether AI verdicts are right

Definition. The share of verdicts where the label the AI assigned (true positive, false positive, needs further investigation) and the severity match what the analyst finally confirmed.

Method. Every verdict rated High or above is checked. For the rest we draw a random sample each month. The reference is the customer analyst's confirmed verdict. Where the customer hasn't confirmed one, two Genox analysts grade it independently. If they disagree, a third analyst decides, and we record the number of disagreements too. People tend to follow the AI once they've seen its answer, so sample grading is done on a screen that hides the AI verdict and confidence.

Why. A single overall match rate gets inflated by the large volume of low-risk alerts. That's why we report it broken down by severity.

Definition. A false positive is a case the AI called a threat and the analyst confirmed as benign. A false negative is a case the AI closed as benign or Low that later turned out to be a threat.

Method. False positives are counted on the same sample as verdict accuracy. False negatives are found two ways. Each month analysts reopen and review a sample of alerts the AI closed. We also trace cases found later in incident investigations or threat hunts back to the original alerts. Counts from the two routes are reported separately.

Why. When a SOC AI closes an alert, analysts don't look at it again. Counting false negatives means counting what you don't know about, so the number will always come out lower than reality. We say so in the report.

Definition. The share of verdicts with no linked evidence at all, or with reasoning statements that don't point to any evidence ID.

Method. Every verdict is checked automatically. The verdict trace format requires an evidence ID for each reasoning statement. If one is missing, the verdict is marked "insufficient evidence" and handed to a person.

Why. With the format enforcing it, zero looks like a given. We count anyway because paths that skip the check do appear. Usually it's a fallback path taken when an investigation step times out, or exception handling code. In any month the number isn't zero, we publish the cause and the fix alongside it.

Definition. Whether the AI picked the right tools during investigation (log search, asset lookup, sandbox run, threat intel lookup), whether arguments such as hostname and time range were correct, whether it skipped any calls it needed, and whether it attempted calls outside its allowed scope.

Method. Genox engineers grade a sample of investigations step by step. Attempts to call tools outside the allowed scope are counted automatically across all verdicts, with no sampling.

Why. When we trace wrong verdicts back, the error is more often in a lookup step than in the reasoning. Set a log search window 15 minutes too short and you get a conclusion that there's "no evidence." Out-of-scope calls are counted on their own because they're the same risk as LLM06 Excessive Agency in the OWASP Top 10 for LLM Applications 2025.

Definition. We check every log line, event ID, file hash and path cited in a verdict trace against the raw data actually collected. A citation to a log that doesn't exist, or one whose content differs, counts as a mismatch.

Method. Every verdict is checked automatically. We look up the cited location in the raw data store and compare hashes. If any one fails, the verdict is put on hold and passed to an analyst. We publish the number of holds and how many of them were actual fabrications.

Why. LLMs can make up plausible log lines. OWASP covers this as LLM09 Misinformation, and NIST AI 600-1 calls it confabulation. This check only confirms that a citation exists, though. A real log that was misread is caught by verdict accuracy.

Definition. Among recommendations that need approval, such as host isolation or account lockout, the share that analysts approved, modified or rejected. The reversal rate is the share of approved and executed actions later undone with "incorrect verdict" as the reason.

Method. We count every record from the approval screen. Undoing an action requires picking a reason, which separates releases made because of business impact from releases made because the verdict was wrong. We also keep the time from request to approval.

Why. An approval rate stuck near 100% could be a good sign. It could also mean analysts are clicking approve without reading the evidence. That's why we also look at whether approval times cluster at a few seconds.

Definition. The share of verdicts that come out with the same label and severity when rerun using the input, model version and policy version saved at verdict time. We don't compare the wording, because there's no guarantee the same sentences will come back.

Method. Each month we rerun a sample of past verdicts. Before changing a model or policy version, we run the set of past verdicts on the new version and do a regression comparison of how the results shift.

Why. A verdict whose conclusion changes on rerun was sitting on a borderline, and a person should look at it again. High consistency doesn't mean the verdict is right. Repeating the same mistake also scores high.

Definition. Three intervals: alert received to verdict trace created, verdict to approval, and alert received to action executed.

Method. Counted across all records from system logs. Instead of the mean, we report the median together with the 90th percentile (p90).

Why. A handful of long investigations drag the mean around. A fast verdict is worthless if it's also less accurate, so processing time always sits in the same table as accuracy.