How to Grade Your AI SOC Analyst

tl;dr

If you’re evaluating AI for the SOC, you’ve probably noticed how hard it is to compare accuracy claims across products. That isn’t really anyone’s fault. There’s no shared test every system runs through, and alert investigation is genuinely hard to benchmark, because the right answer usually isn’t in the alert itself. It depends on the surrounding evidence and on what counts as normal in a given environment, and the data you’d need for a public test set is too sensitive for anyone to publish. So we built one for ourselves, with datasets per attack surface and expert analysts labeling both the final verdict and each step that led to it. We measure three things: is the verdict right, does it reach the same answer every time, and would a senior analyst endorse how the system got there. Those are the questions we hold ourselves to, and we think they’re the right ones to ask of any vendor, including us.

The scoreboard that doesn’t exist

When a frontier lab releases a new model, the announcement typically comes with a table: benchmarks down the rows, models across the columns, scores in the cells. It’s a useful convention in AI. Those benchmarks are imperfect (e.g., test questions leak into training data, vendors report their most favorable configurations, etc.), but they’re a shared yardstick. When a lab says its new model is stronger than its last one, or stronger than a competitor’s, you can look at the same tests, run them yourself if you want, and form your own view. The claim is checkable.

Now look at the AI SOC market. Dozens of vendors, all making confident claims about accuracy; some even say 99%. But no buyer can verify it, because there’s no shared test. There’s no fixed set of alerts and corresponding telemetry with known right answers that every vendor’s system runs through.

It’s actually worse than a missing scoreboard. The numbers that do get quoted aren’t even in the same units. Two vendors both claiming 95% may be measuring entirely different things and on wildly different mixes of alerts. Absent that context, you can’t tell whether to be impressed or underwhelmed.

And the decision riding on these claims isn’t a small one. Buyers are deciding who gets a live connection to their EDR, their identity provider, and their cloud logs, largely on the strength of a demo.

This post is about why that benchmark doesn’t exist, what we built for ourselves, and why we believe it matters.

Benchmarks and why an alert investigation benchmark is hard

A benchmark is a fixed set of tasks with known right answers, so that progress can be measured and claims can be compared. If you’ve read a model announcement in the last few years, you’ve seen them. Some recent tests include:

Terminal-Bench – evaluates how well AI agents can autonomously perform a variety of complex, multi-step technical tasks inside a command-line interface (CLI) environment.
DeepSWE – tests AI coding agents on complex, multi-file software engineering tasks.

Cybersecurity has benchmarks too. Popular examples include:

CyberSecEval – evaluates (1) safety/defensive risks associated with model misuse, and (2) offensive capabilities for performing attacks. Note, the suite includes CyberSOCEval which measures how well AI models can perform malware analysis and threat intelligence reasoning.
ExploitBench – measures AI agents’ ability to perform real-world software exploitation as a multi-stage process.
Cybench – tests a variety of capture the flag (CTF) scenarios.

These are good benchmarks, but look at what makes them scoreable. Each one has a checkable win condition. Did you capture the flag or not? Did you get the shell? Which file contains credentials? The answer is knowable in advance, independent of whose environment you run in, and it doesn’t change based on context.

CyberSOCEval is the closest to our collective problem in the AI SOC world. Malware analysis and threat-intel reasoning are real SOC skills, but it still evaluates knowledge applied to an artifact you’ve been handed. It doesn’t evaluate a decision made about an environment.

Alert investigation has different properties. An alert isn’t a self-contained artifact. The verdict (benign, suspicious, or inconclusive) usually isn’t recoverable from the alert itself. It has to be assembled from the surrounding evidence (the user’s history, the authentication trail) and from what’s normal for this environment. The same alert can genuinely be benign in one company and evidence of an attack in another.

This makes benchmarks for alert investigation challenging. There are real structural challenges:

The data is nobody’s to share. Real alerts contain customer identity: hostnames, usernames, internal IP space, file paths, business context. No one publishes their logs, and heavily anonymized alerts lose the very context that helps determine the right answer.
Ground truth is expensive and perishable. Because the answer isn’t in the alert, someone with real expertise has to reconstruct it from the surrounding evidence. And it’s not just a simple suspicious, benign or inconclusive label. The reasoning is key. It’s also perishable as environments drift and what’s normal changes.
The distribution is heterogeneous. A cloud alert, a phishing email, an EDR detection, an identity anomaly, and a hand-written SIEM rule share a queue, but can have very different solutions. A single blended score across all of them hides more than it reveals.

So we built one

No one can easily publish a dataset like this, but that doesn’t mean it can’t exist. It just means each vendor has to build their own and be judged on whether they did. What follows is the harness we run our own product against: the datasets, the labeling, and the things we measure.

We’ve constructed datasets that are scoped to specific attack surfaces: Phishing, Identity, Endpoint, Cloud, Network, and SIEM. Within each dataset, we use stratified sampling to cover the breadth of alerts that actually show up in production, rather than over-indexing on common alarms.

Then an expert security analyst annotates each case. The labeling is done by people with significant experience working the queue. That includes our CEO Seth who brings decades of experience at NSA and Mandiant. This is the time-consuming, unglamorous part, but it’s key. There are three ways we do this:

Case labels – The expert assigns the case a verdict: benign, suspicious or inconclusive.
Investigation Step (iStep) labels – The expert also labels each step within the investigation as benign, suspicious or inconclusive. This matters because our architecture is built around discrete iSteps, each answering one analyst-relevant question. Labeling at the iStep level helps us measure whether we’re arriving at the right conclusion for the right reasons. You can check out our previous blog for more information on our agentic architecture.
Reasoning feedback – Free-form expert commentary on the quality of the reasoning at various points in the investigation. We look for what’s sound, what’s missing, and what a senior analyst would object to.

To date we have labeled thousands of cases this way, and tens of thousands of individual investigation steps within them. That annotation work feeds the three things we measure.

Accuracy: “Is it right?”

We compare every verdict against the expert label, at two levels: the case as a whole, and each investigation step within it.

We don’t collapse that into a single accuracy percentage, because the ways of being wrong aren’t equivalent. A false negative means a real attack goes uninvestigated. A false positive means an analyst spends time on activity that was never a threat. Both are errors, but they have very different costs, so we track them separately rather than netting them out against each other. We do the same per attack surface, since a system can be strong on endpoint and weak on identity and still post a healthy blended number.

iStep-level scoring is the part most accuracy claims leave out. Final verdicts are the easy number to produce, but they hide a lot. A system can reach the right verdict for entirely the wrong reasons. It can misread the authentication trail, skip the evidence that actually mattered, and still land on “benign” because most alerts of that type are. At the case level, that’s a win. At the step level, it’s a lucky guess. And lucky guesses don’t generalize.

Consistency: “Does it reach the same answer every time?”

Run the same case ten times. Do you get the same answer?

We ask that three ways:

Case decision consistency. Does the final verdict hold across runs?
iStep decision consistency. Do the individual investigation steps reach the same conclusions each time?
iStep distributional consistency. Does the system even choose the same steps to run each time? Agentic systems can take different paths to the same goal, and some of that variation is harmless. What isn’t harmless is dropping a step that mattered: if one run checks whether the login came from a known device and the next one doesn’t, both can land on “benign,” but only one of them earned it.

Consistency is an important metric for any agentic system hoping to establish trust with end users. A recent paper on AI agent reliability (Rabanser et al.) evaluated 15 models and found that rising benchmark scores have not brought corresponding gains in reliability (consistency is one of their axes of agent reliability). This is one reason Embed’s investigations have a deterministic layer by design.

Reasoning alignment: “Would a senior analyst endorse how it got there?”

We score explanations against a rubric built by the same Embed experts. The reasoning feedback they provide during annotation is digested into a codified set of criteria: what a good explanation contains, and, just as usefully, the specific pain points that make an AI explanation worthless. Most of these came from real objections raised while reviewing real investigations, not from a whiteboard. Here are a few examples from our benchmark:

A benign security mechanism is mischaracterized as a “bypass” or a failure. For example, describing a control that worked as designed as though it had been defeated.
Absence of evidence is treated as positive evidence. For example, an enrichment query returns nothing and the investigation reports that as confirmation the activity was benign, when the honest read is that the check was inconclusive.
The alert’s own detection is used to corroborate itself. Citing the fact that the tool fired as though it were independent evidence, but it’s actually circular reasoning.

This isn’t cosmetic. Alignment is what lets a human verify a conclusion in seconds instead of re-investigating from scratch. It’s also why our investigations are structured as discrete steps that expose the evidence gathered, the questions asked, and the conclusion reached. An auditable reasoning chain isn’t only a feature for the analyst; it’s the precondition for being able to measure alignment at all.

The impact of good measurement

Once you have metrics like these, the payoff is not just a scoreboard. It influences how you build.

The obvious benefit is that we can measure improvements and catch regressions before they ship. When we change a function, edit a prompt, swap a model, or adjust how an investigation is structured, we can see whether accuracy, consistency, and reasoning alignment moved in the right direction, or whether we broke something that used to work.

It’s also a great diagnostic. Because we measure at the iStep level and not just the case level, our metrics don’t only tell us that something regressed, they tell us where. A drop in case accuracy that traces back to a specific investigation step is a very different problem than one spread evenly across the whole investigation. iStep distributional consistency tells us whether the system is drifting in which steps it chooses to run. Reasoning alignment tells us which explanations a senior analyst would reject and why. That granularity lets us pinpoint exactly where in the system to spend our attention.

This is also where our architecture and our metrics reinforce each other. We leaned into a system that combines determinism with non-deterministic AI from the start because we believed a completely stochastic system wouldn’t meet analysts’ bar for trust. The structure of an investigation, which questions get asked, what evidence gets gathered, how a conclusion traces back to its source, is repeatable by design. AI is applied where judgment is genuinely required.

Over time, the benchmark itself compounds. Every labeled case, every codified pain point, every regression we caught becomes part of a yardstick that’s hard to replicate and that keeps us honest as models and environments change.

We can’t publish the dataset, but the standard isn’t a secret. Accuracy, consistency, and reasoning alignment are what we hold ourselves to, and we think they’re the right questions for any buyer to ask of anyone, us included.