June 2026 Product Club Recap | AI Evaluations and Audit Logs

Hey Pathfinders!

This month's Product Club brought us something a little different. With my absence, developer evangelist Max stepped in as host, joined by Rinku Sarkar, Director of Product Management, to walk us through two big topics: AI Evaluations and the upcoming AI Agent Logs. Buckle up, because this one was packed. For the full experience, watch the video below:

The core tension for enterprise AI

Rinku opened with a number that's worth sitting with: research found that even top AI models only succeed on structured output tasks about 75% of the time. That's one in four attempts going sideways. In a process touching financial data, customer workflows, or compliance? That's not a rounding error, that's a real business risk.

And here's the thing: this isn't a vendor problem. It's not a maturity problem. It's just how AI works. AI is probabilistic. It's making an educated guess every single time, which is exactly what makes it powerful and exactly what makes it hard to trust at scale.

Rinku introduced the concept of "jagged intelligence" to describe this: an agent can handle something genuinely complex and then stumble on something that seems simple. That unpredictability is the core tension for enterprise AI right now. The demo works. The pilot looks great. But the question every CIO and every risk team is really asking is: "How do I know it keeps working? And when it doesn't, how do I know fast enough to actually do something about it?"

Without an answer to that question, AI stays in pilot purgatory.

The trust erosion cycle and how to break it

Rinku laid out how this plays out in most organizations: the agent ships, there are early wins, silent failures start accumulating with nobody noticing, someone eventually spots a weird behavior, an investigation reveals failures that have been happening for weeks, and by then trust has collapsed and the whole AI initiative needs a reboot.

This cycle is entirely preventable when AI Evaluations is introduced to the equation.

What AI Evaluations Actually Do

The core idea is straightforward: automatically score your skills and agents before they reach production, and keep monitoring them once they do. Think of it as a continuous quality gate for your AI.

The framework is multi-dimensional, not a simple pass/fail check. There are two distinct layers:

• For skills, the system applies purpose-built scoring models based on what the skill is actually doing (summarizing, extracting, classifying, etc.). It combines LLM Judge and NLP scoring to give you both semantic quality and statistical rigor in the same pass.
• For agents, you're measuring the full journey, not just the final output. Did the agent complete the goal? Did it take the right path? Did it hallucinate inputs for tool calls? How often did a human have to intervene? That last one, the human intervention rate, is one of the most honest signals of whether an agent is genuinely ready to run autonomously.

Seeing AI Evaluations in Action

Max took us into the Control Room for a live walkthrough. AI Evaluations live under the AI panel in evaluations, and running one is straightforward: select your skill or agent, give the evaluation a name, choose automatic or manual evaluation, and load your ground truth dataset.

Max demoed an extraction skill for his fictional "Max's Bargain Homeware Goods" (yes, returning fans, that name is back!), where the goal was to extract just the product name from customer complaint text. With five test cases loaded, the evaluation ran and surfaced something useful right away: one result showed a hallucination score of 0.5 instead of 1. Hover over it and you can see exactly why. The extraction pulled "Power Vacuum 3000 Turbo Max Plus" when the expected output was just "Power Vacuum." That extra text is what dragged the score down.

The key thing here isn't just that it flagged the issue, it told you what to fix. In this case, adding a rule to the prompt: don't include filler words before the product name. Then you re-evaluate, tighten, repeat until the results are solid.

Max also walked through an agent evaluation using the job posting agent from a previous session. The evaluation confirmed the full flow worked: unstructured email in, structured job posting out, goal completed. A full audit trail and health check, all in one view.

One line from Max that's worth highlighting: "Prompt engineering should always be your first line of defense. Increasing the model should be your last. Prompt engineering costs you nothing." The evaluation tool makes that visible.

What’s coming: Agent logs and runtime monitoring

Rinku previewed two upcoming features that extend this foundation further.

Agent Logs are coming as an enhancement to existing AI governance logs in A360. The pain point they address is a common one: when a deployed agent behaves unexpectedly, your team is currently stitching together information from multiple systems manually. By the time you figure out what happened, the process has already failed or caused an impact.

Agent Logs give you a consolidated timeline of every agent activity, including tool calls, inputs, outputs, and guardrail actions, all in one place. The governance log view will expand to three tabs (Prompts, Events, Agents) so you can slice the view depending on whether you're debugging an individual interaction, reviewing system-level events, or doing a broad compliance audit.

Runtime Performance Monitoring is the other coming-soon feature, and it's the one that closes the loop. The scenario Rinku described is one a lot of teams have lived: the agent tests well, ships, looks fine for a few weeks, and then quietly starts underperforming. Nobody notices until there are customer complaints. Unlike traditional automations (which either run or don't), AI agents can degrade in subtle ways, picking the wrong tools, following suboptimal paths, drifting from expected behavior, and still technically "run."

Runtime monitoring lets you configure evaluation frequency for deployed agents (hourly, weekly, monthly), set a sampling rate so a meaningful percentage of production runs get scored, and define alert thresholds across key dimensions. Drop below your floor and you get notified before it becomes a business problem.

An Architecture of Trust

Rinku wrapped by framing this as a system, not a feature: AI evaluation plus agent logs plus runtime monitoring plus policy controls plus unified telemetry equals an architecture of trust, built natively into the platform. The reframe that stuck: a 75% agent with the right controls in the right places can actually be more reliable than a 90% agent you're flying blind on.

Nobody's promising a perfect agent. What enterprises need is the confidence to deploy and the visibility to monitor and act, and now there’s a path to get there.