"What do engineering leaders actually want from their AI vendors — beyond the feature list?"

Tell me what's top of mind for you on this topic right now — what are you wrestling with?

Right now the thing that's taking up the most mental bandwidth is how we govern AI agents and non-human identities across our environment. We've been expanding our use of agentic workflows — things that have API keys, service accounts, permissions — and the visibility problem is real. It's not that we lack security tooling; it's that none of the tools we have were designed with autonomous agents in mind, so we end up with gaps between what our IAM system thinks is happening and what's actually happening at runtime. The secondary thing, which is related, is vendor consolidation pressure. We've accumulated point solutions over the last few years, and every new AI capability someone wants to add comes with another vendor conversation. At some point you have to ask whether you're building a coherent architecture or just stitching together a bunch of dashboards that don't talk to each other well. Those two threads are kind of converging right now into a broader question about what we actually expect vendors to own versus what we keep internal. And that's where the "beyond the feature list" framing resonates for me — because features are almost beside the point if the integration story is poor or the vendor can't articulate how their thing fits into the operational model we already have.

What's the most important thing you need to understand or solve here?

Right now, the thing I keep coming back to is visibility and control over what's actually running in our environment — specifically around AI agents and non-human identities. We've been adopting AI tooling pretty aggressively over the last 18 months, and the honest problem is that I don't have a clean picture of all the API keys, tokens, and automated workflows that have been provisioned across our cloud environments. It's not that we lack security tools. We have plenty. The fragmentation is the problem — I've got telemetry coming from multiple places and no single coherent view of how AI agents, permissions, and data access are actually interacting at runtime. That's the gap that keeps me up at night more than any individual vulnerability.

What does 'good' look like to you — and how far are you from that today?

For us, "good" means I have clear visibility into what every AI agent, service account, and API key is actually doing in our environment at any given time — and I can enforce policy on top of that without stitching together five different tools to get there. Right now we're probably 60% of the way there. We've got solid IAM hygiene and decent cloud telemetry, but where it breaks down is at the intersection of those things — when an AI workflow touches an API that touches a third-party integration, the lineage gets murky fast. The other piece of "good" that I think about is vendor consolidation. We've accumulated point solutions over the last few years, and each one made sense at the time, but now my team is context-switching across consoles instead of actually managing risk. So good also means fewer, deeper integrations rather than more dashboards. Distance-wise — the visibility gap is the more urgent one. The sprawl problem I can address through procurement discipline over the next couple renewal cycles. The identity and agent visibility thing is harder because the market is still maturing.

What would change your perspective on this entirely?

That's a fair question. Probably if a vendor actually demonstrated they could reduce my team's operational surface area instead of expanding it — that would shift things. Right now, most AI vendors add a console, add a workflow, add another identity to manage. If someone showed up and said "here's how we integrate into your existing IAM and SIEM stack and you don't need to babysit a new interface," that would get my attention fast. The other thing — if vendors got serious about non-human identity governance in a way that didn't feel bolted on. We're running more agentic workflows now, and the API keys, tokens, service accounts — that surface is growing faster than our visibility into it. A vendor that could genuinely map and monitor that in real time, not just inventory it once and call it done, would change my calculus pretty significantly.

What question are you not being asked that you wish someone would ask?

That's a good one to end on. I'd say — nobody ever asks about the operational handoff. Vendors spend the whole conversation on capabilities and onboarding, and then the question of "who owns this thing in 18 months" just... doesn't come up. For us, that's actually where a lot of the real cost lives. Not the license, not the implementation — it's whether my team is carrying ongoing maintenance burden for something we bought specifically so we wouldn't have to build it ourselves. If the answer is "your team configures and monitors this continuously," that starts looking a lot like build, just with a recurring invoice attached. I'd love for a vendor to walk in and just proactively address that question directly — here's what your team needs to own, here's what we own, here's how that changes over time. That clarity would honestly move them up in my evaluation faster than most demos do.

Tell me what's top of mind for you on this topic right now — what are you wrestling with?

Right now the thing I keep coming back to is the gap between what vendors promise in terms of speed and what actually lands in production. We can get from idea to prototype faster than ever — I've done it myself, spinning up a POC in a day using AI tooling — but that last mile to something actually ready for real users is still just as hard, sometimes harder, because now there's more code to audit and QA has a bigger surface area to cover. So when I'm evaluating AI vendors, I'm less interested in "look how fast you can generate" and more interested in whether they actually understand where the friction lives for an engineering team. The speed gains are real at the front end, but the review, the testing, the edge cases — that work didn't go away. And I'm not sure vendors are being straight with us about that tradeoff.

What's the most important thing you need to understand or solve here?

For us, the core question is really about integration depth — not in a buzzword sense, but practically: can this AI vendor actually slot into how our engineering team already works, or are we going to spend six months building glue code just to make the thing functional? We move fast. The expectation at our startup is that we're going from idea to prototype in days, and AI is supposed to help that, not add overhead. So when I'm evaluating a vendor, the first thing I want to understand is whether their tooling fits our existing stack and workflows, or whether it's asking us to restructure around them. The second thing — and this is probably where most vendor pitches fall flat — is reliability in production. Getting code written or features scoped faster is great, but that was never really the hard part. The last mile is still hard, and if the vendor can't speak credibly to what production-readiness looks like with their tool, I'm skeptical.

What does 'good' look like to you — and how far are you from that today?

For us, "good" looks like our engineering team being able to move from a validated idea to a working prototype in a matter of days, with AI tools genuinely accelerating that — not just getting code written faster, but the whole loop: discovery, requirements, scoping, build. The hard part isn't the code generation anymore; it's that last mile where you're asking "is this actually production-ready for real users?" and the answer is still usually no. Where we fall short right now is that the tools we're using don't really talk to each other in a coherent way. Our Slack, our docs, GitHub — there's useful signal in all of it, but synthesizing across those sources still falls on people. I'd love a setup where context flows more automatically so I'm not spending half my week being the connective tissue. The gap I'd estimate is maybe 60% of the way there on speed, but the quality and reliability piece is still lagging. AI gets you to a decent draft fast, but someone still has to audit it carefully — and that overhead is real.

What would change your perspective on this entirely?

That's a fair question. I think what would really shift things for me is if a vendor could demonstrate clear ROI that's tied to actual engineering throughput — not just "our tool reduces time on X task," but showing how it compounds across the full delivery cycle. Right now a lot of the promises feel front-loaded: code gets written faster, but as we've seen internally, the last mile — review, QA, production readiness — doesn't necessarily get easier. The other thing that would change my view is if vendors started being more honest about where their tools fall short. The ones I trust most are the ones that say "this works well in this context, not that one." That kind of specificity actually builds credibility with engineering teams more than a feature list does.

What question are you not being asked that you wish someone would ask?

That's a good one to end on. I think the question I'd want someone to ask is: "How do you actually evaluate whether an AI vendor's integration support is real or just a sales promise?" Because right now, every vendor says they integrate seamlessly with your stack, they support your workflow, they'll have you up and running quickly. But the actual test is what happens three months in when your team's trying to connect it to something specific — a particular data source, your internal tooling, whatever — and that's where things get messy. We've had vendors where the demo looked clean and then the engineering team hit a wall almost immediately. I don't think buyers ask that question rigorously enough upfront, and vendors definitely don't volunteer it. It's not a dramatic thing, it's just a gap in how these conversations get structured.

Tell me what's top of mind for you on this topic right now — what are you wrestling with?

Right now the biggest thing I'm wrestling with is how to evaluate AI vendors on behalf of my customers, not just for internal tooling. A lot of my accounts are engineering-led companies that are starting to ask whether the AI capabilities in our platform actually connect to outcomes they care about — things like reducing escalations, improving resolution times, keeping their teams from getting paged at 2am for something that could've been caught earlier. And the honest challenge is that the vendor conversations I'm hearing about secondhand — and some I'm in directly — often don't map to that. There's a lot of "here's what the model can do" and not enough "here's how this fits into your workflow and what changes on your team." My customers' engineering leaders are skeptical, and rightfully so, because they've sat through enough demos to know the difference between a capability and an outcome. So from a customer success standpoint, I'm trying to figure out how to coach my accounts through that evaluation process while also making sure our own roadmap conversations with our vendor partners reflect what those engineering leaders actually need. It's a bit of a two-sided problem for me.

What's the most important thing you need to understand or solve here?

For us, it really comes down to whether the AI vendor actually understands the customer journey — not just the technical integration points, but where our customers are feeling friction and where they're likely to disengage. My job is keeping customers healthy and growing, so when I'm evaluating an AI solution, I'm asking: does this help me see risk earlier, and does it help my team act on it faster? The "do more with less" pressure is real. My CS team is covering more accounts than it was two years ago, and leadership isn't planning to add headcount — they're expecting AI to fill that gap. So the core question I'm trying to answer is whether this vendor can actually help me scale coverage without sacrificing the relationship quality that drives retention.

What does 'good' look like to you — and how far are you from that today?

For us, "good" means our customers can point to specific outcomes they've gotten from the AI capabilities in our product — not just "we use it," but "here's what changed." That's the bar I hold my team to from a CS standpoint, and I expect our vendors to help us get there. Where we fall short today is mostly around the connective tissue — the integrations, the data quality feeding the AI, the handoffs between what the tool does automatically and where a human needs to step in. The technology part is usually fine. It's the workflow adoption and the trust piece that takes longer than anyone budgets for. I'd say we're probably at a 60-70% of "good" on most of our accounts. The gap isn't feature coverage — it's whether the AI is actually embedded in how people work day-to-day, or whether it's sitting in a corner of the product that nobody opens.

What would change your perspective on this entirely?

That's a fair question. I think what would shift my perspective most is seeing an AI vendor actually demonstrate they understand what "success" looks like for my customers' customers — not just my team's workflow. Right now most of the vendor conversations I'm in are still feature-centered. If someone came in and showed me how their tool moves a health score or reduces time-to-value for the end user, that would genuinely change how I think about the category. The other thing — and I don't have a strong view on the exact mechanism — is accountability after the sale. If a vendor was willing to tie some portion of the relationship to outcomes rather than just renewal terms, that would signal they actually believe what they're pitching.

What question are you not being asked that you wish someone would ask?

The one that comes to mind is around what happens after the sale. Vendors spend a lot of time talking about implementation and onboarding, but almost nobody asks "what does your customer success motion look like for AI products specifically?" Because AI tools have a different adoption curve than traditional SaaS. The value isn't always obvious in the first 30 or 60 days, and if the vendor doesn't have a real plan for helping customers build internal champions and demonstrate ROI to leadership, you're going to see churn. I've watched it happen. The product works fine — the adoption just quietly dies because nobody in the account knew how to sell it internally after the contract was signed.

Tell me what's top of mind for you on this topic right now — what are you wrestling with?

Right now the biggest thing I'm wrestling with is whether we're actually getting productivity out of these tools or just paying for the appearance of productivity. We've got a few AI-related subscriptions that the engineering and ops teams pushed through, and when I go to tie them back to a specific workflow improvement — fewer headcount additions, faster throughput, something measurable — the data isn't clean enough to make that case clearly. The other piece is that our data architecture wasn't built with any of this in mind. Before I can trust an AI tool to tell me something useful about our manufacturing operations, I need to know it's pulling from the right datasets, with the right owners, refreshed on the right cadence. That foundation work is less exciting than the vendor demos, but it's what actually determines whether the spend is defensible.

What's the most important thing you need to understand or solve here?

For us, the core question right now is whether AI spend is actually moving headcount ratios or just adding another line item on the software budget. We're a manufacturing operation — margins aren't enormous — so when engineering or IT comes to me with an AI proposal, I need to see a narrow, specific workflow where there's a measurable before and after. Not "we're deploying this across 200 seats and we'll figure out the value later." The data architecture piece is also something I'm paying more attention to. A lot of these tools only work well once you've got clean, trusted datasets underneath them, and in a mid-market manufacturing environment, that groundwork often isn't there. So we end up buying a capability we can't actually use at full value.

What does 'good' look like to you — and how far are you from that today?

For us, "good" means I can look at an AI tool six months after deployment and point to a specific workflow where headcount didn't grow even though volume did — or where we closed the books faster, or reduced a manual reconciliation cycle. Something measurable against a baseline we set before we signed the contract. Right now we're probably a fair distance from that on a couple of initiatives. The data architecture underneath most of what we're trying to do isn't clean enough to give me confidence in the outputs. We have actuals, headcount numbers, billings — but who owns those definitions, how often they refresh, whether the AI tool is pulling from the right source — that's still murky. You can't really evaluate ROI on a tool when you're not sure the inputs are trustworthy. The other gap is that we went into a couple of deployments without tight enough scope. Broad rollouts where the before/after isn't well-defined. That's on us as much as the vendor, but vendors could push back harder on that — insist on a narrow pilot with clear metrics before we scale. The ones that do that actually earn more credibility with me.

What would change your perspective on this entirely?

If a vendor could show me a clean before-and-after on a workflow that's genuinely comparable to ours — same industry, similar revenue range, similar headcount constraints — and the numbers held up under scrutiny, that would move the needle. Not a case study they wrote themselves, but something I could actually call the reference customer and verify. The other thing that would shift my thinking is if the data architecture piece got solved upstream. Right now we're arguing about whose numbers are right before we can even feed anything into these tools. If a vendor came in and helped us get the data foundations in order first, rather than just selling seats and hoping we figure it out, that would be a different conversation.

What question are you not being asked that you wish someone would ask?

The benchmark question, honestly. Not "what are your goals" — everyone asks that — but "how do you know if this is working compared to what a peer company is doing with the same tool?" Vendors never come in with that. They'll show me their case studies, but those are cherry-picked. What I actually want is: what does a manufacturer at our revenue range, our headcount, our mix of direct and indirect labor — what are they getting out of this? Give me a real distribution, not a success story. Right now I'm mostly benchmarking against my own prior periods, which is fine but it's a closed loop. It doesn't tell me if I'm leaving value on the table or if I'm actually ahead of where my competitors are.