78% of IT roles now require AI-related skills, yet enterprises face critical shortages in AI ethics, security, and large-language model oversight. That gap matters because LLMs are increasingly embedded in real workflows such as writing code, generating configs, suggesting dependencies, and shaping decisions that can introduce risk at machine speed.
In this episode of AppDevANGLE, I spoke with Sohrob Kazerounian, Distinguished AI Researcher at Vectra AI, about why humans must stay in the loop as AI adoption accelerates, and how new threat patterns, especially slop squatting, are emerging precisely because LLMs are “good enough” to be trusted, but not yet trustworthy enough to run unattended. We also explored the concept of defensive UX as a design principle for safe LLM deployment, and why governance models like a GenAI “center of excellence” are becoming necessary inside enterprises.
The New Oversight Role: From Doing the Work to Vetting the Work
LLMs have shifted the nature of “AI skills” from niche to universal. Five to ten years ago, ML models were typically narrow and used in constrained ways. Now, with GenAI and LLMs, models are being applied to workflows that are totally new and often invented on the fly.
As Sohrob described it, organizations are experimenting with “new workflows and design patterns” as they try to understand what works, what doesn’t, and how to design LLMs as copilots or as partial automation systems.
But the operational shift is clear: humans increasingly act as editors, testers, validators, and interpreters of machine output. That requires a different skill set than traditional development or security alone. It’s not just “how to prompt,” but how to evaluate correctness, detect hallucination patterns, test behavior under edge cases, and understand failure modes before anything reaches production.
Slop Squatting: A New Supply Chain Attack Born from Autocomplete
“Slop squatting” is one of those threats that instantly makes sense once you understand how LLMs behave. Sohrob framed it bluntly: LLMs are, at their core, autocomplete models. They generate outputs that sound plausible based on patterns, even when they’re inventing.
That’s where the attack vector emerges. An LLM might suggest importing a package that doesn’t exist but sounds like it should. An attacker can then publish a malicious package under that invented name to registries like npm or PyPI, betting that a developer (or an AI-driven pipeline) will pull it in without verifying provenance.
“Slop squatting happens when… it’ll invent a completely plausible sounding package and tell you to import that package,” Sohrob explained. “If you’re an enterprising attacker… I’m going to put up a package that takes that name and incorporates malicious backdoor behavior.”
The reason this is particularly dangerous right now is velocity. In our broader research, 24% of organizations want to ship on an hourly basis. When releases move that fast, the temptation to trust AI output and skip verification increases, especially for teams already stretched thin.
The bigger takeaway is that slop squatting won’t be the last “LLM-native” threat. It’s an early example of a broader trend: new exploit patterns will emerge anywhere people don’t fully understand what models do, how they fail, and where they should (and should not) be trusted.
“Good Enough” to Be Trusted, Not Good Enough to Be Safe
One of the sharpest moments in the conversation was Sohrob’s warning about the “danger zone” of LLM capability. The threat isn’t that LLMs are obviously broken. It’s that they work well enough that people will assume they’re ready for hands-off automation.
“The real risk is that it’s sufficiently good enough that it will convince people that it’s ready,” he said. “But not sufficiently good enough that it should be trusted.”
That’s the trust trap. You ask for code, it compiles. You deploy, it runs. The output looks professional. And that’s exactly why teams may miss hidden vulnerabilities, flawed assumptions, insecure dependency choices, or subtle logic errors until something breaks in production or gets exploited.
Sohrob also pointed out the behavioral risk: even when people know they should review AI output, they may slide into lazy patterns as the tool becomes routine, especially when multi-agent setups are generating code and tests automatically. The convenience curve can outpace the discipline curve.
Defensive UX: Designing LLM Systems That Fail Safely
A concept from this episode that deserves wider adoption is defensive UX, or the idea that if you’re building LLMs into products or workflows, you should design them so failure is non-catastrophic.
“Do so in a way that if they fail, it’s not going to be catastrophic,” Sohrob advised. “Do so in a way that the worst case scenario is not terribly harmful.”
This is the opposite of “full automation first.” Defensive UX implies:
- Choosing narrow slices of automation where error bounds are manageable
- Building robust feedback loops for validation and correction
- Preventing irreversible actions without explicit human approval
- Designing escalation paths when confidence is low or outputs are ambiguous
It’s a mindset shift that treats LLMs like probabilistic systems with failure modes, not deterministic software components. That framing naturally leads to safer rollouts.
Who Owns AI Decisions? Governance Is Catching Up Late
We also dug into accountability. Who is responsible when AI makes a bad decision, writes risky code, or introduces policy violations? The person using the tool? A centralized governance function? Security? Legal? Leadership?
Sohrob explained that regulatory mechanisms and policy frameworks are still immature, and it’s easy for any employee to sign up for ChatGPT or another tool and quietly embed it into workflows. That makes “shadow AI” a real risk even in well-governed organizations.
His recommendation was a practical organizational pattern to create a cross-functional GenAI Center of Excellence (CoE), a stakeholder group responsible for experimentation, policy decisions, risk classification, and guardrails based on PII/IP sensitivity and threat models.
“We’ve called it the center of excellence… experimenting and determining what things make sense and what things don’t… what should be allowed and what shouldn’t be allowed,” he said, noting the approach has been developed alongside partners, including AWS.
From my perspective, this aligns with what we’re seeing more broadly; AI adoption is forcing governance to evolve from static policies to living operating models because usage patterns change too fast for traditional control cycles.
Analyst Take
This episode reinforces a hard truth that LLM deployment is as much a human systems problem as it is a technical one. The models are powerful, but fallible. The tooling is accessible, but easily misused. And the incentives to move fast are stronger than most organizations’ ability to govern risk.
Three themes stood out:
First, slop squatting is a preview of the next wave of supply chain attacks. It exploits a uniquely LLM-shaped behavior: plausible invention. As velocity increases, provenance and package authentication become non-negotiable.
Second, the near-term danger is not failure; it’s believable failure. LLMs are “just good enough” to encourage over-trust. That’s exactly why human-in-the-loop controls, staged autonomy, and verification workflows must be treated as standard operating procedure.
Third, defensive UX should become a default design pattern. If you’re deploying LLMs, design for safe failure, narrow automation, and clear escalation paths. Then wrap it in governance: a CoE or equivalent cross-functional group that can define what’s allowed, what’s not, and who is accountable.
Enterprises that get this right will move faster and safer because they’ll build systems where speed is powered by guardrails, not undermined by shortcuts.
