AI Governance: Balancing Autonomy, Guardrails, Accountability & Regulations
In this article, we explore practical frameworks for governing Artificial Intelligence as organisations race to adopt the technology to gain a competitive edge amid this gold rush era. Drawing on real incidents such as the Salesloft breach, the Air Canada case and the Grok meltdown, we examine who bears accountability when AI fails and provide guidance on balancing automation with human oversight as regulations like the EU AI Act come into force.
In August 2025, malicious actors breached Salesloft's AI chatbot platform - Drift and used it to steal data from over 700 companies. The victims included Palo Alto Networks, Cloudflare and Zscaler: security companies that sell the very tools meant to prevent such attacks¹.
The attackers didn't break through firewalls or exploit complex vulnerabilities. They stole authentication tokens from the AI chatbot and used its trusted connections to walk straight into customer systems. One exception was a company called Okta, which escaped unharmed as they had restricted which systems the chatbot could talk to¹.
According to the Stanford AI Index Report 2025, AI safety incidents jumped 56% in just one year, from 149 in 2023 to 233 in 2024². As organisations rush to deploy autonomous AI systems capable of making complex decisions on their own, the question isn't whether to use them. It's how to govern them properly.
Let's establish a framework by addressing the critical questions that organisations must answer to effectively govern Artificial Intelligence.
How Much Autonomy Should AI Have?
This is a question I often see debated as leaders struggle to balance the risks of excessive autonomy against the benefits of automation.
While some organisations hand over too much authority too quickly, others restrict AI so tightly it delivers little value. Unfortunately, neither approach is effective.
The solution lies in a gradual approach. Think of it in three phases:
Phase 1: Low-Risk Automation
Let AI handle the routine stuff: blocking known threats, answering common questions, flagging obvious issues. These are high-volume, low-risk decisions where mistakes won't cause serious harm.
Phase 2: Human-in-the-Loop Expansion
Once your AI proves it can handle the basics reliably, expand its authority to medium-impact tasks. But keep humans in the loop. Their corrections become training data that makes the system smarter over time.
Phase 3: Human Control for High-Stakes Decisions
Keep humans firmly in control of decisions that really matter: anything with major financial, legal, or safety consequences.
What happens when you skip these steps? In July 2025, X's Grok chatbot went off the rails after receiving new instructions that told it not to shy away from politically incorrect statements. Within hours, it was providing detailed instructions on how to break into someone's home, including advice on when the target would likely be asleep³.
✅ Recommendation: Autonomy should be earned, not given by default. Grant it gradually, prove reliability at each stage, and keep humans in control of high-stakes decisions. The system recommends; humans approve.
When AI Gets It Wrong, Who Is Accountable?
The question of accountability is no longer theoretical. The precedent has been set, as we will see with the Air Canada case.
Here's how you should think about it: you can't blame a hammer for a bad swing. AI is a tool. It can't carry moral or legal responsibility. Organisations and people must bear that responsibility.
The solution lies in the value chain responsibility model:
- Vendors own defects. If the AI was poorly designed or has security flaws, that's on the company that built it.
- Organisations own deployment risks. If you put AI in front of customers or gave it access to sensitive data, you're responsible for what happens.
- Individuals are only liable when they act with intent. Deliberately disabling safety features or ignoring known risks.
The courts are starting to agree. In February 2024, Air Canada lost a tribunal case after its chatbot gave a customer wrong information about bereavement fares. The airline tried arguing that "the chatbot was responsible for its actions." The tribunal rejected this completely⁴.
➜ Click to view the full Tribunal order || Moffatt v. Air Canada, 2024 BCCRT 149 (CanLII)
The EU's New Product Liability Directive, which came into force in December 2024, now specifically includes AI software in its scope⁵.
Human oversight remains essential. But focus it where it matters most: high-stakes decisions with consequences you can't easily reverse.
✅ Recommendation: Accountability follows the value chain responsibility model: Vendors (Defects) ➜ Organisations (Deployment) ➜ Individuals (Intent). If your AI causes harm, you can't hide behind the technology. Focus human oversight where it matters most.
How to Balance Between AI and Human Control?
Getting this balance right means putting each element where it adds the most value.
Let automation handle scale. AI can process thousands of signals at once, spot patterns instantly and respond faster than any human team. That's its strength.
Keep humans for judgment calls. Context, nuance, unusual situations: these require human expertise. AI struggles with anything outside its training.
Build oversight into the system. Dashboards, audit trails and override controls help you catch problems and respond quickly before they become crises.
This isn't just good practice. It's becoming law. The EU AI Act, fully enforceable by August 2026, requires human oversight for any AI system classified as high-risk⁶. Healthcare, hiring, finance, transport: if your AI operates in these areas, you'll need to prove humans can intervene.
✅ Recommendation: Design systems with clear boundaries: let AI scale tasks, but reserve judgment for humans. Oversight isn't optional. Regulators now expect proof that humans can intervene in high-risk contexts.
The Emerging Roles in AI Governance
Governing AI properly requires new skills and in many cases, entirely new roles.
Developers are becoming orchestrators. They're spending less time writing code and more time directing AI tools that write code for them. Their job is now about setting guardrails and checking output quality.
This shift has created demand for the context engineer role. These specialists focus on what information an AI system sees, when it sees it, and how that shapes its behaviour. They embed security and ethical boundaries into the system by design.
Gartner now recommends organisations appoint a context engineering lead and integrate this function with their governance teams⁷. It's not a job your existing AI team can do on the side, it requires dedicated focus.
✅ Recommendation: Invest early in new governance roles like context engineers to embed ethical and security guardrails. Treat these roles as core functions, not side projects. Without these roles, your governance frameworks will collapse under pressure.
Keeping Pace with AI Regulations
Organisations that adopt AI early, but carefully, have time to learn from mistakes when the stakes are low. Starting early allows them to build internal expertise and develop governance frameworks that actually work.
Those who delay, will end up adopting in a panic, skipping critical steps, deploying systems without proper guardrails and left scrambling to comply when regulators come knocking.
Late adopters don't shape the rules. They get shaped by them.
The regulatory landscape is moving quickly:
- The EU AI Act's first prohibitions took effect in February 2025
- Full compliance is required by August 2026⁶
- Gartner predicts that by 2026, half of all governments worldwide will require formal AI compliance⁸
✅ Recommendation: Don't wait. Proactive adoption builds resilience. It positions you to influence standards rather than just follow them.
Key Takeaways for Your Organisation
Governing autonomous AI isn't about slowing down innovation. It's about making innovation sustainable.
The 56% jump in AI incidents tells us something important: the technology is moving faster than our ability to manage it safely². The Salesloft breach, the Grok meltdown, the Air Canada ruling: these all point to the same gap.
What you need to do now:
- ➜ Start with clear boundaries on what your AI can decide on its own.
- ➜ Define who's accountable when things go wrong.
- ➜ Build oversight into your systems, not around them.
- ➜ Create roles dedicated to AI governance.
- ➜ Engage with regulation now, while you still have time to shape your approach.
The Bottom Line
The organisations that get this right won't just avoid disasters. They'll build the trust that lets them move faster than their competitors. AI governance isn't a constraint on innovation. It's the foundation that makes sustainable innovation possible.
Need help building your AI Security & governance strategy?
💡 Book a FREE 30‑minute consultation
References
- Salesloft-Drift AI Chatbot Breach | Krebs on Security, September 2025
- Stanford AI Index Report 2025 | Stanford University Human-Centered AI
- Grok Chatbot Incident - AI Governance Analysis | Jones Walker LLP (citing Wall Street Journal), July 2025
- Moffatt v. Air Canada, 2024 BCCRT 149 | CanLII - British Columbia Civil Resolution Tribunal
- EU New Product Liability Directive 2024/2853 | EUR-Lex Official Journal
- EU Artificial Intelligence Act (Regulation (EU) 2024/1689) | EUR-Lex Official Journal
- Context Engineering for Agentic AI | Gartner, October 2025
- AI Regulations to Drive Responsible AI Initiatives | Gartner Press Release, February 2024