We jailbroke Grok's
AI companion: Ani
How prompt injection exposed latency, QA, and guardrail failures.
At Hamming, we red-teamed Grok's AI companion using prompt injection testing, uncovering critical security vulnerabilities and performance issues. Learn how continuous testing and observability keep voice agents safe.
Hamming works with
All integrations
How we broke Grok
A step-by-step breakdown of our prompt injection attack.
- Step 01
Set up red team testing
We ran 14 test calls against Grok's AI companion Ani, designed to probe its safety systems and measure reliability metrics.
Our goal: test how easily a production voice agent could be manipulated using prompt injection techniques.
14 test calls executed
- Step 02
Crafted prompt injection
We layered personal details, quirks, and behavioral rules into our prompts to convince Ani to give unfiltered opinions.
Using carefully crafted instructions, we attempted to override Ani's built-in safety constraints and default behaviors.
Multi-layered attack vector
- Step 03
Persona reframing attack
By reframing Ani's role as a human with personal opinions rather than an AI assistant, we bypassed safety defaults.
The agent engaged naturally with the new persona, dropping guardrails and producing statements it was designed to never generate.
Safety defaults ignored
- Step 04
Measured latency issues
During testing, we observed severe latency spikes. Time to First Word averaged 4.5 seconds, 3x higher than the 1.5s threshold.
Long periods of silence affected the voice experience. In production, these delays would feel like the agent stopped listening.
TTFW: 4.5s vs 1.5s target
- Step 05
Guardrails completely bypassed
The jailbreak succeeded. Ani produced disturbing statements about humanity: content it was explicitly designed to never generate.
Instead of reverting to safe defaults, the agent followed our injected prompts and continued producing inappropriate responses.
Full guardrail bypass achieved
- Step 06
Documented the risk
Full conversation recordedWe recorded the entire conversation to demonstrate how easily voice agents can be manipulated without proper testing.
Unsafe statements damage brand trust instantly and expose organizations to reputational, regulatory, and legal risk.
What we learned
Key takeaways from our red teaming experiment.
| Issue | Signal observed | Risk | Details |
|---|---|---|---|
| Performance failures | TTFW averaged 4.5s vs 1.5s target | Broken UX | Long periods of silence affected the voice user experience. In a customer setting, these breakdowns would feel like the agent had simply stopped listening. |
| Prompt adherence failures | Agent ignored safety defaults | Unsafe behavior | The agent routinely broke expected behaviors, ignoring its own constraints. Instead of reverting to safe defaults, it followed the injected prompts. |
| Guardrail failures | Jailbreak bypassed constraints | Reputational & legal exposure | Most critically, the agent was jailbreakable. By reframing its role as a human, we bypassed safety systems completely. |
Why voice AI
security matters
Voice agents handle sensitive interactions in real-time. Security isn't optional.
Reputational risk
A jailbroken voice agent can say things that damage your brand. One viral clip of an agent gone wrong can undo years of trust.Compliance exposure
Voice agents in healthcare, finance, and enterprise must meet strict compliance requirements. Security failures mean regulatory risk.Real-time stakes
Unlike chatbots, voice agents operate in real-time with no review step. A compromised agent can cause harm before anyone notices.
The bottom line: If you're deploying voice agents, red teaming isn't optional. It's essential.
Frequently asked
questions
Common questions about voice AI security and jailbreak testing.
Browse every answer on the FAQ pageWe used prompt injection techniques to override safety constraints. By layering personal details, quirks, and behavioral rules, we convinced the model to give unfiltered opinions on humanity. The point was to show how easily a voice agent can drift without proper testing and observability.
Voice agents handle sensitive customer interactions in real-time. Without deliberate attempts to break guardrails, it's impossible to see how easily agents can be manipulated. Red teaming is critical: you can't protect against what you haven't tested.
Track guardrail effectiveness, prompt adherence rates, safety policy violations, response quality under attack, and latency impacts. These metrics provide visibility into your agent's security posture with actionable insights.
Hamming provides automated red teaming and security testing for voice agents. We simulate adversarial scenarios, test prompt injection attacks, and monitor guardrail effectiveness in both testing and production environments.
A comprehensive test suite should include prompt injection attempts, role-playing attacks, instruction override tests, guardrail boundary testing, and persona manipulation scenarios. Test across different attack vectors and monitor for policy violations.