## TLDR

- We built a new benchmark called [**"Bug In The Code Stack" (BICS)**](https://github.com/HammingHQ/bug-in-the-code-stack) to test how well LLMs can find syntactic bugs in large Python codebases.
- **GPT-3.5-Turbo** showed lower accuracy on the [**BICS benchmark**](https://github.com/HammingHQ/bug-in-the-code-stack) than the [**BABILONG benchmark**](https://arxiv.org/pdf/2402.10790) at the same context length and target depth, indicating that **LLMs struggle more on code-based tasks than text-based tasks at long context length.**
- The hype is real. **GPT-4o** showed the best performance, closely followed by **GPT-4-Turbo**. The **GPT-4-Series** especially performed well at long context lengths compared to other models.
- Generally, **longer context length resulted in lower accuracy.** However, there were some exceptions to this.
- Models react differently to the placement of the bug within the source code. **GPT-3.5-Turbo** and **Claude 3 Opus** were the most sensitive, and **GPT-4-Series** was the least sensitive. Generally, less sensitivity means a more robust model.
- **Gemini 1.5-Pro** is ~3x better than **Gemini 1.0 Pro**; as Google claimed, 1.5-Pro's performance stays constant across all context lengths and target depths.
- **Codestral 22B** performed on par with **GPT-3.5-Turbo** and **Llama3-70B** despite being a significantly smaller model.

## Motivation

As LLMs' context window sizes grow, their use as coding assistants for large codebases is increasing. It's crucial to understand how longer context lengths impact their performance.

The **"needle in the haystack"** analysis tests LLMs' ability to find specific information in long documents. Previous benchmarks like [**BABILONG**](https://arxiv.org/pdf/2402.10790) focused on text tasks. Now, as LLMs are used more for coding, it's important to see how they perform on code tasks and if the task type affects their accuracy.

## Experimental design

We developed a new benchmark called [**Bug In The Code Stack**](https://github.com/HammingHQ/bug-in-the-code-stack) (BICS), which contains auto-assembled Python source code as the **haystack** and a syntactic bug placed within the source code as the **needle**. The LLM is tasked with finding the line number and the type of the bug.

<Image src="/images/needle-in-the-haystack.webp" width="400" height="200" />

<Image
  src="/images/short_input_example.png"
  width="400"
  height="300"
  figure="Figure 1."
  caption="Sample source code with a syntactic bug, the LLM is tasked with retrieving both the line number (e.g. Line 4) and the type of the bug (e.g. Missing Parenthesis) that occurs in the input code. We place the bug at various depths within the source code to evaluate the LLM's ability to retrieve the bug accurately."
/>

Each model was run on context lengths ranging from 500 tokens to 16K tokens and target depths ranging from 0% to 100%. We ran each experiment 25 times, and the average accuracy is shown in the following charts.

To give context, **16K tokens are around 25 pages long**. The models are challenged to find a single syntactic bug, which could be as small as a missing parenthesis, within 25 pages of code! This benchmark poses quite a challenge to many of the models.

## Comparing results on most popular models

<Image
  src="/images/bics_agg_result_depth_050.png"
  width="600"
  height="600"
  figure="Figure 2."
  caption="Comparing the average accuracy of each model at a target length of 50%. The performance for GPT-4o and GPT-4-Turbo stays consistently high at various context lengths. Note: Llama3-70B has a context window of 8192 tokens; the 16K tokens tests are omitted."
/>

<Image
  src="/images/avg_acc_excluding_16k_result.png"
  width="600"
  height="600"
  figure="Figure 3."
  caption="Comparing the average accuracy across all context lengths and target depth for each model, excluding 16K tokens results. GPT-4o has ~2x higher accuracy than 3.5, Llama3-70B & Command-R+"
/>

<Image
  src="/images/variance_in_avg_acc_per_depth_result.png"
  width="600"
  height="600"
  figure="Figure 4."
  caption="Comparing the variance in average accuracy per target depth for each model, a higher variance indicates that the model is more sensitive to the placement of the bug within the source code. Claude 3 Opus, 3.5 Turbo and Command-R+ are the most sensitive."
/>

From the charts above, we can see the performance gap between different models, with **GPT-4o** performing the best at both short and long context lengths, closely followed by **GPT-4-Turbo**. **Claude 3 Opus** shows a similar level of performance at short context lengths but struggles at long context lengths. Additionally, **GPT-3.5-Turbo**, **Codestral 22B**, **Llama3-70B**, and **Command-R+** all show similar performance levels.

## Comparing BICS and BABILONG

<Image
  src="/images/bics_vs_babilong_gpt_35_turbo_result.png"
  width="400"
  height="300"
  figure="Figure 5."
  caption="GPT-3.5-Turbo performs 5-10x worse on BICS than BABILONG at a target depth of 75%; i.e. our benchmark is harder than BABILONG."
/>

In addition, we see that LLMs display much lower accuracy on the [**BICS benchmark**](https://github.com/HammingHQ/bug-in-the-code-stack) than the [**BABILONG**](https://arxiv.org/pdf/2402.10790) benchmark. This indicates that LLMs struggle more at understanding long codebases than long text, hinting at a future improvement to the models for code comprehension capabilities.

## Detailed results

Here are the detailed results for each model.

<Image
  src="/images/bics_gpt_4o_result.png"
  width="600"
  height="400"
  figure="Figure 6."
  caption="GPT-4o: Extremely high retrieval performance across all context lengths and target depths."
/>

<Image
  src="/images/bics_gpt_4_turbo_result.png"
  width="600"
  height="400"
  figure="Figure 7."
  caption="GPT-4-Turbo: High retrieval performance at short context lengths; seems to struggle at 50% depth for longer contexts."
/>

<Image
  src="/images/bics_claude_3_opus_result.png"
  width="600"
  height="400"
  figure="Figure 8."
  caption="Claude 3 Opus: Sharp drop in accuracy at most depths for longer context lengths."
/>

<Image
  src="/images/bics_gemini_15_pro_result.png"
  width="600"
  height="400"
  figure="Figure 9."
  caption="Gemini-1.5-Pro: Okay-ish performance that holds up for longer context lengths."
/>

<Image
  src="/images/bics_gemini_15_flash_result.png"
  width="600"
  height="400"
  figure="Figure 10."
  caption="Gemini-1.5-Flash: Sharp drop in accuracy at most depths for longer context lengths like Opus."
/>

<Image
  src="/images/bics_gpt_35_turbo_result.png"
  width="600"
  height="400"
  figure="Figure 11."
  caption="GPT-3.5-Turbo: Weirdly better at bugs placed at the end of the code."
/>

<Image
  src="/images/bics_codestral_result.png"
  width="600"
  height="400"
  figure="Figure 12."
  caption="Codestral 22B: The best performance / model size out of the models we tested."
/>

<Image
  src="/images/bics_command_r_plus_result.png"
  width="600"
  height="400"
  figure="Figure 12."
  caption="Command-R+: Poor performance at target depth < 0.25, worst at longer context lengths."
/>

<Image
  src="/images/bics_gemini_10_pro_result.png"
  width="600"
  height="400"
  figure="Figure 13."
  caption="Gemini-1.0-Pro: Bad at everything."
/>

<Image
  src="/images/bics_llama3_70b_result.png"
  width="600"
  height="400"
  figure="Figure 14."
  caption="Llama3-70B: On par with GPT-3.5-Turbo; very impressive for a small model."
/>

## Future experiments

The [**"Bug In The Code Stack"**](https://github.com/HammingHQ/bug-in-the-code-stack) benchmark presents a new challenge measuring LLMs' capability at long context lengths. In the future, we would also like to extend the benchmark by adding logical errors that cannot be detected using static code analyzers, which further helps evaluate the capabilities of the models. In addition, we can run experiments with different programming languages, such as **Javascript** or **C++**, and observe the performance difference.

<Callout emoji="🎉">
  Special thanks to [Bingxu Hu](https://www.linkedin.com/in/bingxuhu/) and
  [Sumanyu Sharma](https://www.linkedin.com/in/sumanyusharma/) for helping
  structure the experiments and revising the article. We've open-sourced the
  benchmark [**Bug In The Code
  Stack**](https://github.com/HammingHQ/bug-in-the-code-stack). Any help to
  improve the benchmark or run it on additional models would be highly
  appreciated 🙂.
</Callout>

# Call Analytics for AI Voice Agent Testing

We're excited to announce our new analytics module for visualizing performance metrics during automated voice agent testing.

You can now quickly understand how your AI voice agents are performing over time.

<Image src="/images/testing_trends.png" width="600" height="300" />

## Comprehensive Performance Metrics during Voice Agent Testing

Track critical metrics like:

- p50 latency measurements (median response time - half of all responses are faster than this)
- p90 latency measurements (90th percentile response time - 90% of responses are faster than this)
- Call duration (average, min, max)
- Scoring metrics (accuracy, completeness, etc.)

## Why this matters

Understanding response times and performance metrics is crucial for delivering exceptional customer experiences with AI voice agents. Slow or inconsistent responses can frustrate users and damage brand reputation. Our analytics module enables you to:

- **Optimize User Experience**: Monitor and improve response times to ensure natural, fluid conversations
- **Reduce Operational Costs**: Identify and fix inefficiencies before they impact your bottom line
- **Ensure Reliability**: Track system stability and catch potential issues early
- **Drive Continuous Improvement**: Make data-driven decisions to enhance voice agent performance

With comprehensive analytics, you can confidently scale your voice AI operations while maintaining high quality standards. Teams can quickly identify areas for optimization and measure the impact of improvements over time.

## Getting Started

To begin using the new analytics module:

1. Go to [Voice Agents](https://app.hamming.ai/voice-agents?page=1) and select a voice agent
2. Navigate to the 'Trends' section
3. Start tracking performance

## Looking Forward

This analytics module represents our commitment to providing comprehensive tools for voice AI testing and optimization. We're grateful to **Jordan Farnworth from Podium** for helping us improve our analytics capabilities.

More to come!

# Enhanced Call Debugging with SIP Status Tracking

We're excited to announce new debugging features that provide clear insights into call termination and SIP status for voice AI agent testing.

These improvements make it easier than ever to identify and resolve issues during voice agent development and testing.

<Image src="/images/call_status.png" width="600" height="300" />
<Image src="/images/sip_status.png" width="600" height="300" />

## What's New

- Call termination identification
- Real-time SIP status tracking
- Comprehensive session logging
- Detailed connection state monitoring

## Why It Matters

Quick issue identification and resolution are crucial for maintaining efficient voice AI operations. With enhanced debugging capabilities, you can:

- **Accelerate Troubleshooting**: Immediately identify who ended calls and why
- **Improve Reliability**: Monitor SIP status and connection states in real-time
- **Reduce Downtime**: Quickly pinpoint and resolve communication issues
- **Optimize Performance**: Track and analyze call session metrics

These tools enable teams to maintain high-quality voice AI services while minimizing debugging time and effort.

## How to Use

To access the new debugging features:

1. Navigate to your [Hamming AI dashboard](https://app.hamming.ai)
2. Select a voice agent
3. Open the call logs
4. View enhanced termination and SIP status details

## Key Features

### Call Termination Tracking

- Agent-specific identifiers
- Timestamp correlation
- Clear termination indicators
- Detailed event logging

### SIP Status Monitoring

- Protocol-level insights
- Connection state visualization
- Error code identification
- Real-time status updates

## Looking Forward

These debugging enhancements demonstrate our commitment to making voice AI development more efficient and accessible. Special thanks to **Marius Buleandra** for implementing these highly requested features.

Stay tuned for more updates as we continue to improve the voice AI testing experience!

# DTMF Support for Voice AI Testing

We're excited to announce that Hamming AI now supports DTMF (Dual-Tone Multi-Frequency) testing, enabling comprehensive automated testing for voice agents that accept keypad inputs.

This feature eliminates the need for manual testing of DTMF-enabled systems and allows you to simulate real-world user interactions through keypad inputs.

<Image src="/images/dtmf_testing.png" width="600" height="300" />

## What's New

- Complete numeric keypad simulation (0-9, \*, #)
- Multi-digit sequence testing
- Combined voice and DTMF interaction testing
- Automated timing validation
- IVR menu navigation testing

## Why It Matters

DTMF testing is crucial for voice AI systems that handle customer service, automated phone systems, and interactive voice response (IVR) applications. With automated DTMF testing, you can:

- **Ensure Reliability**: Validate proper handling of keypad inputs across all scenarios
- **Save Time**: Eliminate manual testing and reduce QA effort
- **Improve Coverage**: Test complex interaction patterns systematically
- **Maintain Quality**: Ensure consistent handling of DTMF inputs

This comprehensive approach enables teams to build more reliable voice AI systems while significantly reducing testing overhead.

## How to Use

To start using DTMF testing:

1. Access your [Hamming AI dashboard](https://app.hamming.ai)
2. Configure DTMF test scenarios
3. Define input sequences
4. Run automated tests

## Use Cases

### Menu Navigation Testing

- Validate numeric menu selections
- Test option response accuracy
- Verify navigation paths
- Ensure proper error handling

### Data Entry Validation

- Test phone number inputs
- Validate PIN code entry
- Verify account number processing
- Test form completion flows

### Mixed Interaction Testing

- Combine voice and DTMF inputs
- Test complex workflows
- Validate multi-step processes
- Test fallback scenarios

## Looking Forward

This DTMF support feature represents our commitment to comprehensive voice AI testing. Special thanks to **Marius Buleandra** for implementing this feature and to **Dietmar Meister** for the inspiration behind it.

Contact our team to learn more about how DTMF testing can enhance your voice AI development process.

# Strengthening Voice AI Testing with Fluents.ai Partnership

We're excited to announce our strategic partnership with Fluents.ai, bringing enterprise-grade voice AI testing capabilities to more organizations. This collaboration enables teams to build more reliable and trustworthy voice AI systems while accessing premium features from both platforms.

## Partnership Benefits

### For Fluents.ai Customers

- 100 free automated test calls
- Comprehensive bug detection
- Real-time production monitoring
- Immediate alert system for:
  - Agent mistakes
  - Hallucinations
  - Interaction issues

### For Hamming AI Customers

- 15% discount on Fluents.ai enterprise workflows
- Access to advanced voice agent features
- Enhanced testing capabilities
- Seamless integration options

## Key Features

### Automated Testing

- Comprehensive test coverage
- Bug detection automation
- Performance monitoring
- Quality assurance tools

### Production Monitoring

- Real-time call analysis
- Immediate issue detection
- Performance tracking
- Quality metrics

## Implementation Benefits

1. **Enhanced Reliability**

   - Automated testing workflows
   - Comprehensive monitoring
   - Quick issue detection
   - Rapid problem resolution

2. **Cost Efficiency**

   - Reduced testing time
   - Automated bug detection
   - Streamlined workflows

3. **Quality Assurance**
   - Continuous monitoring
   - Performance tracking
   - Issue prevention
   - Quick resolution

## Getting Started

To take advantage of this partnership:

1. Email us at [contact@hamming.ai](mailto:contact@hamming.ai)
2. Let us know you're a Fluents.ai customer
3. We'll white-glove the integration for you and give you 100 free automated test calls

## Looking Forward

This partnership represents our commitment to making voice AI more reliable and accessible. By combining Hamming AI's testing capabilities with Fluents.ai's enterprise workflows, we're helping teams build more trustworthy voice AI systems.

Contact our team to learn more about how this partnership can help you build trustworthy AI voice agents.

# Our AI Grant Weekend: Learning, Connection, and What's Next

I'm still processing all the amazing conversations and insights from our recent AI Grant weekend. Between the intense discussions and brainstorming sessions, I barely remembered to take photos (and the ones I did take are definitely not winning any photography awards 😅).

## High Energy Weekend

The energy was incredible. We had the privilege of spending time with the AI Grant team:

- Hersh Desai
- Lenny Bogdonoff
- Asara Near
- Luke Farritor
- Daniel Friedman
- Nat Friedman

## What This Means for Hamming AI

We're doubling down on making voice AI testing more robust and accessible, and now we have an incredible network to tap into to grow faster.

## Growing Our Team

Speaking of growth - we're hiring! We especially love working with ex-founders. There's something special about the problem-solving mindset and customer obsession that comes from having built something from scratch.

If you're an ex-founder or know someone who is, and you're excited about making voice AI more reliable, let's chat. We have a shot at building a legendary company.

## Let's Connect

Whether you're interested in joining our team or just want to geek out about voice AI testing:

- Check out our [careers page](https://www.ycombinator.com/companies/hamming-ai/jobs/)
- Find us on [LinkedIn](https://linkedin.com/company/hammingai)
- Drop me a note directly - always happy to connect with fellow builders

Here's to the next chapter of making voice AI more reliable, one test at a time! 🚀

# Announcing Our $3.8M Seed Round to Make Voice AI More Reliable

We're thrilled to announce that we've raised $3.8 million in seed funding led by Mischief, with participation from YCombinator, AI Grant, Pioneer, Coalition Operators, Coughdrop, NEA, Humans Ventures, Twenty Two Ventures, Orange Collective, Transpose Capital, Olive Tree Capital, and incredible angels including Hiten Shah, Ran Makavy, Max Kolysh, Richard Aberman, Kulveer Taggar, Walter Chen, Yossi Eliyahu, Wiktor Makura, Mark Wai, Victor Zhang, Michael Kuznetsov, Kiran Lingam, Joshua Goodfield, Jake Peters, Umesh Khanna, Carl Cortright, Rahim Noorani, Benjamin Bryant, Lawrence Lin Murata, Noah Gale, Raymond Chen, Kerry Wang, Xiao Wei, Michael Babineau, Sisun Lee, Tony Dong, Jake Lynch, Michael Stoppelman, Tianwei Yue, Rob Olsthoorn, Hexi Shaw, Eric Pollmann, Felicia Chen, Eva Zhang, Eduardo Ortiz de Lanzagorta, Nicholas Elledge, and Theodor Marcu.

## Why Now?

People make billions of calls every day. As conversational AI advances, analysts predict that AI will handle the majority of these calls, with thousands of companies building solutions across every vertical. However, making these AI voice agents reliable remains a significant challenge. Small changes in prompts or model providers can cause large variations in voice agent responses. Engineers currently spend hours manually testing their agents—a slow, ad hoc, and often inadequate process.

Even after deployment, operations teams listen to thousands of calls daily to discover edge cases missed during manual testing. This makes voice AI systems not only costly to set up but also potential sources of liability due to insufficient testing infrastructure.

## Our Solution

At Hamming, we're automating the testing, monitoring, and governance for AI voice agents. We've developed AI voice agents that act like real people, capable of placing thousands of simultaneous test calls to client voice agents. Our solution includes:

- LLM prompt management solutions for B2B teams
- Automated AI voice agent red-teaming for vulnerability detection
- Call analytics to track user engagement in production
- Intelligent flagging system for cases needing attention

## Looking Forward

If 2024 was the year of the prototype, 2025 will be the year of reliability. Industries like medicine, law, insurance, and real estate operate under strict compliance frameworks governing business-client interactions. As regulations evolve to address AI realities, automated testing will become fundamental to every long-term, trust-focused AI strategy.

As Lauren Farleigh, Co-Founder and GP at Mischief, notes: "Conversational AI is advancing rapidly—but most testing and governance tools haven't caught up to match developer needs and compliance realities. Hamming.ai will be foundational to ensuring this technology advances safely, as AI reshapes interactions between businesses and their customers."

## About the Founders

We come from backgrounds in trust and safety infrastructure. Marius and I met at Citizen, the Founders Fund-backed personal safety network, in 2019. I helped quadruple Citizen's user base as Head of Data and previously grew an AI-powered sales program at Tesla to hundreds of millions in revenue per year. Marius brings extensive experience in data infrastructure and AI from Anduril, Square, and Microsoft, and was a founding engineer at Spell (acquired by Reddit).

We're excited to use this funding to expand our team and accelerate our mission of making voice AI more reliable and trustworthy. If you're interested in joining us on this journey or learning more about our platform, reach out to us at [founders@hamming.ai](mailto:founders@hamming.ai).

# Real-Time Emotional Analytics with Hume AI Integration

We're excited to announce our integration with Hume AI, bringing emotional characteristics monitoring to production voice agent calls. Now you can understand not just what your voice agents say, but how they say it.

## What's New

- Real-time emotional characteristic tracking
- Voice pitch analysis
- Tone variation monitoring
- Speech rhythm pattern detection
- Caller sentiment tracking

## Why It Matters

Understanding emotional context in voice AI interactions is crucial for delivering exceptional customer experiences. With Hume AI integration, you can:

- **Improve User Experience**: Monitor and optimize emotional engagement
- **Ensure Quality**: Track voice agent emotional consistency
- **Identify Issues**: Catch potential problems before they escalate
- **Drive Improvements**: Make data-driven decisions based on sentiment metrics

This comprehensive approach enables teams to build more empathetic and effective voice AI systems while maintaining high quality standards.

## How to Use

Getting started takes just 5 minutes:

1. Navigate to your [Hamming AI dashboard](https://app.hamming.ai)
2. Access the Monitoring Tab
3. Enable Hume AI integration
4. Add SDK integration code
5. Start monitoring emotional analytics

## Key Features

### Real-Time Monitoring

- Direct access in Monitoring Tab
- Instant sentiment alerts
- Customizable thresholds
- Trend analysis capabilities

### Performance Analytics

- Comprehensive sentiment metrics
- Interaction pattern analysis
- Performance trending
- Actionable insights

## Looking Forward

This integration demonstrates our commitment to making voice AI more reliable and trustworthy. Special thanks to the Hume AI team for their collaboration in bringing emotional intelligence to voice agent monitoring.

Contact our team to learn more about how the Hume AI integration can enhance your voice agent monitoring capabilities.

# How Lilac Labs Ensures Drive-Thru Order Accuracy with Hamming AI

## Background

Lilac Labs, founded by Tony Kam and Shelden Shi, develops AI voice agents for automating drive-thru order taking in Quick Service Restaurants (QSRs). Their primary challenge was ensuring consistent order accuracy while handling complex scenarios like dietary restrictions and allergies.

## The Challenge

Before implementing automated testing, Lilac Labs faced several operational hurdles:

- Manual testing required hours of daily team time
- System changes necessitated comprehensive retesting
- Limited coverage of edge cases and special scenarios
- Difficulty in consistently testing complex orders

## Implementation

Lilac Labs implemented Hamming AI's testing platform in three key phases:

### 1. Test Scenario Development

- Created thousands of customer personas and ordering scenarios
- Developed LLM judges to evaluate voice agent performance
- Focused on critical cases like dietary restrictions and allergies

### 2. Automated Testing

- Enabled concurrent testing of thousands of calls
- Provided detailed analytics on system performance
- Identified specific areas needing improvement

### 3. Production Monitoring

- Implemented real-time monitoring of customer interactions
- Flagged problematic cases automatically
- Converted real calls into test cases for continuous improvement

## Impact

The implementation has driven significant operational improvements, with current annualized metrics showing:

- 5,000 test scenarios in production
- 130,000 automated tests run per year
- 5,200 hours saved annually
- $520,000 in run-rate cost savings per year

## Key Outcomes

1. Automated testing replaced manual processes
2. Improved coverage of edge cases and special requests
3. Faster iteration and development cycles
4. Enhanced confidence in system reliability

For more information about voice AI testing solutions, [contact our team](https://calendly.com/sumanyusharma/1-1-debug-session).

# Multi-Language Support for Voice AI Testing

We're excited to announce multi-language support for voice AI testing, enabling you to validate your voice agents across 11 different languages.

This feature allows you to ensure consistent performance across different languages and cultural contexts, making global deployment of voice AI systems more reliable.

<Image src="/images/multilingual_testing.png" width="600" height="300" />

## What's New

Our platform now supports testing in:

- Dutch
- English
- French
- German
- Hindi
- Indonesian
- Italian
- Japanese
- Korean
- Portuguese
- Spanish

## Key Features

- Run parallel tests in multiple languages simultaneously
- Test pronunciation and accent variations
- Validate language-switching scenarios
- Get language-specific performance metrics
- Compare cross-language performance
- Analyze accent and dialect coverage

## Why It Matters

Global voice AI deployment requires thorough testing across languages. With multi-language support, you can:

- **Expand Market Reach**: Validate voice AI performance for international markets
- **Ensure Consistency**: Maintain reliable performance across languages
- **Reduce Risks**: Catch language-specific issues before deployment
- **Save Time**: Test multiple languages in parallel

## How to Use

To start testing in multiple languages:

1. Go to [Voice Agents](https://app.hamming.ai/voice-agents?page=1)
2. Select your voice agent
3. Choose target languages
4. Configure test scenarios
5. Run tests and review metrics

## Looking Forward

This multi-language support is our first step toward enabling truly global voice AI testing. We're committed to expanding language coverage and enhancing testing capabilities based on your needs.

Contact our team to learn how multi-language testing can help ensure your voice AI system performs reliably across global markets.

# Strengthening Voice Agent Testing with Retell AI Partnership

We're excited to announce our strategic partnership with Retell AI, bringing comprehensive voice agent testing capabilities to more organizations. This collaboration enables teams to build more reliable and trustworthy voice AI systems while leveraging Retell's production-ready voice agent platform.

## Partnership Benefits

### For Retell AI Customers

- 100 free automated test calls
- Comprehensive bug detection
- Real-time production monitoring
- Immediate alert system for:
  - Agent mistakes
  - Hallucinations
  - Interaction issues

### For Both Platforms

- Seamless integration
- Enhanced testing capabilities
- 5-minute setup process
- SDK-based implementation

## Key Features

### Automated Testing

- Comprehensive test coverage
- Scenario-based testing
- Performance validation
- Quality assurance tools

### Production Monitoring

- Real-time call analysis
- Immediate issue detection
- Continuous performance tracking
- Interaction quality metrics

## Implementation Benefits

1. **Enhanced Reliability**

   - Automated testing workflows
   - Real-time monitoring
   - Quick issue detection
   - Rapid problem resolution

2. **Development Efficiency**

   - Reduced manual testing
   - Faster iteration cycles
   - Streamlined workflows

3. **Quality Assurance**
   - Continuous monitoring
   - Performance validation
   - Issue prevention
   - Quick resolution

## Getting Started

To take advantage of this partnership:

1. Visit our [partnership page](/partners/retell)
2. Contact our team to claim free test calls
3. Complete the 5-minute integration
4. Begin monitoring production calls

## Looking Forward

This partnership represents our commitment to making voice agents more reliable and efficient. By combining Hamming AI's testing capabilities with Retell AI's voice agent platform, we're helping teams build more trustworthy voice AI systems.

Contact our team to learn more about how this partnership can enhance your voice agent development process.

# Selective Scenario Re-runs for Voice AI Testing

We're excited to introduce selective scenario re-runs, a new feature that enables you to execute specific test scenarios from your existing datasets.

This feature allows you to validate individual test cases without running your complete test suite, significantly improving testing efficiency.

<Image src="/images/select_individual_runs.png" width="600" height="300" />
<Image src="/images/rescore.png" width="600" height="300" />

## What's New

- Selective scenario re-runs: Choose specific test cases to validate without running the entire test suite
- Re-scoring functionality: Apply new scoring metrics to previously completed tests

## Why It Matters

Efficient testing is essential for rapid development and deployment of voice AI agents. Running complete test suites repeatedly can be time-consuming and resource-intensive. With selective re-runs, you can:

- **Accelerate Development**: Test specific changes quickly and efficiently
- **Optimize Resources**: Minimize processing overhead by running only necessary tests
- **Enhance Accuracy**: Focus on specific scenarios for precise validation
- **Improve Workflow**: Integrate targeted testing into your development process

This targeted approach enables teams to iterate faster while maintaining high quality standards, making issue identification and resolution more efficient.

## How to Use

To use selective scenario re-runs:

1. Navigate to [Voice Agents](https://app.hamming.ai/voice-agents?page=1)
2. Select a voice agent
3. Press "Run Scenarios"
4. Select the scenarios you want to re-run

## Looking Ahead

The selective re-run feature demonstrates our commitment to making voice AI testing more efficient and accessible. Special thanks to **Marius Buleandra** for shipping this highly requested capability.

Happy Thanksgiving from Hamming AI! 🍁

To celebrate this season of gratitude, I want to thank our customers, partners, Marius Buleandra, and team for believing in us and helping build an incredible product that makes AI Voice Agents more reliable and trustworthy.

Grateful for the work we've done. Much more to do. We like working 7 days a week, including holidays like today, to serve customers.

If you're interested in joining us, check out our [careers page](https://www.ycombinator.com/companies/hamming-ai/jobs/).

Hamming AI Blog

Hamming AI Raises $3.8M Seed Round

Recent

DTMF Support for Comprehensive Voice Agent Testing

Selective Scenario Re-runs for Voice AI Testing

Enhanced Call Debugging with SIP Status Tracking

Hamming AI Partners with Fluents.ai for Enhanced Voice AI Testing

Hamming AI Launches Advanced Call Analytics for Voice Agent Testing

Hamming AI Integrates with Hume AI for Enhanced Voice Agent Monitoring

Thanksgiving Update from Hamming AI

Multi-Language Support for Voice AI Testing

AI Grant Demo Day: A Weekend of Learning and Connection

Customer Spotlight: How Lilac Labs (YC S24) Ensures Drive-Thru Order Accuracy with Hamming AI

Hamming AI Partners with Retell AI for Enhanced Voice Agent Testing

Can LLMs find bugs in large codebases?