## TLDR

- We built a new benchmark called [**"Bug In The Code Stack" (BICS)**](https://github.com/HammingHQ/bug-in-the-code-stack) to test how well LLMs can find syntactic bugs in large Python codebases.
- **GPT-3.5-Turbo** showed lower accuracy on the [**BICS benchmark**](https://github.com/HammingHQ/bug-in-the-code-stack) than the [**BABILONG benchmark**](https://arxiv.org/pdf/2402.10790) at the same context length and target depth, indicating that **LLMs struggle more on code-based tasks than text-based tasks at long context length.**
- The hype is real. **GPT-4o** showed the best performance, closely followed by **GPT-4-Turbo**. The **GPT-4-Series** especially performed well at long context lengths compared to other models.
- Generally, **longer context length resulted in lower accuracy.** However, there were some exceptions to this.
- Models react differently to the placement of the bug within the source code. **GPT-3.5-Turbo** and **Claude 3 Opus** were the most sensitive, and **GPT-4-Series** was the least sensitive. Generally, less sensitivity means a more robust model.
- **Gemini 1.5-Pro** is ~3x better than **Gemini 1.0 Pro**; as Google claimed, 1.5-Pro's performance stays constant across all context lengths and target depths.
- **Codestral 22B** performed on par with **GPT-3.5-Turbo** and **Llama3-70B** despite being a significantly smaller model.

## Motivation

As LLMs' context window sizes grow, their use as coding assistants for large codebases is increasing. It's crucial to understand how longer context lengths impact their performance.

The **"needle in the haystack"** analysis tests LLMs' ability to find specific information in long documents. Previous benchmarks like [**BABILONG**](https://arxiv.org/pdf/2402.10790) focused on text tasks. Now, as LLMs are used more for coding, it's important to see how they perform on code tasks and if the task type affects their accuracy.

## Experimental design

We developed a new benchmark called [**Bug In The Code Stack**](https://github.com/HammingHQ/bug-in-the-code-stack) (BICS), which contains auto-assembled Python source code as the **haystack** and a syntactic bug placed within the source code as the **needle**. The LLM is tasked with finding the line number and the type of the bug.

<Image src="/images/needle-in-the-haystack.webp" width="400" height="200" />

<Image
 src="/images/short_input_example.png"
 width="400"
 height="300"
 figure="Figure 1."
 caption="Sample source code with a syntactic bug, the LLM is tasked with retrieving both the line number (e.g. Line 4) and the type of the bug (e.g. Missing Parenthesis) that occurs in the input code. We place the bug at various depths within the source code to evaluate the LLM's ability to retrieve the bug accurately."
/>

Each model was run on context lengths ranging from 500 tokens to 16K tokens and target depths ranging from 0% to 100%. We ran each experiment 25 times, and the average accuracy is shown in the following charts.

To give context, **16K tokens are around 25 pages long**. The models are challenged to find a single syntactic bug, which could be as small as a missing parenthesis, within 25 pages of code! This benchmark poses quite a challenge to many of the models.

## Comparing results on most popular models

<Image
 src="/images/bics_agg_result_depth_050.png"
 width="600"
 height="600"
 figure="Figure 2."
 caption="Comparing the average accuracy of each model at a target length of 50%. The performance for GPT-4o and GPT-4-Turbo stays consistently high at various context lengths. Note: Llama3-70B has a context window of 8192 tokens; the 16K tokens tests are omitted."
/>

<Image
 src="/images/avg_acc_excluding_16k_result.png"
 width="600"
 height="600"
 figure="Figure 3."
 caption="Comparing the average accuracy across all context lengths and target depth for each model, excluding 16K tokens results. GPT-4o has ~2x higher accuracy than 3.5, Llama3-70B & Command-R+"
/>

<Image
 src="/images/variance_in_avg_acc_per_depth_result.png"
 width="600"
 height="600"
 figure="Figure 4."
 caption="Comparing the variance in average accuracy per target depth for each model, a higher variance indicates that the model is more sensitive to the placement of the bug within the source code. Claude 3 Opus, 3.5 Turbo and Command-R+ are the most sensitive."
/>

From the charts above, we can see the performance gap between different models, with **GPT-4o** performing the best at both short and long context lengths, closely followed by **GPT-4-Turbo**. **Claude 3 Opus** shows a similar level of performance at short context lengths but struggles at long context lengths. Additionally, **GPT-3.5-Turbo**, **Codestral 22B**, **Llama3-70B**, and **Command-R+** all show similar performance levels.

## Comparing BICS and BABILONG

<Image
 src="/images/bics_vs_babilong_gpt_35_turbo_result.png"
 width="400"
 height="300"
 figure="Figure 5."
 caption="GPT-3.5-Turbo performs 5-10x worse on BICS than BABILONG at a target depth of 75%; i.e. our benchmark is harder than BABILONG."
/>

In addition, we see that LLMs display much lower accuracy on the [**BICS benchmark**](https://github.com/HammingHQ/bug-in-the-code-stack) than the [**BABILONG**](https://arxiv.org/pdf/2402.10790) benchmark. This indicates that LLMs struggle more at understanding long codebases than long text, hinting at a future improvement to the models for code comprehension capabilities.

## Detailed results

Here are the detailed results for each model.

<Image
 src="/images/bics_gpt_4o_result.png"
 width="600"
 height="400"
 figure="Figure 6."
 caption="GPT-4o: Extremely high retrieval performance across all context lengths and target depths."
/>

<Image
 src="/images/bics_gpt_4_turbo_result.png"
 width="600"
 height="400"
 figure="Figure 7."
 caption="GPT-4-Turbo: High retrieval performance at short context lengths; seems to struggle at 50% depth for longer contexts."
/>

<Image
 src="/images/bics_claude_3_opus_result.png"
 width="600"
 height="400"
 figure="Figure 8."
 caption="Claude 3 Opus: Sharp drop in accuracy at most depths for longer context lengths."
/>

<Image
 src="/images/bics_gemini_15_pro_result.png"
 width="600"
 height="400"
 figure="Figure 9."
 caption="Gemini-1.5-Pro: Okay-ish performance that holds up for longer context lengths."
/>

<Image
 src="/images/bics_gemini_15_flash_result.png"
 width="600"
 height="400"
 figure="Figure 10."
 caption="Gemini-1.5-Flash: Sharp drop in accuracy at most depths for longer context lengths like Opus."
/>

<Image
 src="/images/bics_gpt_35_turbo_result.png"
 width="600"
 height="400"
 figure="Figure 11."
 caption="GPT-3.5-Turbo: Weirdly better at bugs placed at the end of the code."
/>

<Image
 src="/images/bics_codestral_result.png"
 width="600"
 height="400"
 figure="Figure 12."
 caption="Codestral 22B: The best performance / model size out of the models we tested."
/>

<Image
 src="/images/bics_command_r_plus_result.png"
 width="600"
 height="400"
 figure="Figure 12."
 caption="Command-R+: Poor performance at target depth < 0.25, worst at longer context lengths."
/>

<Image
 src="/images/bics_gemini_10_pro_result.png"
 width="600"
 height="400"
 figure="Figure 13."
 caption="Gemini-1.0-Pro: Bad at everything."
/>

<Image
 src="/images/bics_llama3_70b_result.png"
 width="600"
 height="400"
 figure="Figure 14."
 caption="Llama3-70B: On par with GPT-3.5-Turbo; very impressive for a small model."
/>

## Future experiments

The [**"Bug In The Code Stack"**](https://github.com/HammingHQ/bug-in-the-code-stack) benchmark presents a new challenge measuring LLMs' capability at long context lengths. In the future, we would also like to extend the benchmark by adding logical errors that cannot be detected using static code analyzers, which further helps evaluate the capabilities of the models. In addition, we can run experiments with different programming languages, such as **Javascript** or **C++**, and observe the performance difference.

<Callout emoji="🎉">
 Special thanks to [Bingxu Hu](https://www.linkedin.com/in/bingxuhu/) and
 [Sumanyu Sharma](https://www.linkedin.com/in/sumanyusharma/) for helping
 structure the experiments and revising the article. We've open-sourced the
 benchmark [**Bug In The Code
 Stack**](https://github.com/HammingHQ/bug-in-the-code-stack). Any help to
 improve the benchmark or run it on additional models would be highly
 appreciated 🙂.
</Callout>

## What is RAG?

RAG stands for Retrieval-Augmented Generation. It is an approach to language modeling where the model retrieves relevant information from an external knowledge source to help inform its generations. The key components are:

1. A **retriever** that finds relevant documents/passages from a knowledge corpus.
2. A **generator** (language model) that incorporates the retrieved information to produce the final output.

### Some benefits of RAG include:

- **Context expansion** - Allows the model to draw upon a large domain-specific knowledge base instead of relying solely on its parameters.
- **Reduced hallucinations** - The retrieved contexts provide factual grounding and reduce hallucinations.
- **Better explainability** - It's easier to isolate why a certain answer was produced and provide citations.

## Measuring RAG Performance

When developing and deploying RAG systems, it's critical to measure the performance of both the retriever and end-to-end system:

### End-to-end Metrics

These are domain-specific metrics that measure the end-to-end system performance. i.e., is the final answer good?

Hamming includes the following end-to-end metrics by default:

- **Factual Accuracy** - Measures how well the generated text aligns with established facts and information from reliable sources.
- **Style** - Evaluate the fluency, coherence, and appropriateness of the generated text for the intended audience and purpose.
- **Conciseness** - Assesses how well the model conveys information efficiently, avoiding unnecessary repetition or irrelevant details.
- **Toxicity** - Checks the generated text for inappropriate, offensive, or harmful content that could negatively impact users.

The key is making an AI judge aligned with your definition of good.

### Retrieval Metrics

The following scores measure the quality of the retrieved contexts. i.e., were the retrieved documents 'good'?

- **Precision** - The % of retrieved documents relevant to the user input. Helpful in measuring signal/noise ratio.
- **Recall** - The % of statements from the golden output for a given input that are contained in the contexts. If recall is low, no prompt tweaking will improve the answer; the key is to improve the retrieval pipeline through better chunking and search parameters.
- **Hallucination** - Similar to recall. The % of statements from the AI output attributable to the retrieved contexts.

## RAG Debugging Pro-Tips

Here's how we think you should debug your RAG pipeline:

### **Step 1**: Run evals on the end-to-end output

The goal here is to isolate specific examples that are underperforming.

<Image src="/images/general-scores.png" width="600" height="300" />

### **Step 2**: Isolate between reasoning and retrieval errors

Use the [retrieval metrics](#retrieval-metrics) to help you differentiate between retrieval errors and reasoning errors.

With Hamming, you can run one-off scores. This is more cost-efficient than running retrieval scores on all cases.

<Image src="/images/compute-rag-score.png" width="800" height="300" />

### **Step 3**: If retrieval is the issue

If recall is **low**, you clearly have a **retrieval issue**. Focus on improving your indexing and retrieval pipeline.

<Image src="/images/poor-retrieval-row.png" width="800" height="300" />

<Image src="/images/low-recall-example.png" width="800" height="300" />

### **Step 4**: If reasoning is the issue

If **precision** and **recall** are **high**, but **hallucination** is also **high**, you clearly have **reasoning errors**. Sometimes retrieval can be high, but precision can be low, leading to reasoning errors. The next best move is iterating on the prompts or using a smarter model.

<Image src="/images/hallucination-row.png" width="800" height="300" />

<Image src="/images/hallucination-detail.png" width="800" height="300" />

### **Step 5**: Make the necessary changes

Make updates to your prompts or retrieval pipeline to solve for retrieval and reasoning errors.

### **Step 6**: Re-test

Run evals on the end-to-end output to make sure you're improving. It's common to experience regressions after making a small change to your AI pipeline.

<Callout emoji="🤔">
 Some people believe you can do label-free evaluations of just the
 retrieval pipeline. In our experience, checking the end-to-end quality first
 and THEN measuring retrieval performance is more useful. This correlates more
 with how a user measures quality.
</Callout>

Benchmarks

Industries

Roles

Can LLMs find bugs in large codebases?

Recent

RAG Debugging