# Can Bölük: I Improved 15 LLMs at Coding in One Afternoon. Only the Harness Changed.

Source: https://www.youtube.com/watch?v=kFtHOtKlb7Q
Recap page: https://rapidrecap.app/video/kFtHOtKlb7Q
Generated: 2026-02-15T16:03:17.887+00:00

---
## Quick Overview

Can Bölük improved the coding performance of 15 Large Language Models (LLMs) by implementing a custom harness called Hashline, which involved adding specific tags to code lines, leading to a significant drop in failure rates from 68% to 46.2% for the GLM 4 model compared to the standard approach.

**Key Points:**
- Hashline, a custom harness, improved the coding success rate of 15 LLMs, including GLM 4, which saw its failure rate drop from 68% to 46.2%.
- The technique involves tagging every line of code with a pseudo-random token ID (e.g., 1.83) to provide context, effectively removing the need for perfect recall of line numbers.
- The improvement was achieved in a single afternoon by swapping the editing tool/harness, not by retraining the models.
- The core issue with standard methods (like Google's default setup) is that they rely on explicit line numbers, which causes failures when models try to edit code.
- The author argues that vendors are locking users into their ecosystems by using proprietary, non-standard harnesses, preventing users from easily adopting superior open-source models.
- The success of Hashline shows that engineering the prompt/context (the harness) can yield massive performance gains without expensive model retraining.
- The failure rate reduction for GLM 4 was nearly a tenfold improvement (from 68% failure rate to 46.2% failure rate is an 8% boost in success, or a 21.8 point drop in failure rate, which is a significant relative improvement).

![Screenshot at 00:00: The video opens with an animated image of two podcasters in front of microphones, overlaid with a pulsing sound wave graphic and the text 'BECOME A MEMBER TODAY!', which is the prompt used for testing the LLMs' performance.](https://ss.rapidrecap.app/screens/kFtHOtKlb7Q/00-00-00.jpg)

**Context:** The video discusses a technique developed by Can Bölük to significantly improve the performance of Large Language Models (LLMs) in coding tasks, specifically when editing existing code. The common approach for LLMs to modify code involves referencing line numbers, which the author argues leads to errors and brittleness, especially when context is manipulated. The experimental setup involved testing 15 different LLMs against a specific coding task using the 'Become a Member Today!' prompt, comparing the standard method against the new harness.

## Detailed Analysis

The discussion centers on an improvement technique called Hashline, developed by Can Bölük, which boosted the coding performance of 15 different LLMs by changing the harness used for editing code, rather than retraining the models themselves. The standard method, often used by models like GPT-4 and those from Google and Anthropic, relies on explicit line numbers (e.g., line 49, 51) for edits, which the author claims leads to failures when the model misinterprets context or indentation. Hashline solves this by assigning a unique, pseudo-random token ID (like 1.83) to every line, effectively acting as a stable, trusted anchor that de-couples the edit intent from the line number. This allowed models to correctly perform edits even when formatting shifted. For example, GLM 4's failure rate on a specific task dropped from 68% using the standard approach to 46.2% using Hashline, an 8% boost in success rate. The author stresses that this improvement, achieved in one afternoon without retraining, highlights the importance of good engineering around the model's execution environment, contrasting this with the perceived short-sightedness of vendors who lock users into proprietary harnesses, which can lead to vendor lock-in and stifle broader adoption of better open-source models.

### The Hashline Harness

- Improved coding success rate on 15 LLMs
- Hashline uses unique, pseudo-random token IDs (e.g., 1.83) to tag every line
- This removes reliance on fragile line numbers for editing context

### Performance Results

- GLM 4 failure rate dropped from 68% to 46.2% (an 8% boost in success)
- Improvement achieved in one afternoon without retraining models
- The technique forces models to process edits based on context, not absolute position

### Vendor Lock-in and Security Risks

- Vendors like Google and Anthropic use proprietary harnesses, creating vendor lock-in
- This forces users to stick with older models (like GPT-4) instead of better open-source options
- The author suggests this is a business model choice favoring control over performance

### The Core Problem

- Standard edit prompts (like 'delete line 50') fail when context shifts or when formatting/indentation changes
- The model's inability to robustly map edits to line numbers causes mechanical failures

### Conclusion and Implication

- Hashline demonstrates massive performance gains through better engineering of the execution layer
- It proves that engineering the context can be as impactful as large model size increases

![Screenshot at 00:00: The video opens with an animated image of two podcasters in front of microphones, overlaid with a pulsing sound wave graphic and the text 'BECOME A MEMBER TODAY!', which is the prompt used for testing the LLMs' performance.](https://ss.rapidrecap.app/screens/kFtHOtKlb7Q/00-00-00.jpg)
![Screenshot at 00:44: A slide briefly appears with text related to the paper being discussed: 'I improved 15 LLMs at coding in one afternoon. Only the harness changed.'](https://ss.rapidrecap.app/screens/kFtHOtKlb7Q/00-00-44.jpg)
![Screenshot at 01:18: A visual representation of code structure is discussed, showing how the model struggles with nesting, brackets, and indentation.](https://ss.rapidrecap.app/screens/kFtHOtKlb7Q/00-01-18.jpg)
![Screenshot at 01:56: A chart or graphic is referenced showing the performance boost, noting GLM 4's failure rate dropping from 68% to 46.2%.](https://ss.rapidrecap.app/screens/kFtHOtKlb7Q/00-01-56.jpg)
![Screenshot at 08:23: A visual comparison is implied, contrasting the failure of standard code editing \(like deleting line 50\) versus the success of using Hashline's token IDs.](https://ss.rapidrecap.app/screens/kFtHOtKlb7Q/00-08-23.jpg)