Throughout my career, I’ve occasionally been tasked with identifying and fixing bizarre production bugs. I talk about those issues where logs provide little insight, and debugging is neither straightforward, practical, nor efficient.

The first few times I encountered such problems, I spent many hours reading code and trying to debug, only to get closer, but not close, to the source of the issue.

Surely, there had to be a better way, I thought.

If I could pinpoint when the bug was introduced, I could only focus on analysing a few code changes to understand the cause of the problem, thus significantly reducing total effort required.

In the sections that follow, I’ll share how I managed to improve this process and add an invaluable tool to my debugging toolkit.

The linear search

Let’s consider the following commit history as an example. Suppose the bolded line contains an elusive bug we’re trying to track down. We know the first commit (at the bottom) doesn’t have the bug, while the last commit (at the top) does.

4c1d8ea Add pagination support to API endpoints
e9f6b3d Refactor auth middleware for better readability
7df52ab Update Dockerfile to use newer Node.js version
2ab8f3c Add unit tests for payment gateway integration
b1a9e83 Remove deprecated `getUserData` function
f6a2dcd Improve error handling in order processing
9e2c47a Implement dark mode toggle in UI
d3cfa4b Adjust layout spacing for mobile viewports
5b8e1e7 Replace lodash with native JavaScript methods
a82f47c Update translations for Spanish locales
6d4f3b9 Add logging for failed login attempts

In order to identify the commit with the issue, I would have to go through every commit to determine whether the bug was present or not. If I started searching from the top, I’d need to inspect 9 commits to find the one that introduced the bug. Starting from the bottom would only require checking 2 commits, but that approach relied on either sheer luck to choose the right direction or extra information that was usually unavailable.

To make the process easier, I realised that having a quick method to confirm whether the bug was present or not could make all the difference. This didn’t need to be anything complex: a unit test, a script, an API call, or even a simple code hack could serve the purpose, as long as it was fast to execute and reliable.

However, even with this improvement, the process was still inefficient, especially for repositories with a large number of commits.

There had to be a better way, I thought again.

Welcome git bisect

One day, I stumbled across the missing piece of the puzzle.

It turns out that Git offers a very convenient feature called git bisect.

Using the divide and conquer algorithm, git bisect reduces the number of commits that we need to check to identify where a bug was introduced. The following article describes how you can use Divide and Conquer it in different scenarios.

Divide and conquer

Coming back to or Git history problem, let’s use the same example as before.

bbb Add pagination support to API endpoints
aaa Refactor auth middleware for better readability
999 Update Dockerfile to use newer Node.js version
888 Add unit tests for payment gateway integration
777 Remove deprecated `getUserData` function
666 Improve error handling in order processing
555 Implement dark mode toggle in UI
444 Adjust layout spacing for mobile viewports
333 Replace lodash with native JavaScript methods
222 Update translations for Spanish locales
111 Add logging for failed login attempts

First, we start the bisect mode, and then we tell Git the oldest commit we know to be bug-free and the newest commit where the bug is present.

$ git bisect start
$ git bisect bad bbb
$ git bisect good 111

While a closer range is better, there’s no need to spend excessive effort narrowing it down: doing so defeats the purpose of git bisect. Simply identify commits that are reasonably close and meet the criteria.

Git will now check out the commit halfway between the good and bad commits we specified.

555 Improve error handling in order processing

At this point, we need to determine whether this commit has the bug. If our check confirms it does, we tell Git by running:

$ git bisect bad # no need to specify the commit hash

Now, Git will check out the commit halfway between the good commit and this new bad commit. In this case, it will check out the commit with the bug:

333 Replace lodash with native JavaScript methods

When we run our check again, we’ll find that this commit also has the bug. In a real scenario we’ll just know that this is a bad commit, not the first bad commit that we are looking for. We need to continue until git bisects completes. So, we now have do mark this as a bad commit:

$ git bisect bad

Git will one more time check out the commit between that last known good commit and the last known bad commit:

222 Update translations for Spanish locales

After confirming that this commit is bug-free, we tell Git that this is a good commit:

git bisect good

Since there aren’t any commits left to check, Git is able to determine what is the commit that introduced the problem:

333 is the first bad commit
commit 3331e76b62f9dc735c670f29e9c1583bce06509
Author: Jose Miguel Armijo <jm@example.com>
Date:   Wed Dec 25 09:00:00 2024 +1100

    Replace lodash with native JavaScript methods

The Divide and Conquer algorithm has a time complexity of O(log₂(N)) with respect to the length of the commit history. In our case, we only needed to check 3 commits, which is very close to the theoretical 2.197 steps. This is just one step more than the best-case linear scenario and three times faster than the worst-case linear scenario.

Limitations

For git bisect to work effectively, every commit in the branch must be functional. This means that all tests pass, and that the code can execute. I give more insights about this In this article:

Clean Commits With Git

If there are commits where the code does not work, the tests don’t pass, the build crashes, or any work-in-progress mid-step, then git bisect may have false negatives. The script or test you use to validate the commit might fail not because the bug is present, but because someone merged incomplete code.

Another challenge arises when the feature being tested evolved so quickly that the testing logic no longer applies to every commit. A new argument in a function, moving code to another file, etc., may require constant updates to the test criteria. This can also lead to false positives if you don’t realise that the failure is due to changes in the code, not the bug you’re trying to identify.

---

In this article, I shared the true story of how I gradually became more effective at tracking down bugs using Git. While my learning process wasn’t the most efficient, it unfolded in a series of natural sequential steps that. I believe sharing the story as it truly occurred helps better understand the problem that we want to address, and how git bisect actually helps solving it.

Cheers!
JM

Share if you find this content useful, and Follow me on LinkedIn to be notified of new articles.