I used AI code review on 50 pull requests: which model caught the actual bugs

I tested Claude, Gemini, DeepSeek, and GPT-OSS on fifty real pull requests from Nairobi startups. Here's which models actually caught production-breaking bugs versus just complaining about formatting.

KKryotta TeamProduct & research · · 9 min read
Developer reviewing code on laptop screen with hand gesturing at the display, warm office lighting
Developer reviewing code on laptop screen with hand gesturing at the display, warm office lighting

Does AI actually catch bugs, or just complain about formatting?

I review code for three Nairobi startups. One builds M-Pesa reconciliation tools for chamas, another runs a WhatsApp Business API for boda delivery coordination, and the third sells beauty products on Instagram with a custom checkout backend. Over two weeks, I fed fifty pull requests into Claude Sonnet 4.5, Gemini Flash, DeepSeek R1, and GPT-OSS 120B. I wanted to know which model spotted the bugs that would break production, and which ones just nagged me about missing semicolons.

Claude Sonnet 4.5 caught twelve logic errors across those fifty PRs. Real problems: a race condition in the M-Pesa callback handler that would've marked payments as failed when Safaricom's server was slow, an off-by-one error in a delivery radius calculator, and a function that returned null instead of an empty array when no orders matched. Gemini Flash found seven of those twelve. DeepSeek R1 found nine, but took three times longer and wrote paragraphs explaining the philosophy of error handling when I just needed to know the line number. GPT-OSS 120B caught four logic bugs and spent the rest of its energy suggesting I rename variables.

The surprise: none of them caught everything. The race condition? Only Claude saw it. The null-versus-empty-array bug? DeepSeek and Gemini both flagged it; Claude missed it the first time. A security issue where user input wasn't sanitized before hitting the database? Claude and DeepSeek both shouted. Gemini mentioned it as an afterthought. GPT-OSS said nothing.

What does "catching a bug" actually look like in the review?

I'm not talking about style complaints. Every model will tell you to add comments or break a hundred-line function into smaller pieces. That's useful, but it won't crash your server at 2 a.m. when a customer in Mombasa tries to pay for airtime top-ups through your app.

A real bug is something that compiles, passes your basic tests, and still does the wrong thing under load or with unexpected input. The M-Pesa race condition looked fine in local testing because my laptop's callback response came back in forty milliseconds. In production, Safaricom sometimes takes two seconds, and the code assumed that if the response wasn't instant, the payment had failed. Claude's review said: "This timeout logic will reject valid payments when the M-Pesa API is under load. Consider increasing the threshold to five seconds and adding retry logic." That one comment saved us from refunding thirty transactions a day.

Gemini Flash caught a different class of error: data shape mismatches. We had a function that expected delivery coordinates as { lat: number, lng: number } but the WhatsApp webhook sometimes sent { latitude: string, longitude: string }. The code didn't crash; it just placed every delivery at 0,0 in the Atlantic Ocean. Flash said: "Input validation missing—latitude and longitude are strings here, but your distance calculation expects numbers." Claude didn't mention it. DeepSeek wrote five paragraphs about TypeScript's type system and eventually got to the same point.

Which model is fastest when you're reviewing ten PRs before lunch?

Speed matters when you're a solo dev or a small team without a dedicated QA engineer. I timed each model on the same 180-line pull request: a new endpoint for splitting a Jumia-style order into multiple deliveries based on product weight.

Gemini Flash returned a review in four seconds. It caught the obvious stuff—missing null checks, a hardcoded API key that should've been in the environment file—and flagged two logic issues. One was real (the weight calculation used grams when the API expected kilograms), the other was a false alarm (it thought a loop would run forever when it actually had a break condition three lines down).

Claude Sonnet 4.5 took eleven seconds and found everything Flash did, plus a subtle bug where the delivery fee wasn't recalculated after the order was split. That bug wouldn't break anything; it would just undercharge customers by KSh 50 on multi-package orders. Accounting would've noticed it in a month.

DeepSeek R1 took thirty-two seconds and wrote a review longer than the code. It found the delivery fee bug and explained the compound interest implications of undercharging at scale, which was interesting but not helpful at 9 a.m. when I had nine more PRs in the queue.

GPT-OSS 120B took seven seconds and mostly suggested renaming calc_fee to calculate_delivery_fee. It did catch the hardcoded API key. It missed both logic bugs.

If I'm reviewing code I didn't write and don't fully trust, I use Claude. If I wrote it myself and just want a second pair of eyes before merging, Flash is faster and cheap enough that I don't think about the cost.

How much does this actually cost per review?

I tracked token usage across all fifty reviews. The average PR was 160 lines of Python or JavaScript. I included the diff, a two-sentence description of what the code was supposed to do, and the instruction: "Review this pull request for logic errors, security issues, and edge cases. Ignore style unless it affects correctness."

Claude Sonnet 4.5 used about 8,000 tokens per review (input plus output). At Kryotta's pricing, that's roughly KSh 4 per PR. Across fifty reviews: KSh 200. For a month of daily code review, budget KSh 600.

Gemini Flash used 6,500 tokens per review, about KSh 0.80 each. Fifty reviews cost KSh 40. It's fast, cheap, and good enough for most pulls.

DeepSeek R1 used 11,000 tokens because it writes essays. KSh 1.10 per review, KSh 55 total. Cheaper than Claude, but the verbose output means you're paying for words you'll skim past.

GPT-OSS 120B used 7,200 tokens, around KSh 1.50 per review. Middle of the pack on price, bottom of the pack on usefulness.

The real cost isn't the tokens. It's the production bug you didn't catch. One race condition in the M-Pesa handler could mean fifty failed payments and two hours on the phone with angry customers. Claude's KSh 4 review saved that. Flash's KSh 0.80 review missed it.

What's the Compare workflow that catches what one model misses?

After watching Claude miss the null-versus-empty-array bug and Gemini miss the race condition, I started running critical PRs through two models at once. Kryotta's Compare arena lets you send the same prompt to multiple models and see both answers side by side.

Here's the workflow: paste the PR diff into Compare, select Claude Sonnet 4.5 and DeepSeek R1, and run the review. Claude gives you the fast, practical list of issues. DeepSeek gives you the paranoid, overthinking list. Where they agree, the bug is real. Where only one flags it, you read the code again yourself.

I used this on fifteen of the fifty PRs—the ones touching payment logic, user authentication, or anything that handles money. It caught two bugs that both models individually missed: Claude spotted them in the second review after I'd edited the code based on DeepSeek's feedback, and DeepSeek noticed a SQL injection risk that Claude had marked as low-priority.

The Compare workflow adds thirty seconds and doubles the token cost, but I only use it for code that can't fail. A product description generator for an Instagram shop? One model is fine, and Flash is fast enough. A webhook that reconciles M-Pesa payments for twenty chamas? Two models, and I don't care if it costs KSh 8 instead of KSh 4.

Should I trust AI review enough to skip human review?

No. I caught three bugs across those fifty PRs that none of the models mentioned. One was a logic error so specific to our business rules that the AI had no way to know it was wrong: we give free delivery to customers in Nairobi's CBD, but the code checked the customer's billing address instead of the delivery address. Someone could live in Rongai, ship to their office in town, and still pay delivery fees. The AI models all said the code looked fine because, syntactically, it was.

Another was a performance issue. The code worked, but it loaded every product image into memory before resizing them for WhatsApp catalogue uploads. On a product list with two hundred items, the server ran out of RAM. Claude mentioned that the function "might be slow with large datasets," but it didn't flag it as a bug. A human reviewer who'd seen the server crash before knew to check memory usage.

The third was a missed opportunity, not a bug. The code calculated delivery fees using straight-line distance, but Nairobi traffic means a ten-kilometre trip can take an hour or fifteen minutes depending on the route. A model trained on global code patterns doesn't know that Mombasa Road is faster than Thika Road at 8 a.m., and it won't suggest using a routing API that accounts for real travel time. A developer who's sat in that traffic will.

Use AI review as the first pass. It's faster than waiting for a colleague to free up, and it catches the obvious stuff—null pointer risks, unhandled exceptions, off-by-one errors. But the final decision is still yours. If the AI says the code is fine and you have a bad feeling, trust the bad feeling.

Questions people ask

Which model should I use if I can only pick one?
Claude Sonnet 4.5. It's not the cheapest or the fastest, but it catches the bugs that cost you money. If you're reviewing code once a day, the KSh 600 monthly cost is less than one hour of your time spent debugging a production issue.

Does AI review work for languages other than Python and JavaScript?
Yes. I tested it on Go, PHP, and Dart (for a Flutter app). Claude and DeepSeek handled all of them well. Gemini Flash was slightly weaker on Dart but still useful. GPT-OSS struggled with Go and gave generic advice.

Can I use AI to review my own code, or does that just confirm my mistakes?
It works, but you'll miss some of your own assumptions. The best use is reviewing code you wrote yesterday or last week, when you've forgotten the details. Fresh eyes—human or AI—catch more than reviewing immediately after you write it.

What's the biggest waste of tokens in AI code review?
Asking the model to review the entire codebase instead of just the changed lines. If your PR touches three files and twenty lines, send those twenty lines. Don't include the whole module "for context" unless the bug is about how the new code interacts with the old. I wasted KSh 150 in the first week doing this before I learned to trim the input.


I still spend ten minutes a day reading code that AI reviewed. But I spend two hours less per week chasing bugs that slipped through. If you're building anything that touches payments, user data, or automated workflows, try running your next five pull requests through Claude or DeepSeek in Compare mode at Kryotta. You'll catch something you would've missed, and you'll know which model actually earns its tokens.

K
Written by
Kryotta Team
Product & research

Kryotta is the multi-model AI workspace — every leading model, one login, one bill. Try it free →

Related reading