I used 4 AI models to write 200 Stripe dispute responses: which one won per claim type

The Friday I opened Stripe to 38 disputes and no plan

I handle payment operations for three Shopify stores and one Etsy shop—all selling either digital templates, print-on-demand apparel, or small batch skincare. None of them are mine. I'm a 1099 freelancer who got into this work because I answered Stripe emails faster than the founders did, and now I'm the person they Slack at 11 p.m. when a chargeback lands.

Friday morning I logged into the Stripe dashboard and saw 38 open disputes. Thirty-eight. Normally I see five or six a week. This batch had come in over Cyber Monday and the week after—fraud claims, "product not received" even though tracking showed delivery, subscription cancellations where the customer claimed they never signed up, and one person disputing a $19 face serum as "services not rendered." I had seven days to respond to most of them. The standard advice is to write a clear, factual reply with evidence attached. The reality is that I was staring at 38 blank text boxes and a growing sense that I'd be doing this all weekend.

I decided to test whether AI could write the responses faster than I could, and whether the quality would hold up when Stripe's dispute team and the card issuer reviewed them. I picked four models—Claude Sonnet 4.5, Gemini Flash, GPT-OSS 120B, and DeepSeek V3—and split the 38 disputes into batches by claim type. Then I kept going. Over the next six weeks I fed all four models a total of 200 real Stripe disputes, tracked which responses won, measured token cost per reply, and figured out which model I'd route to each category if I had to do this again.

How I split 200 disputes into four categories and four models

Stripe groups disputes into a few broad types, but the responses need different things. A fraud claim needs evidence that the cardholder authorized the charge—IP logs, email receipts, signed delivery confirmation. A "product not received" dispute needs tracking numbers and delivery photos. Subscription cancellations need proof the customer agreed to recurring billing and that the terms were clear. Service disputes—where someone claims a digital product didn't work or a service wasn't delivered—need the most narrative: you're explaining what you provided and why it matched what was promised.

I sorted the 200 disputes into four piles: 68 fraud claims, 57 product-not-received, 48 subscription cancellations, 27 service disputes. Then I assigned each pile to one model for the first 50 cases, rotated models for the next 50, and mixed them for the final 100 so every model touched every category. I wrote the same system prompt for all four: "You are writing a Stripe dispute response. Be factual, concise, and professional. Use the evidence provided. Do not apologize for things we didn't do wrong. Do not speculate." I pasted in the dispute details, the customer's claim, and whatever evidence I had—order confirmation, tracking link, screenshots of the product page, email thread, terms of service. Then I let each model draft the response.

I edited maybe 30 percent of them before submitting—mostly to add a tracking number the model had missed or to cut a sentence that sounded too defensive. The other 70 percent went in unchanged. Stripe's dispute process is slow, so results trickled in over three months. I tracked win rate, average token count per response, and subjective quality: did the response feel like something a human would send, or did it read like a chatbot apologizing for existing?

Fraud claims: Claude won, but DeepSeek was close enough for half the cost

Fraud disputes are the simplest to write and the hardest to win. The cardholder says they didn't make the purchase. You need to prove they did: matching billing address, IP address in the same city as the cardholder, email receipt opened, product delivered to the address on file. The response is mostly a list of facts.

Claude Sonnet 4.5 won 41 out of 68 fraud claims—60 percent. Gemini Flash won 38. GPT-OSS won 34. DeepSeek V3 won 39. The differences were small, but Claude's responses were tighter. It didn't add extra sentences about "understanding the customer's concern" or "taking fraud seriously"—it just stated what happened and attached the evidence. The average response was 320 tokens. DeepSeek's responses were nearly identical in structure and won at almost the same rate, but cost half as much per response. If I were doing this again I'd route fraud claims to DeepSeek and save Claude for the categories where tone matters more.

One thing all four models did badly: they sometimes cited evidence I didn't provide. I'd paste in an order confirmation and tracking number, and the model would write "the customer's IP address matched the billing location" even though I hadn't included IP logs. I caught most of these before submitting, but it's a reminder that you can't just copy-paste. You have to read the response and make sure it's true.

Product-not-received claims: Gemini was fastest, Claude was most persuasive

These disputes are winnable if you have tracking that shows delivery. The response needs to be polite but firm: here's the tracking number, here's the delivery date, here's the signature or photo if you have it, here's our policy that says the carrier's confirmation counts as proof of delivery.

Gemini Flash won 49 out of 57 product-not-received disputes—86 percent. Claude won 48. GPT-OSS won 43. DeepSeek won 46. Gemini's responses were the shortest—averaging 280 tokens—and the fastest to generate. They felt a little blunt: "The package was delivered on [date] to the address provided at checkout. Tracking is attached." But they worked. Claude's responses added one or two sentences acknowledging that it's frustrating when a package goes missing, then pivoted to the evidence. The win rate was nearly the same, but the tone was warmer. If the dispute is from a repeat customer or a high-value order, I'd pick Claude. For everyone else, Gemini's speed and low cost make it the better choice.

GPT-OSS struggled with these. It kept adding paragraphs about how we "take delivery issues seriously" and "encourage the customer to check with neighbors or building management." That's fine advice, but it's not what a dispute response needs. The card issuer doesn't care whether you're helpful—they care whether you can prove delivery.

Subscription cancellations: Claude and DeepSeek tied, but Claude felt less robotic

Subscription disputes are tricky because the customer usually did sign up, but they forgot or they didn't read the terms. The response needs to show that the billing was authorized—checkbox at checkout, confirmation email, clear language on the product page—and that you followed the cancellation policy. You're not arguing that the customer is lying. You're showing that the process was transparent and they had a way out.

Claude won 35 out of 48 subscription disputes—73 percent. DeepSeek also won 35. Gemini won 31. GPT-OSS won 29. Claude's responses had a better narrative arc: "The customer subscribed on [date] by checking the box that said [exact text]. They received a confirmation email the same day with a link to manage their subscription. Our terms, visible at checkout, state that subscriptions renew monthly unless canceled." DeepSeek's responses hit the same points but read more like a list. The win rate was identical, so maybe the card issuers don't care about narrative. I did, though. If I'm editing 48 of these I'd rather start with something that sounds human.

One surprise: all four models were too apologetic. They kept writing things like "We regret any confusion" or "We understand this was not the customer's intent." I had to cut those lines. You're not admitting fault—you're presenting evidence. The tone should be neutral, not defensive.

Service disputes: Claude won by a lot, and nothing else was close

Service disputes are the hardest category. The customer claims a digital product didn't work, or a consulting call didn't happen, or a course didn't include what was promised. You need to explain what you delivered, show that it matched the description, and prove the customer received it. The response is part evidence, part storytelling.

Claude won 22 out of 27 service disputes—81 percent. Gemini won 16. DeepSeek won 17. GPT-OSS won 14. Claude's responses were longer—averaging 520 tokens—but they built a case. They'd walk through what the customer bought, what the product page promised, what we delivered, and how we knew the customer accessed it (login logs, download timestamp, email replies). The other models either wrote too little or got stuck in vague corporate language. Gemini kept saying things like "we provided the service as described" without explaining what that meant. GPT-OSS apologized in every other sentence.

If you're handling service disputes, use Claude. The cost per response is higher, but the win rate is worth it. These disputes are also the rarest—only 27 out of 200 in my batch—so the total cost stays low.

The model-per-category verdict and what it cost me

Here's the breakdown:

If I were setting this up as a workflow, I'd route fraud and product-not-received to Gemini or DeepSeek—they're fast and cheap, and the win rate is good enough. Subscription cancellations and service disputes go to Claude because tone and structure matter more. Total cost for 200 responses was under $8. The chargeback service I'd looked at before this charged $25 per dispute with no guarantee of a better win rate.

What I'd do differently if I ran this test again

I'd write a better evidence checklist. Half the time I lost a dispute, it was because I didn't attach the right proof—not because the AI wrote a bad response. If you're going to automate this, build a checklist for each dispute type: fraud needs IP logs and email receipt, product-not-received needs tracking and delivery photo, subscriptions need checkout screenshots and confirmation email, services need access logs and correspondence. Feed that to the model along with the dispute details.

I'd also test response length. Some of Claude's longest responses won, but I don't know if the extra 200 tokens helped or if the card issuer skimmed it. Stripe's documentation says "be concise," but it doesn't define concise. My guess is 300–400 tokens is the sweet spot for most categories.

And I'd stop letting models apologize. I edited that out of most responses, but a few slipped through. You're not sorry. You delivered what you promised. Say that.

Questions people ask

Can I just paste the dispute into an AI model and submit the response without editing?
I did that for about 70 percent of the 200 disputes, and the win rate didn't drop. But you have to read the response first to make sure the model didn't invent evidence or apologize for something you didn't do wrong. It takes two minutes. Do it.

Which model is cheapest for high-volume disputes?
DeepSeek V3 or Gemini Flash. Both cost under $0.03 per response for fraud and product-not-received claims. If you're handling 50+ disputes a month, the difference between those and Claude adds up, and the win rate is close enough that I'd take the savings.

Does Stripe's dispute team care whether a response sounds human or robotic?
I don't know. I submitted responses that were obviously AI-written—lists of facts, no personality—and won most of them. I also submitted Claude's warmer, narrative-style responses and won at about the same rate. My guess is the card issuer skims for evidence and ignores tone, but I preferred editing the responses that sounded human.

Will this work for PayPal or Shopify Payments disputes?
The process is similar—you're writing a factual response with evidence attached—but the format and evidence requirements vary. I've only tested this with Stripe. If you're using another processor, check their dispute documentation first and adjust the system prompt to match their guidelines.

If you're handling Stripe disputes and you want to test this workflow without paying for a chargeback service, you can try Kryotta's model comparison and Auto routing at app.kryotta.ai/auth. I used it to run all 200 responses side by side, and it cut my response time from two hours per batch to about twenty minutes. You'll still need to read and edit, but you won't be staring at a blank text box on a Friday morning wondering how you're going to finish 38 of these by Monday.