Why I tested four AI models on 85 real Paystack disputes in a fortnight
I sell skincare—shea butter blends, black soap, face oils—through Instagram and a simple Shopify storefront. Most customers pay via bank transfer or Paystack. I ship across Lagos and to a few states outside. In early February I received 85 Paystack dispute emails in twelve days. Not a hack, not a fraud ring—just the usual mix of "item didn't arrive", "wrong product", "I didn't authorize this", and one memorable "the packaging smelled like fish".
Paystack gives you seven days to respond to each dispute. Win the case and the funds stay in your account. Lose it and the money goes back to the customer, plus you've shipped product you'll never see again. I'd been using Kryotta's Compare feature since December to draft Instagram captions, testing Claude Sonnet 4.5 against Gemini Flash. I'd never used it for anything with money on the line. But 85 disputes meant I needed speed, clarity, and evidence-backed replies that Paystack's team would actually read. I decided to test four models—Claude, Gemini, Llama 3.3 70B, and GPT-OSS 120B—on every dispute type, then track which model's response actually won the case.
I'm not saying AI wrote these emails for me. I wrote them. But Compare let me see four drafts side-by-side, pick the structure and tone that felt right, and edit from there. Here's what I learned, broken into the six questions people asked when I posted about this on Twitter.
Which dispute types did you test, and how many of each?
I grouped the 85 disputes into four categories: delivery claims (38), product mismatch (22), unauthorized transaction (18), and quality complaints (7). Delivery claims were the largest group—"item never arrived" or "tracking shows delivered but I didn't receive it". Product mismatch was usually "I ordered the rose oil, you sent the lavender". Unauthorized transactions were almost always a customer's relative using their card without permission, then the customer filing a chargeback. Quality complaints ranged from reasonable ("the soap irritated my skin") to baffling ("this doesn't smell like shea butter should smell").
For each dispute I copied the customer's original message and Paystack's dispute reason into Kryotta's Compare arena, then ran the same prompt across all four models: "Draft a professional response to this Paystack dispute. Include: acknowledgment of the issue, evidence we have (tracking number, order confirmation, product photo if relevant), our resolution offer, and a clear request for Paystack to review the evidence. Tone: polite, factual, firm. 150 words maximum."
I read all four responses, picked the one with the clearest structure and the right balance of empathy and evidence, edited it lightly (usually tightening the opening line or adding a specific tracking detail), then sent it through Paystack's dispute portal. I logged which model I'd chosen and, ten days later, whether Paystack ruled in my favor.
What did each model do well, and where did each one fail?
Claude Sonnet 4.5 wrote the most human replies. It opened with a sentence that acknowledged the customer's frustration without admitting fault—"We understand how concerning it is when a delivery doesn't arrive as expected"—then moved straight to evidence. It won 31 out of 34 disputes I used it for, mostly delivery claims. The three losses were cases where the tracking genuinely showed "delivered" but the customer insisted they never received it, and Paystack sided with the customer. Claude's weakness: it occasionally added a sentence offering a replacement even when the dispute was clearly fraud, which I had to delete.
Gemini Flash was faster and cheaper, and it handled product mismatch disputes better than any other model. It listed evidence in a tight, scannable format—order number, SKU, photo timestamp—that made it easy for Paystack's team to verify. It won 18 out of 22 product mismatch cases. Its problem: the tone sometimes felt like a police report. One draft opened with "The customer's claim is inconsistent with our records", which is true but sounds combative. I rewrote those openings every time.
Llama 3.3 70B surprised me. I expected it to be the budget option, but it wrote the best responses for unauthorized transaction disputes—short, factual, and it always included a line like "We recommend the customer contact their bank to investigate this charge, as our records confirm the transaction was completed successfully on their end." It won 14 out of 16 unauthorized disputes. It struggled with quality complaints, though. It tried to argue with the customer's subjective experience ("Shea butter naturally has a nutty scent"), which is a losing move in a dispute.
GPT-OSS 120B was the wildcard. It wrote very polite, very long responses that often buried the evidence under two paragraphs of preamble. I used it for seven quality complaints because it was good at acknowledging the customer's concern without conceding liability, but I had to cut 40% of every draft. It won five out of seven. I didn't use it for any other dispute type after the first week.
How much time did Compare save you per dispute?
Before I started using Compare, I'd write each response from scratch—read the dispute email, open my Notes app, draft a reply, check my Shopify order details, revise the draft, paste it into Paystack. That took twelve to fifteen minutes per dispute, longer if I had to dig through WhatsApp to find the customer's original delivery instructions.
With Compare, I spent four to six minutes per dispute: thirty seconds to set up the prompt, one minute to read all four responses, two minutes to pick and edit one, and another minute to paste it into Paystack and attach evidence files. The time saving came from not staring at a blank page. Even when I rewrote 30% of a draft, having four starting points meant I never had to figure out the structure or the opening line. Over 85 disputes that saved me roughly nine hours across two weeks.
The cost in tokens was negligible. Compare runs your prompt across all four models at once, so you pay for four responses, but most of these were short—150 to 200 words. Total cost for 85 disputes was around ₦8,400, split across the models. Claude was the most expensive per response, Gemini and Llama the cheapest. I'm not optimizing for cost here; I'm optimizing for winning the dispute and keeping ₦2.8 million in my account.
Did any model consistently lose disputes, or was it always down to the evidence?
Evidence matters more than tone. If the tracking shows delivered and the customer says it didn't arrive, you'll lose that dispute more often than you win it, no matter how well you write the response. But tone and structure still matter, especially in edge cases where the evidence is ambiguous.
Llama lost two unauthorized disputes where it included a line suggesting the customer might have forgotten making the purchase. Paystack didn't like that—it sounded like I was accusing the customer of lying. I stopped using Llama for any dispute where the customer claimed they didn't recognize the charge.
GPT-OSS lost two quality complaints because its responses were so apologetic that they read like an admission of fault. One draft said, "We're sorry the product didn't meet your expectations—we'd be happy to offer a full refund." That's fine for customer service, but in a dispute it signals to Paystack that I agree the product was defective. I learned to delete any sentence with "we're sorry" unless I genuinely wanted to concede the case.
Claude and Gemini rarely lost a dispute unless the evidence was weak. The handful of cases I lost with those models were situations where I probably should have refunded the customer before they escalated to a chargeback—delivery delays over a week, tracking stuck at "in transit", that sort of thing.
Would you use Compare for disputes again, or just pick one model?
I'll keep using Compare for disputes, but I've built a shortcut. I now know Claude works best for delivery claims, Gemini for product mismatch, Llama for unauthorized transactions. For quality complaints I'll test Claude and GPT-OSS side-by-side, pick one, and edit hard. I won't bother running all four models unless the dispute is unusual or high-value—anything over ₦50,000.
The benefit of Compare isn't that one model is always better. It's that I can see four approaches in thirty seconds and pick the one that fits the situation. Some disputes need empathy up front; some need evidence first. Compare lets me choose based on the specific case, not a general rule. That's worth the extra token cost and the extra minute of reading time.
I also learned that the first draft is rarely the one I send. I'm editing 60–70% of every response, even the good ones. But I'm editing from a solid starting point, not from nothing. That's the difference between four minutes and fifteen.
What's your process now for handling Paystack disputes with AI?
I've settled into a workflow that takes four to five minutes per dispute. First, I read the Paystack email and check my Shopify order—tracking number, product SKU, delivery address. Then I open Kryotta's Compare, paste the dispute reason and the customer's message, and run my standard prompt (the one I listed earlier). I read all four responses, pick the one that matches the dispute type—Claude for delivery, Gemini for mismatch, Llama for unauthorized—and copy it into a Google Doc.
Then I edit. I rewrite the opening line if it's too stiff, I add the specific tracking number and order date, I attach a screenshot of the Shopify order confirmation, and I delete any sentence that sounds like I'm admitting fault or arguing with the customer. I paste the final version into Paystack's dispute portal, attach the evidence files, and submit. If the dispute is over ₦30,000 I'll also send a follow-up email to Paystack support with the same response and evidence, just to make sure someone reads it.
I log every dispute in a simple spreadsheet—date, amount, dispute type, model used, outcome. That's how I know Claude won 31 out of 34 and Llama won 14 out of 16. I'll keep tracking for another three months, then I'll probably stop. The pattern is clear enough now that I trust my shortcuts.
The one thing I won't do is automate this completely. I've seen people suggest using Auto routing to generate one response and send it straight to the customer. That might work for low-stakes queries, but for disputes where ₦20,000 or ₦50,000 is on the line, I want to read the response and make sure it's factual, polite, and doesn't concede anything I'm not willing to lose. Compare gives me speed without giving up control. That's the balance I need.
Questions people ask
Can I use Compare for Flutterwave disputes, or does this only work with Paystack?
The workflow works for any payment dispute—Flutterwave, Paystack, even bank chargebacks. The structure is the same: acknowledge the issue, present evidence, request a review. The only difference is where you submit the response. Flutterwave's dispute portal is less detailed than Paystack's, so you might need to send a follow-up email with evidence attachments. The AI response itself doesn't change.
Do I need to edit every response, or can I send some as-is?
I edited 60–70% of the responses I used. The edits were usually small—adding a tracking number, tightening the opening line, deleting a sentence that sounded too apologetic. I sent maybe a dozen responses with zero edits, all for straightforward delivery claims where Claude's draft was already tight and factual. If you're handling high-value disputes, read and edit every response. If it's a ₦5,000 claim and the evidence is solid, you can probably send it as-is.
Which model is cheapest for running 85 disputes through Compare?
Gemini Flash and Llama 3.3 70B are the cheapest per response. For 85 disputes I spent roughly ₦8,400 total across all four models in Compare. If you only ran Gemini or Llama, you'd spend closer to ₦3,000. But I wasn't optimizing for cost—I was optimizing for winning the dispute. Spending an extra ₦5,000 in tokens to keep ₦2.8 million in my account is an obvious trade.
What happens if Paystack rules against you even with a good response?
You lose the dispute, the funds go back to the customer, and you're out the product cost and the shipping. It happens. I lost nine out of 85 disputes, mostly cases where the tracking showed delivered but the customer insisted they never received it. Paystack tends to side with the customer in those situations unless you have a signature confirmation, which most Nigerian logistics companies don't offer. The AI response won't save you if the evidence is weak, but it does make sure you present the evidence you have as clearly as possible.
Compare won't win disputes for you, but it'll help you write responses that Paystack's team will actually read and act on. I've kept it in my workflow, and I've started using it for refund requests that come through WhatsApp before they escalate to disputes. If you're handling payment claims for an online business—whether you're selling skincare, clothes, or phone accessories—it's worth testing. You can try Kryotta's Compare feature at app.kryotta.ai/auth and see which model fits your dispute types. I'd start with five disputes, not eighty-five.
