Coinbase finds newer AI models detect less payment fraud in historical benchmark
Coinbase's historical replay test showed newer versions of three AI model families caught fewer fraudulent payments despite unchanged decision policy.
Why it matters
The findings challenge the assumption that upgrading AI models automatically improves fraud detection in payment screening systems. This has implications for how companies evaluate and deploy model updates in financial security contexts.
Key facts
Coinbase reported on October 7 that newer versions of three major AI model families caught fewer fraudulent payments in a historical test of payment screening for its Onramp service
Reported by CryptoSlate
The evaluation replayed 16,140 transactions across 7,293 users including 813 confirmed fraudulent transactions
16,140 transactionsReported by CryptoSlate
Coinbase compared Opus 4.5 with Opus 5, Sonnet 4.6 with Sonnet 5, and GPT-5.4 with GPT-5.6 (sol)
Reported by CryptoSlate
Every newer model version had lower recall and a lower combined precision-and-recall score under a fixed decision policy
Reported by CryptoSlate
GPT's precision improved while fraud coverage declined
Reported by CryptoSlate
Separate custom-model and production-latency tests showed gains
Reported by CryptoSlate
Neither the historical replay nor custom-model tests measured customer losses from the tested model upgrades
Reported by CryptoSlate
What happened
CryptoSlate reported that Coinbase conducted a historical replay evaluation of its Onramp payment screening system on October 7, testing three major AI model family upgrades. The benchmark replayed 16,140 transactions across 7,293 users including 813 confirmed fraud cases from a nine-week period before the risk agent launched. Coinbase compared Opus 4.5 against Opus 5, Sonnet 4.6 against Sonnet 5, and GPT-5.4 against GPT-5.6 (sol), finding every newer version had lower recall and a lower combined precision-and-recall score under a fixed decision policy. While GPT's precision improved, its fraud coverage declined. Separate custom-model and production-latency tests showed gains, though neither measured actual customer losses from the model upgrades.
How the story developed
- Update
Coinbase publishes benchmark results
Coinbase reported findings from historical replay evaluation of AI model upgrades for Onramp payment screening
- First report
First report by CryptoSlate
Newer AI models missed more payment fraud in Coinbase’s benchmark
- Update
CryptoSlate covers the findings
CryptoSlate published article detailing Coinbase's benchmark results showing newer AI models detected less fraud
Affected entities
Original sources
| Publisher | Report | Role | Published |
|---|---|---|---|
| CryptoSlate Crypto media | Newer AI models missed more payment fraud in Coinbase’s benchmark | Original |