Coinbaseexchange
DevelopingResearch

Coinbase finds newer AI models detect less payment fraud in historical benchmark

Coinbase's historical replay test showed newer versions of three AI model families caught fewer fraudulent payments despite unchanged decision policy.

Why it matters

The findings challenge the assumption that upgrading AI models automatically improves fraud detection in payment screening systems. This has implications for how companies evaluate and deploy model updates in financial security contexts.

Key facts

Coinbase reported on October 7 that newer versions of three major AI model families caught fewer fraudulent payments in a historical test of payment screening for its Onramp service

Reported by CryptoSlate

The evaluation replayed 16,140 transactions across 7,293 users including 813 confirmed fraudulent transactions

16,140 transactionsReported by CryptoSlate

Coinbase compared Opus 4.5 with Opus 5, Sonnet 4.6 with Sonnet 5, and GPT-5.4 with GPT-5.6 (sol)

Reported by CryptoSlate

Every newer model version had lower recall and a lower combined precision-and-recall score under a fixed decision policy

Reported by CryptoSlate

GPT's precision improved while fraud coverage declined

Reported by CryptoSlate

Separate custom-model and production-latency tests showed gains

Reported by CryptoSlate

Neither the historical replay nor custom-model tests measured customer losses from the tested model upgrades

Reported by CryptoSlate

What happened

CryptoSlate reported that Coinbase conducted a historical replay evaluation of its Onramp payment screening system on October 7, testing three major AI model family upgrades. The benchmark replayed 16,140 transactions across 7,293 users including 813 confirmed fraud cases from a nine-week period before the risk agent launched. Coinbase compared Opus 4.5 against Opus 5, Sonnet 4.6 against Sonnet 5, and GPT-5.4 against GPT-5.6 (sol), finding every newer version had lower recall and a lower combined precision-and-recall score under a fixed decision policy. While GPT's precision improved, its fraud coverage declined. Separate custom-model and production-latency tests showed gains, though neither measured actual customer losses from the model upgrades.

How the story developed

  1. Update

    Coinbase publishes benchmark results

    Coinbase reported findings from historical replay evaluation of AI model upgrades for Onramp payment screening

    CryptoSlate

  2. First report

    First report by CryptoSlate

    Newer AI models missed more payment fraud in Coinbase’s benchmark

    CryptoSlate

  3. Update

    CryptoSlate covers the findings

    CryptoSlate published article detailing Coinbase's benchmark results showing newer AI models detected less fraud

    CryptoSlate

Affected entities

Original sources

PublisherReportRolePublished
CryptoSlate
Crypto media
Newer AI models missed more payment fraud in Coinbase’s benchmarkOriginal