Coinbase recently conducted a study that revealed surprising results regarding the effectiveness of newer versions of AI models in detecting fraudulent payments. The study, which focused on the company’s Onramp service, compared the performance of three major AI model families – Opus, Sonnet, and GPT – in catching fraudulent transactions.
In the evaluation, Coinbase replayed over 16,000 transactions across a period of nine weeks, including both legitimate and confirmed fraudulent transactions. The study aimed to determine whether upgrading to newer versions of AI models would improve the existing payment screening process.
The results of the study were unexpected. Despite using the same decision policy for all models, the newer versions of Opus, Sonnet, and GPT performed poorly compared to their older counterparts. The newer models had lower recall rates, lower combined precision-and-recall scores (F1), and lower dollar-weighted recall rates. This means that the newer models caught fewer fraudulent transactions and a smaller share of the fraud value.
For example, Sonnet’s recall rate dropped significantly, as did its dollar-weighted recall rate. Both Opus and GPT also showed a decrease in recall rates, precision rates, and dollar-weighted recall rates. While GPT’s precision rate improved, its recall and dollar-weighted recall rates decreased, indicating that more fraud cases and value escaped detection with the newer version.
It is important to note that the study did not establish customer losses resulting from deploying the newer versions of the AI models. Coinbase also mentioned that they were able to identify the regressions in performance without pinpointing the exact cause.
In a separate case, Coinbase reported positive results with a custom model called Qwen3.5-9B. This post-trained model outperformed Opus 4.5 in fraud-detection metrics, with significant improvements in F1 and dollar-weighted recall rates. The company attributed the success of this custom model to its specialization using historical fraud outcomes and deterministic rewards.
Overall, the study highlights the importance of testing new AI models in a real-world decision setup before implementation. Coinbase recommends evaluating changes in prompts or thresholds separately and considering factors like latency, reliability, and cost alongside detection quality. The findings challenge the assumption that upgrading to newer AI models will always improve fraud detection and emphasize the need for thorough testing and evaluation before making changes to existing systems.
