United Statescountry
DevelopingResearch

George Washington University physicists publish formula to predict when AI chatbots produce harmful output

Physicists Neil Johnson and Frank Yingjie Huo published a formula estimating how many good responses an AI chatbot gives before turning harmful, with early tests showing predictive accuracy.

Why it matters

The research offers a potential method to predict harmful outputs from AI chatbots without requiring cloud-based safety tools, which could improve safety for offline models.

Key facts

Physicists Neil Johnson and Frank Yingjie Huo at George Washington University published a formula estimating how many good tokens an AI model produces before its first bad one

Reported by Decrypt

The study appeared in the journal Patterns and builds on a preprint first released in February

Reported by Decrypt

In the preprint the formula correctly predicted whether a model would tip immediately or after a delay in 15 of 16 clear-cut cases

Reported by Decrypt

The authors propose a parallel monitor that flags when models fall below a safety threshold

Reported by Decrypt

The researchers trace the problem to the attention head which decides which earlier words matter most when choosing the next one

Reported by Decrypt

Existing safety tools often depend on a cloud connection that offline models lack

Reported by Decrypt

What happened

Physicists Neil Johnson and Frank Yingjie Huo at George Washington University published a formula in the journal Patterns that estimates how many good tokens an AI model produces before its first bad one. The preprint version was first released in February. In testing, the formula correctly predicted whether a model would tip immediately or after a delay in 15 of 16 clear-cut cases. The authors propose a parallel monitor that flags when models fall below a safety threshold. They trace the problem to the attention head, which decides which earlier words matter most when choosing the next one, and note that accumulated context pulls attention toward one cluster of answers until it tips. Existing safety tools often depend on cloud connections that offline models lack.

How the story developed

  1. First report

    First report by Decrypt

    Here’s a Way to Predict When AI Chatbots Will Turn Bad

    Decrypt

  2. Update

    Preprint first released

    Preprint version of the study first released in February

    Decrypt

  3. Official statement

    Study published in Patterns journal

    The study by Neil Johnson and Frank Yingjie Huo appeared in the journal Patterns

    Decrypt

Affected entities

Original sources

PublisherReportRolePublished
Decrypt
Crypto media
Here’s a Way to Predict When AI Chatbots Will Turn BadOriginal