George Washington University physicists publish formula to predict when AI chatbots produce harmful output
Physicists Neil Johnson and Frank Yingjie Huo published a formula estimating how many good responses an AI chatbot gives before turning harmful, with early tests showing predictive accuracy.
Why it matters
The research offers a potential method to predict harmful outputs from AI chatbots without requiring cloud-based safety tools, which could improve safety for offline models.
Key facts
Physicists Neil Johnson and Frank Yingjie Huo at George Washington University published a formula estimating how many good tokens an AI model produces before its first bad one
Reported by Decrypt
The study appeared in the journal Patterns and builds on a preprint first released in February
Reported by Decrypt
In the preprint the formula correctly predicted whether a model would tip immediately or after a delay in 15 of 16 clear-cut cases
Reported by Decrypt
The authors propose a parallel monitor that flags when models fall below a safety threshold
Reported by Decrypt
The researchers trace the problem to the attention head which decides which earlier words matter most when choosing the next one
Reported by Decrypt
Existing safety tools often depend on a cloud connection that offline models lack
Reported by Decrypt
What happened
Physicists Neil Johnson and Frank Yingjie Huo at George Washington University published a formula in the journal Patterns that estimates how many good tokens an AI model produces before its first bad one. The preprint version was first released in February. In testing, the formula correctly predicted whether a model would tip immediately or after a delay in 15 of 16 clear-cut cases. The authors propose a parallel monitor that flags when models fall below a safety threshold. They trace the problem to the attention head, which decides which earlier words matter most when choosing the next one, and note that accumulated context pulls attention toward one cluster of answers until it tips. Existing safety tools often depend on cloud connections that offline models lack.
How the story developed
- First report
First report by Decrypt
Here’s a Way to Predict When AI Chatbots Will Turn Bad
- Update
Preprint first released
Preprint version of the study first released in February
- Official statement
Study published in Patterns journal
The study by Neil Johnson and Frank Yingjie Huo appeared in the journal Patterns
Affected entities
Original sources
| Publisher | Report | Role | Published |
|---|---|---|---|
| Decrypt Crypto media | Here’s a Way to Predict When AI Chatbots Will Turn Bad | Original |