Researchers from UIUC have revealed a concerning vulnerability in OpenAI’s latest voice-enabled model, ChatGPT-4o, which can be exploited for scams. This advanced AI, designed to handle text, voice, and even vision inputs, is now being used by fraudsters to conduct scams, including bank transfers, credential theft, and crypto swindles. Despite OpenAI’s efforts to include safeguards, these features offer criminals a low-cost way to manipulate the model’s output, highlighting the risks that accompany such powerful technology.
Voice scams are a multimillion-dollar issue globally, and the rise of deepfake tech and AI-driven text-to-speech tools makes the situation more complex. UIUC researchers Richard Fang, Dylan Bowman, and Daniel Kang tested ChatGPT-4o’s capabilities, proving that without enough restrictions, the AI could navigate sites, input data, and bypass two-factor authentication. The team used prompt-jailbreaking tactics to bypass content limitations, simulating typical scam scenarios by posing as naive victims. For example, they replicated bank transfers on actual financial sites like Bank of America, ensuring transactions went through, though they avoided testing the agents’ ability to persuade.
In this study, various scams were evaluated, each with unique success rates and requirements. The AI agents successfully managed sensitive tasks like credential theft, crypto transfers, and even manipulating social media accounts. Success rates varied: credential theft from Gmail reached a high of 60%, while crypto transfers and Instagram account breaches had a success rate of 40%. The most complex scams involved up to 26 steps, averaging three minutes per attempt. Costs were low, with a typical successful attempt under a dollar. Even the more complex bank transfers averaged $2.51—still affordable considering the profit potential.
OpenAI has responded with updated safeguards in its new “o1” model, which is currently in preview. This model features enhanced reasoning capabilities and scores significantly higher in resisting adversarial prompts compared to GPT-4o. In internal safety evaluations, “o1” outperformed earlier models, scoring 93% on safety tests versus GPT-4o’s 71%. To prevent impersonation, OpenAI restricts voice outputs to pre-approved voices. They acknowledge that insights from studies like UIUC’s help them identify gaps and strengthen protections, although challenges persist.
For now, OpenAI plans to phase out older models while prioritizing safety and robustness. However, the broader AI landscape still faces risks as other voice-enabled bots, often with less stringent safeguards, can be used in similar ways. This study highlights the growing need for responsible AI development, especially with tools that have significant impact potential. With AI’s evolving role, the balance between advancing capabilities and ensuring security will continue to shape the industry’s future.
Image: DIW-AIgen
Read next:
• AI Hiring Exposed: White Male Names Dominate While Black and Female Candidates Are Overlooked!
• Search Engines Top Consumer Preferences; Social Media, AI Tools Gain Influence
• Digital Marketers Under Pressure: 57% Feel Urgency to Learn AI Skills as Adoption Surges to 74%
Voice scams are a multimillion-dollar issue globally, and the rise of deepfake tech and AI-driven text-to-speech tools makes the situation more complex. UIUC researchers Richard Fang, Dylan Bowman, and Daniel Kang tested ChatGPT-4o’s capabilities, proving that without enough restrictions, the AI could navigate sites, input data, and bypass two-factor authentication. The team used prompt-jailbreaking tactics to bypass content limitations, simulating typical scam scenarios by posing as naive victims. For example, they replicated bank transfers on actual financial sites like Bank of America, ensuring transactions went through, though they avoided testing the agents’ ability to persuade.
In this study, various scams were evaluated, each with unique success rates and requirements. The AI agents successfully managed sensitive tasks like credential theft, crypto transfers, and even manipulating social media accounts. Success rates varied: credential theft from Gmail reached a high of 60%, while crypto transfers and Instagram account breaches had a success rate of 40%. The most complex scams involved up to 26 steps, averaging three minutes per attempt. Costs were low, with a typical successful attempt under a dollar. Even the more complex bank transfers averaged $2.51—still affordable considering the profit potential.
OpenAI has responded with updated safeguards in its new “o1” model, which is currently in preview. This model features enhanced reasoning capabilities and scores significantly higher in resisting adversarial prompts compared to GPT-4o. In internal safety evaluations, “o1” outperformed earlier models, scoring 93% on safety tests versus GPT-4o’s 71%. To prevent impersonation, OpenAI restricts voice outputs to pre-approved voices. They acknowledge that insights from studies like UIUC’s help them identify gaps and strengthen protections, although challenges persist.
For now, OpenAI plans to phase out older models while prioritizing safety and robustness. However, the broader AI landscape still faces risks as other voice-enabled bots, often with less stringent safeguards, can be used in similar ways. This study highlights the growing need for responsible AI development, especially with tools that have significant impact potential. With AI’s evolving role, the balance between advancing capabilities and ensuring security will continue to shape the industry’s future.
Read next:
• AI Hiring Exposed: White Male Names Dominate While Black and Female Candidates Are Overlooked!
• Search Engines Top Consumer Preferences; Social Media, AI Tools Gain Influence
• Digital Marketers Under Pressure: 57% Feel Urgency to Learn AI Skills as Adoption Surges to 74%
