According to a new study by a software company called Instabug, AI models work better with Apple’s iOS than Google's Android when it comes to fixing mobile crashes. The company created a tool called SmartResolve which uses AI to detect app crashes, it causes and suggests code fixes. It was also revealed that it worked more effectively with iOS than Android. The tool was tested using various AI models from Anthropic, OpenAI, Meta, and Google on real app crashes.
The main finding on all this was that AI models do better crash fixing on Apple’s iOS than on Android. The fixes on iOS were more accurate, better structured, and clearer across almost all models that were tested. Gemini 1.5 Pro, Google’s own AI model, wasn't able to do well on Android and scored 51.41% as compared to 58.53% on iOS. GPT-4o scored 59.81% on iOS and 48.97% on Android, while the o1 model scored 61.79% on iOS and 26.31% on Android. Claude's Sonnet 3.5 V1 scored 58.33% on iOS and 55.56% on Android.
According to Sherief Abul-Ezz's blog post, "The results highlight that most models performed better on iOS, with GPT-4o, Claude 3.5 Haiku V1, and Claude 3.5 Sonnet V1 emerging as the strongest contenders due to their consistency and structured outputs." Adding further, "Conversely, models like LLaMA-3-70b and OpenAI o1 struggled significantly, particularly on Android, due to poor correctness, frequent failures, and slow response times."
The chief product officer of Instabug, Kenny Johnston, said that iOS’s bigger success rate is mostly because of how its native languages like Objective-C and Swift are structured which makes AI models to detect and generate accurate fixes. On the other hand, Android uses Kotlin and Java have more variability in crash formats so AI cannot detect it accurately.
Read next: Study Finds Openness to AI’s Utility But Concern Grows Over Chatbots Replacing Real Human Relationships
The main finding on all this was that AI models do better crash fixing on Apple’s iOS than on Android. The fixes on iOS were more accurate, better structured, and clearer across almost all models that were tested. Gemini 1.5 Pro, Google’s own AI model, wasn't able to do well on Android and scored 51.41% as compared to 58.53% on iOS. GPT-4o scored 59.81% on iOS and 48.97% on Android, while the o1 model scored 61.79% on iOS and 26.31% on Android. Claude's Sonnet 3.5 V1 scored 58.33% on iOS and 55.56% on Android.
According to Sherief Abul-Ezz's blog post, "The results highlight that most models performed better on iOS, with GPT-4o, Claude 3.5 Haiku V1, and Claude 3.5 Sonnet V1 emerging as the strongest contenders due to their consistency and structured outputs." Adding further, "Conversely, models like LLaMA-3-70b and OpenAI o1 struggled significantly, particularly on Android, due to poor correctness, frequent failures, and slow response times."
The chief product officer of Instabug, Kenny Johnston, said that iOS’s bigger success rate is mostly because of how its native languages like Objective-C and Swift are structured which makes AI models to detect and generate accurate fixes. On the other hand, Android uses Kotlin and Java have more variability in crash formats so AI cannot detect it accurately.
Read next: Study Finds Openness to AI’s Utility But Concern Grows Over Chatbots Replacing Real Human Relationships
