The researchers from Brigham Young University (BYU), US, and 186 other universities, in their journal Issues in Accounting Education, have published their findings revealing that the AI chatbot scored much less than the students taking the same exam.
As per the findings, students scored an overall average of 76.7 percent, compared to ChatGPT’s score of 47.4 percent. The findings reveal that while the AI chatbot scored higher than the student average in 11.3 percent of the questions, doing particularly well on accounting information systems (AIS) and auditing, the chatbot was seen struggling with tax, financial, and managerial assessments. One of the possible reasons for poor performance, as per the researchers, could be the mathematical processes required to solve such questions.
ALSO READ l ChatGPT just passed a major medical exam; know details
According to the findings ChatGPT showed greater accuracy rates on true/false questions, scoring 68.7% correctly, and multiple-choice questions, with 59.5% accuracy. However, when it came to short-answer questions, the accuracy rate ranged between 28.7% and 39.1%, indicating some difficulties in this area.
The researchers also spotted some interesting trends like the AI chatbot doesn’t always understand mathematical questions and therefore sometimes makes illogical errors adding two numbers in a subtraction problem, or dividing numbers incorrectly. The chatbot was also seen giving explanation for its incorrect answers and sometimes even make up facts.
Despite this, the researchers have summed up their findings calling ChatGPT’s performance “impressive” and a “game changer that will change the way everyone teaches and learns – for the better.”
“It’s an opportunity to reflect on whether we are teaching value-added information or not,” said study co-author and fellow BYU accounting professor Melissa Larson. “This is a disruption, and we need to assess where we go from here. Of course, I’m still going to have TAs, but this is going to force us to use them in different ways.”
ALSO READ l ChatGPT fails India’s JEE entrance exam miserably, could answer only 11 questions correctly
ChatGPT recently was reported to flunk Joint Entrance Examination (JEE) Advanced, a highly competitive engineering entrance exam in India. The AI program reportedly failed miserably after solving just 11 questions in the exam.
