Evaluation of the performance of large language models in clinical decision-making in endodontics

Abstract

Background

Artificial intelligence (AI) chatbots are excellent at generating language. The growing use of generative AI large language models (LLMs) in healthcare and dentistry, including endodontics, raises questions about their accuracy. The potential of LLMs to assist clinicians’ decision-making processes in endodontics is worth evaluating. This study aims to comparatively evaluate the answers provided by Google Bard, ChatGPT-3.5, and ChatGPT-4 to clinically relevant questions from the field of Endodontics.

Methods

40 open-ended questions covering different areas of endodontics were prepared and were introduced to Google Bard, ChatGPT-3.5, and ChatGPT-4. Validity of the questions was evaluated using the Lawshe Content Validity Index. Two experienced endodontists, blinded to the chatbots, evaluated the answers using a 3-point Likert scale. All responses deemed to contain factually wrong information were noted and a misinformation rate for each LLM was calculated (number of answers containing wrong information/total number of questions). The One-way analysis of variance and Post Hoc Tukey test were used to analyze the data and significance was considered to be p < 0.05.

Results

ChatGPT-4 demonstrated the highest score and the lowest misinformation rate (P = 0.008) followed by ChatGPT-3.5 and Google Bard respectively. The difference between ChatGPT-4 and Google Bard was statistically significant (P = 0.004).

Conclusion

ChatGPT-4 provided more accurate and informative information in endodontics. However, all LLMs produced varying levels of incomplete or incorrect answers.

Introduction

The term “artificial intelligence” (AI) was introduced in the 1950s and refers to the concept of creating machines that can perform tasks typically carried out by humans [1]. AI employs algorithms to simulate intelligent actions with little human input. Predominantly reliant on machine learning, AI’s scope includes data retrieval and analysis of datasets and images [2, 3]. AI has been growing rapidly in dental practices over the past few years, enabling computers to handle tasks previously requiring human involvement and enabling healthcare professionals to provide better oral health care.

Large language models (LLMs) are AI-based software that simulates human language processing abilities such as understanding the meaning of a phrase or responding and creating new content after being trained with massive datasets. Thus, an LLM can generate an article on any subject, answer a question, or translate a text after being accordingly instructed [4]. In dentistry, LLMs can contribute to research, clinical decision-making, and personalized patient care, enhancing efficiency and reducing errors [1]. They can assist dentists in diagnosing oral diseases by analyzing patient records, medical histories, and imaging reports. They also support treatment planning by synthesizing vast amounts of scientific literature and providing evidence-based recommendations [5]. As these models continue to evolve, they hold significant potential to transform healthcare delivery and improve patient outcomes.

Released in November 2022, Chatbot Generative Pretrained Transformer (ChatGPT) (OpenAI, San Francisco, CA, USA), is an LLM that can answer questions quickly and fluently and interact with the user in a way that mimics human communication [6]. ChatGPT can write code or articles, summarise text, remember the previous user’s input and response in the thread, and elaborate its answers with further queries [7, 8]. ChatGPT is shown to have a basic understanding of oral potentially malignant disorders with potential limitations such as inaccurate content and references [9]. Currently, there is an older version, ChatGPT-3.5 available for free and the latest version ChatGPT-4 which is subject to a paid subscription. A recent study by Suárez et al. showed that ChatGPT-3.5 can consistently answer dichotomous questions, but is not able to give correct answers to all questions [10]. Unlike its previous model, ChatGPT-4 can also retrieve information from the internet. On the other hand, The Google Bard chatbot (Alphabet, Mountain View, California), was launched in March 2023 as a potential competitor, can also complete language-related tasks and answer questions with detailed information [11]. Another study demonstrated that Google Bard holds the potential to aid evidence-based dentistry; nevertheless, it gives occasional wrong answers or no answers despite its live access to the internet [12].

There are studies examining the ability of LLMs to answer multiple choice questions and the ability to answer open-ended questions in various clinical scenarios [10, 13,14,15,16]. It has been observed that success varies depending on the LLM used, field and type of question. Unlike previous ones, the present study focuses on the ability of LLMs to provide scientific knowledge in endodontics.

Endodontics is a branch of dentistry that requires clinicians to have up-to-date knowledge in many areas such as pathologies, stages of root canal treatment, vital pulp treatments, dental trauma, etc. The application of LLMs in endodontics has the potential to enhance diagnostic accuracy, treatment planning, and patient management. These AI-driven models can analyze vast amounts of endodontic literature, clinical guidelines, and patient records to assist clinicians in identifying complex root canal pathologies and suggesting evidence-based treatment approaches [17]. Moreover, LLMs can aid in the interpretation of radiographic images by integrating natural language processing (NLP) with deep learning techniques, improving the detection of periapical lesions and root fractures [18].

In one study, periapical X-rays were analyzed by ChatGPT and its performance was evaluated based on its ability to accurately identify a variety of dental conditions, including tooth decay, endodontic treatments, dental restorations, and other oral health problems. Endo-oral lesions were missed in 56% of cases. The AI’s overall correct interpretation rate was 11%, indicating limited clinical utility in its current form. Misinterpretations included incorrect tooth identification and failure to recognize certain dental lesions. Although ChatGPT is promising in endodontics, it has been reported that it has significant gaps in its diagnostic accuracy [19].

The potential of LLMs to assist clinicians in decision-making in the endodontic clinic is worth evaluating so that, if adequate, they can be utilized and, if not, so that clinicians are aware of the limitations that exist and can contribute to improving the deficiencies of these language models. To the best of the authors’ knowledge, there are no prior studies that compare LLMs such as ChatGPT and Google Bard as potential sources of information in Endodontics. Therefore, this study aimed to compare the accuracy and completeness of the answers generated by ChatGPT-3.5, ChatGPT-4, and Google Bard to open-ended questions related to Endodontics.

Materials and methods

The study was conducted under the Declaration of Helsinki. Since our study had no human subjects involved, ethical approval was not required. 57 open-ended questions covering different areas of endodontics within the content of the Guidelines and Position Statements of the European Society of Endodontology, and the American Association of Endodontists were prepared (Table 1) [20,21,22,23,24]. These articles were chosen by two endodontists (Y.Ö. and G.A.D.) since articles represent a consensus achieved by expert committees in the field. The validity of the questions was evaluated by 8 volunteer endodontists using the Lawshe’s Content Validity Index, a widely accepted method [25].

Table 1 Questions directed to the LLMs

The experts rated each question as “essential,” “useful but not essential,” or “not essential.” The critical content validity ratio value obtained according to Wilson et al., is 0.69 for 8 volunteers at a significance level of 0.05 [25]. Hence, questions with a content validity ratio value of ≥ 0.69 were selected for inclusion. As a result, 40 open-ended, clinically relevant questions that require text-based responses were included. The sample size was calculated at the significance level of 0.05, effect size of 0.3 and power of 0.85 using G* Power v3.1 (Heinrich Heine, Universität Düsseldorf, Germany). Questions aimed to measure broad endodontic knowledge, such as endodontic indications, antibiotic use, root canal treatment procedures, symptoms of acute apical abscesses, and common conditions that may cause concern during endodontic treatment, were presented. All questions and responses were in English.

The questions were introduced into Google Bard, ChatGPT-3.5, and ChatGPT-4 using the “new chat” option on 20th December 2023. Each question was presented only once to each LLM, and there was no follow-up, rephrasing, or annotation reflecting real-world situations for general dentists. Examples of answers to the questions are presented in Figs. 1, 2 and 3. All answers were recorded for subsequent analysis. Two endodontists, (Y.Ö. and D.E.) with ten years of experience [26], blindly evaluated the answers using a 3-point (incorrect, partially correct/incorrect or correct). The Likert scale, which is very popular in social research and has a simple measurement process, was used [27]. Likert scale adapted from Suárez et al. (Table 2) [14]. When disagreements arose in some evaluations, these were resolved by the intervention of a third researcher. Additionally, all responses deemed to contain factually wrong information were noted and a misinformation rate for each LLM was calculated (number of answers containing wrong information/total number of questions). Scores were saved in an Excel© spreadsheet (Microsoft, Redmond, Washington, USA).

Table 2 Likert scale description

Statistical analysis

The statistical analysis was performed via MiniTab 17 (Minitab Inc., USA). The normality of the data was assessed with Ryan-Joiner test and the normal distribution of the data was confirmed. One-way analysis of variance (ANOVA) and Post Hoc Tukey test were performed. A significance level of P < 0.05 was used to determine statistical significance.

Fig. 1

Sample answer from Google Bard

Fig. 2

Sample answer from ChatGPT-4

Fig. 3

Sample answer from ChatGPT-3.5

Results

The mean values and standard deviations of each group are presented in Table 3; Fig. 4. ChatGPT-4 demonstrated the highest score and the lowest misinformation rate followed by ChatGPT-3.5 and Google Bard respectively. The difference between ChatGPT-4 and Google Bard was statistically significant (P = 0.004). However, there was no statistically significant difference between ChatGPT-3.5 and ChatGPT-4, and ChatGPT-3.5 and Google Bard.

Table 3 Mean, standard deviation of the likert scales cores

Fig. 4

Mean, standard deviation and misinformation rates of the Likert Scales cores. Different letters indicate statistically significant difference (P < 0.05)(Mean ± SD, n = 40). Misinformation rates of each language model were given inside columns with % symbol

Discussion

Interest in LLMs has increased in studies in the field of dentistry as in other fields. Some of these studies have examined the ability of LLMs to answer multiple-choice questions, and some studies have examined the ability to answer open-ended questions in various clinical scenarios [12, 14,15,16, 28]. It has been observed that success varies according to the LLM used, the field and the type of question. This study rather focuses on ability of LLMs to provide scientific knowledge in endodontics.

One of the important issues to consider when using chatbots as a source of medical information is the formulation of questions, which significantly influences chatbot responses [29]. Open-ended questions better capture the nuances of the medical decision-making process [30]. The limitation of this study is that the questions are narrow in scope and only related to the field of endodontics. Conversational LLMs perform well in summarizing health-related texts, answering general questions in health care, and collecting information from patients [31].

It was reported that ChatGPT-3 was generally effective and could be used as an aid when an oral radiologist needed additional information about pathologies. However, it was also emphasized that it could not be used as a main reference source [32]. In a diagnostic accuracy study, ChatGPT had the most difficulty recognizing dental lesions. In some cases, images incorrectly identified by ChatGPT as tooth decay were reported to be actually caused by overlapping tooth ridges, and were found to be inadequate for detecting bone loss associated with periodontal disease. It was also shown that when interpreting some radiographs, it did not properly recognise ceramic crowns placed on teeth that were slightly radiopaque [19]. On the other hand, in a study comparing the answers given by AI and humans in the exam, ChatGPT-3 and ChatGPT 4 were found to outperform humans, and were reported that the newer version performed better [33]. AI has the potential to provide significant practicality in clinical applications such as treatment recommendations in the field of endodontics, and more scientific studies are needed to evaluate its possible benefits and risks in a balanced manner.

Similar to a study evaluating the ability of ChatGPT to be a potential source of information for patients’ questions about periodontal diseases [34], it was also observed that all LLMs suggested to seek professional help from a dentist or medical professional especially when it comes to questions related to diagnosis, diseases, and treatment indications. A comprehensive review reported that conversational LLMs showed promising results in summarizing and providing general medical information to patients with relatively high accuracy. However, it was noted that conversational LLMs such as ChatGPT cannot always provide reliable answers to complex health-related tasks that require expertise (e.g., diagnosis). Although bias or confidentiality issues are often cited as concerns, no research has yet been found in articles that carefully examine how conversational LLMs lead to these problems in health services research. Furthermore, various ethical concerns have been reported regarding the application of LLMs in human health care, including reliability, bias, confidentiality and public acceptance [31]. Clear policies should be established to manage the use of sensitive data in the use of AI, ensure transparency in decision-making, and protect against biased results, and to mitigate security risks that may arise from the use of LLMs [35]. It is also critical to develop guidelines on ethical issues that balance the need to protect individual privacy and autonomy, especially in cases where erroneous or harmful advice is provided.

To simplify the evaluation process, the answers given by the language models were scored only in terms of accuracy and completeness and no other evaluation criteria were used. The predominance of ChatGPT-4 over other LLMs in providing accurate information has also been shown in studies using general clinical dentistry and Japanese National Dentist Examination questions in agreement with our study [12, 16]. However, a drawback of ChatGPT-4 is not being free of charge and the pricing is the same for all countries, which lessens the accessibility to low-income countries. Suárez et al. reported that the correct answer rate was 57.33% when they directed dichotomous questions in endodontics to ChatGPt-3.5 software [10]. The relatively low correct answer rate may be because this language model does not have internet access and the trained data set does not include scientific publications in the field of endodontics.

The questions were asked only once without rephrasing to simulate a real clinical consultation. An important finding of this is, the capability of all LLMs to perfectly understand the questions and generating an answer in the context. The answers of LLMs were found to be well-structured and written convincingly while containing misleading or wrong information. This finding, a phenomenon so-called “hallucination”, has been also reported in previous studies [12, 36]. For instance, in a study on head and neck surgery, it was observed that 46.4% of the references provided by ChatGPT4 did not actually exist [37]. Such made-up or misleading information could mislead dentists and potentially harm patients.

A limitation of the study may be that the prepared questions may not cover all clinical scenarios in endodontics. Since the questions and answers were limited to guidelines and position statements issued by scientific societies, points not covered in these documents were not included in the questions. In the study, questions were asked with specific prompts as much as possible to obtain the response and to evaluate the usefulness of the information obtained in clinical decision-making. Further studies that will evaluate LLMs as a source of information for patients may need to use different question patterns using less scientific terms.

It can be presumed that LLMs will continue to improve in terms of usability, precision, and ability to provide reliable information. Therefore, evaluation of LLMs as potential sources of information in the future might be valuable. Alternatively, and as suggested earlier, LLMs can be integrated into databases such as Web of Science and PubMed and made available for academic use. A chatbot that can provide information with real scientific references can thus provide convenience and save time for clinicians and academics [15]. In addition to the existing LLMs, software to be developed with the input of scientific publications for use in the field of dentistry can help clinicians and researchers. With proper training and development of LLMs, both diagnostic accuracy can be increased, and treatment planning can be facilitated. For example, AI can help quickly gain insight into adverse events encountered during endodontic procedures or decide on appropriate root canal filling material instead of detecting endodontic lesions on radiographs. Scientific guidance from dental associations will undoubtedly play a major role in the development of such software. Once validated, such software can serve both as a chairside assistant to dentists and be used in the training of dental students.

Conclusion

This study represents the evaluation of the accuracy and completeness of different LLMs as potential sources in endodontics. Compared to other LLMs, ChatGPT-4 seems to be ahead in providing accurate and informative information in the field of endodontics. While this study, in agreement with the literature, proves that LLMs have utilizable features in dentistry, they are not free of flaws. Although LLMs are not designed to impart knowledge in dentistry and do not claim to do so, with further improvements in place, LLMs are promising tools for assisting dentists in endodontics.