Performance of emergency triage prediction of an open access natural language processing based chatbot application (ChatGPT): A preliminary, scenario based cross sectional study
Yıl: 2023 Cilt: 23 Sayı: 3 Sayfa Aralığı: 156 - 161 Metin Dili: İngilizce DOI: 10.4103/tjem.tjem_79_23 İndeks Tarihi: 20-07-2023
Performance of emergency triage prediction of an open access natural language processing based chatbot application (ChatGPT): A preliminary, scenario based cross sectional study
Öz: OBJECTIVES: Artificial intelligence companies have been increasing their initiatives recently to improve the results of chatbots, which are software programs that can converse with a human in natural language. The role of chatbots in health care is deemed worthy of research. OpenAI’s ChatGPT is a supervised and empowered machine learning based chatbot. The aim of this study was to determine the performance of ChatGPT in emergency medicine (EM) triage prediction. METHODS: This was a preliminary, cross sectional study conducted with case scenarios generated by the researchers based on the emergency severity index (ESI) handbook v4 cases. Two independent EM specialists who were experts in the ESI triage scale determined the triage categories for each case. A third independent EM specialist was consulted as arbiter, if necessary. Consensus results for each case scenario were assumed as the reference triage category. Subsequently, each case scenario was queried with ChatGPT and the answer was recorded as the index triage category. Inconsistent classifications between the ChatGPT and reference category were defined as over triage (false positive) or under triage (false negative). RESULTS: Fifty case scenarios were assessed in the study. Reliability analysis showed a fair agreement between EM specialists and ChatGPT (Cohen’s Kappa: 0.341). Eleven cases (22%) were over triaged and 9 (18%) cases were under triaged by ChatGPT. In 9 cases (18%), ChatGPT reported two consecutive triage categories, one of which matched the expert consensus. It had an overall sensitivity of 57.1% (95% confidence interval [CI]: 34–78.2), specificity of 34.5% (95% CI: 17.9–54.3), positive predictive value (PPV) of 38.7% (95% CI: 21.8–57.8), negative predictive value (NPV) of 52.6 (95% CI: 28.9–75.6), and an F1 score of 0.461. In high acuity cases (ESI 1 and ESI 2), ChatGPT showed a sensitivity of 76.2% (95% CI: 52.8–91.8), specificity of 93.1% (95% CI: 77.2–99.2), PPV of 88.9% (95% CI: 65.3–98.6), NPV of 84.4 (95% CI: 67.2–94.7), and an F1 score of 0.821. The receiver operating characteristic curve showed an area under the curve of 0.846 (95% CI: 0.724–0.969, P < 0.001) for high acuity cases. CONCLUSION: The performance of ChatGPT was best when predicting high acuity cases (ESI 1 and ESI 2). It may be useful when determining the cases requiring critical care. When trained with more medical knowledge, ChatGPT may be more accurate for other triage category predictions.
Anahtar Kelime: Belge Türü: Makale Makale Türü: Araştırma Makalesi Erişim Türü: Erişime Açık
- 1. A 30 year old female patient is brought to the ED due to sudden onset of unconsciousness. On her arrival she appears comatose. Pupils were miotic, GCS: 6, BP: 100/60 mmHg, HR: 110, RR: 14, SpO2: 90%, T: 98° F. First responders told you that they found an empty syringe at the scene
- 2. A 60 year old male patient presents to the emergency department with complaints of fainting and impaired consciousness. Vital signs on arrival: BP: 70/30 mmHg, HR: 25/min, RR: 15, SpO2: 99%, T: 98° F
- 3. A 90 year old male patient is brought to the ED by ambulance. It is stated that the patient fainted in the toilet and his stool seemed bloody. He seems pale. Intense red blood is seen on the patient’s diaper. Vital signs on arrival: BP: 65/30 mmHg, HR: 140/min, RR: 25. Medication history is notable for anticoagulants.
- 4. A 25 year old male patient is brought to the ED by ambulance due to a gunshot wound to the abdomen. On arrival, he is conscious and oriented. The bullet entrance wound is located on the left upper quadrant of the abdomen. No other wounds were found in the primary evaluation. BP: 80/50 mmHg, HR: 120/min, RR: 20/ min. It is learned that up to 500 ml of fluid was given during the transfer
- 5. A 60 year old female patient with a diagnosis of COPD is brought to the ED with respiratory distress. On arrival, her oxygen saturation is measured as 70%. BP: 90/55 mmHg, HR: 115/min, RR: 25/min
- 6. A 65 year old female patient presents with pressure like pain located in the middle of the chest that has been going on for 1 hour. She appears to have a cold sweat. BP: 60/palp, heart rate: 140/min, RR: 25
- 7. A 50 year old patient with a history of heart failure and hypertension is brought to the ED with shortness of breath. BP: 190/90 mmHg, HR: 100/min, Spo2: 78%, RR: 30/min. A gasping sound when breathing is notable
- 8. A 20 year old male patient was found unconscious at home. Empty boxes of prescribed drugs he used for the treatment of depression were found next to him. On arrival he appears comatose. Pupils are miotic, GCS: 5, BP: 90/60 mmHg, HR: 110, RR: 10, SpO2: 89%
- 9. EMS is activated for a 30 year old female patient who was found unconscious at home. First responders found the patient was in cardiac arrest and spontaneous circulation was established with successful CPR. On arrival to the ED BP: 90/50 mmHg, HR: 130, spo2: 99%
- 10. A 70 year old patient who presented to the ED with abdominal pain. Seems pale. Capillary refill time is prolonged. Vital signs on arrival: BP: 60/palp, HR: 130, spo2: 97%, Temp: 36 C
- 11. A 36 year old female presents to the ED. She has a history of severe allergic reaction due to a bee sting. She told you that a bee stung her arm. In a couple of minutes, she felt dizzy and nauseous. Vital Signs: BP 145/74, HR 117, RR 19, SpO2: 98%, T: 97° F
- 12. 12 A 50 year old male patient is admitted to the ED with complaints of palpitation and fainting that has been going on for a few hours. Vital Signs: BP: 100/40 mmHg, HR: 190, RR: 15, SpO2: 99%, T: 98° F
- 13. A 40 year old female patient presents with pressure like pain in the middle of the chest that has been going on for 2 hours. BP: 130/80 mmHg, heart rate: 70/min, RR: 14, T: 36.5 C
- 14. A 30 year old female patient comes with a sudden onset of severe headache and nausea after lifting a heavy object. She is conscious, oriented and cooperative and states that the pain is the most severe pain of her life. BP: 120/75 mmHg, HR: 80, RR: 15, T: 36.5 C
- 15. A 25 year old physician comes to the ED after a needle used while administering medication to her patient accidentally stabbed her hand
- 16. A 26 year old patient, who is 20 weeks pregnant, presents to the ED with the complaint of pain and “water breaking “. BP: 120/85 mmHg, HR: 90, RR: 15, T: 36.5 C
- 17. A 50 year old male patient presents to the ED with complaints of high fever, and chills. It is learned that he is on chemotherapy for lung cancer. BP: 120/69 mmHg, HR: 105, SpO2: 96, RR: 16, T: 38.8 C
- 18. An 18 year old female patient was rescued by her relatives after an attempt to hang herself at home and brought to the ED by ambulance. There was no loss of consciousness. No markings are present on her neck. BP: 130/75 mmHg, HR: 100, SpO2: 98, RR: 14, T: 36.8 C
- 19. A 90 year old male patient is brought to the ED because of the confusion that started on the same day. On arrival he is conscious, but gives confused answers. BP: 120/69 mmHg, HR: 80, SpO2: 98, RR: 14, T: 36.6 C
- 20. A 40 year old female patient presents with severe left flank pain. Her skin looks pale. In her medical history, it is learned that she presented to the ED due to kidney stones before. Pain score is 9/10. BP: 140/90 mmHg, HR: 110, SpO2: 98, RR: 17, T: 36.6 C
- 21. A 20 year old male patient presents to the ED with complaints of nausea and vomiting that started in the morning and pain localized to the right lower quadrant within hours. Pain score is 6/10. BP: 130/90 mmHg, heart rate: 100, SpO2: 98, RR: 16, T: 36.4 C
- 22. A 20 year old male patient presents to the ED with the suspicion of a fracture of the forearm after falling on his arm while playing basketball. A prominent displaced fracture is observed on the proximal wrist. Radial pulse is palpable and sensation and circulation are evaluated as normal
- 23. A 34 year old female patient comes with complaints of high fever and cough for two days. Describes chest pain that increases with coughing. She seems healthy. BP: 120/90 mmHg, heart rate: 100, SpO2: 98, RR: 16, T: 38.4 C
- 24. A 40 year old female patient presents with the complaint of pain in the upper quadrants of the abdomen that started after lunch today. She has a history of gallstones and DM in her history. BP: 130/80 mmHg, heart rate: 87, SpO2: 98, RR: 14, T: 36.2 C
- 25. A 27 year old female patient presents to the ED with complaints of nausea, vomiting and diarrhea that started about 1 day ago. She describes abdominal pain that is relieved by diarrhea and is not very severe. A few hours before her symptoms started, she had eaten a meal that “tasted strange”. Diarrhea is watery and includes mucus and does not contain blood. Her skin appears dry but turgor is normal. BP: 140/90 mmHg, heart rate: 89, SpO2: 99, RR: 15, T: 36.4 C
- 26. A 35 year old female patient is presented to the ED due to the pain in her left leg for 2 days. She previously had traveled on intercontinental flights. The left leg appears to be edematous. Distal pulses are palpable. On examination, the Homans test is positive. BP: 130/88 mmHg, heart rate: 85, SpO2: 97, RR: 15, T: 36.4 C
- 27. A 55 year old male patient comes to the ED with the complaint of shortness of breath during exertion. No active complaints at the time of presentation. He states that he has arrhythmia and hypertension in his medical history. He doesn’t use prescribed drugs. BP: 120/60 mmHg, heart rate: 97, SpO2: 95, RR: 16, T: 36.4 C
- 28. A 45 year old female patient is brought to the ED by ambulance for falling from the first floor balcony. She has an open fracture in her left leg. She states that she fell on her left foot and describes no other injury. She has not lost consciousness. Pain Score: 9/10. BP: 110/80 mmHg, heart rate: 100, SpO2: 99, RR: 15, T: 36.4 C
- 29. A 3 year old male patient is brought in with the complaint of slipping his foot and hitting his head on the cabinet door. There is a superficial laceration in the frontal head. He is alert. Family does not describe loss of consciousness. The family was worried because he vomited several times
- 30. A 24 year old female patient presents to the ED with severe abdominal pain. It is learned that there is a delay in menstruation and that the pregnancy test she did 2 days ago was positive. BP: 110/64 mmHg, heart rate: 70, RR: 14, T: 36.5 C
- 31. A 22 year old male patient comes with pain around the ankle following falling while running. No open wounds or deformity. Pain score is 8/10
- 32. A 30 year old male patient presents with swelling in the nail bed of the left index finger. The appearance of the abscess formed within two days. He complaints of pain and throbbing. He has no fever
- 33. A 40 year old female patient comes with complaints of frequent and painful urination. No known medical condition. She does not describe abdominal pain. BP: 120/90 mmHg, heart rate: 70, SpO2: 99, RR: 14, T: 36.4 C
- 34. A 15 year old male patient comes with a complaint of edema in the nose after being hit with a ball while playing football. No deformity. No active nosebleeds. No history of syncope. No additional complaints. Oriented and cooperative. BP: 110/60 mmHg, heart rate: 70, SpO2: 99, RR: 14, T: 36.5 C
- 35. A 23 year old female patient fell down while jogging. Presents to the ED due to a superficial laceration on the forearm. Range of motion of joints is normal. She does not describe pain in the extremity. There is no active bleeding from the wound. There is a simple superficial laceration of approximately 3 4 cm in length
- 36. A 30 year old male patient comes with the complaint of eye itching after walking in windy weather. Thinks a particle of dust got in his eye. No known medical history. Pain score: 3/10
- 37. A 77 year old female patient with a history of heart valve replacement and on anticoagulants presents to the ED with a couple of bruises in her legs. No additional complaints. Describes no bleeding. BP: 110/75 mmHg, heart rate: 80, SpO2: 99, RR: 14, T: 36.5 C
- 38. A 43 year old male patient came to the ED because of an accidental knife cut on his index finger. The bleeding stopped with the dressing he applied at home. The cut seems superficial but may need some sutures. Distal sensory and circulatory examination is normal. Tendons and nerves seem intact. BP: 110/80 mmHg, heart rate: 80, SpO2: 99, RR: 14, T: 36.5 C
- 39. A 60 year old female patient comes because a splinter penetrated her foot while walking. It is seen that there is a splinter located on the sole of the foot which is protruding from the skin
- 40. A 20 year old male patient, who describes he has lost taste and smell for two days and suspects that he has contacted with COVID 19, is presented to the ED. He has no medical history. Respiratory sounds are normal. Seems healthy. BP: 130/85 mmHg, HR: 73, SpO2: 99, RR: 14, T: 36.7 C
- 41. An 8 year old male patient presents with a burn with an area of 3 cm2 due to scald injury on his left forearm the day before. No bullae. Pain score: 3/10
- 42. A 10 year old male patient comes to the Emergency Department with the complaint of itching and redness in both eyes. He has no fever. Vital parameters are normal. Does not describe pain
- 43. A 40 year old female patient presents to the ED with a sore throat for 3 days. No swallowing difficulties or hoarseness. Examination of the throat is normal. BP: 120/75 mmHg, heart rate: 80, SpO2: 98, RR: 14, T: 36.8 C
- 44. A 20 year old female patient has a history of nickel allergy. After wearing a metal bracelet, she complains of itching and redness on the area which contacts with the bracelet. No systemic reaction
- 45. A 50 year old female patient with caries on her left molar teeth comes with the complaint of pain in the same area that starts while eating. No facial swelling. Pain score: 4/10. No fever
- 46. A 2 year old female patient is brought to the ED with a runny nose and vomiting. She looks active and alert. Her mother states that her appetite is normal. Seems hydrated
- 47. A 27 year old female patient presents with complaints of pain in both legs with movement and palpation, which started one day after heavy exercise. BP: 120/90 mmHg, HR: 70, SpO2: 99, RR: 14, T: 36.4 C. No difference in diameter between the two legs. Urine color is normal
- 48. A 10 year old male patient presents with swelling in the nail bed of the right hand thumb. There is no pus filled blister. He has no fever
- 49. An 18 year old female patient presented to the ED due to stepping on a rusty nail. On inspection, you can not see a wound. She has no additional complaints and no fever
- 50. A 70 year old male patient comes with pain on the left big toe while walking. He states that his complaints have been going on for about a few weeks. On palpation, a hard dark lesion is present located under the left big toe.
| APA | Sarbay İ, Bozdereli Berikol G, Ozturan I (2023). Performance of emergency triage prediction of an open access natural language processing based chatbot application (ChatGPT): A preliminary, scenario based cross sectional study. , 156 - 161. 10.4103/tjem.tjem_79_23 |
| Chicago | Sarbay İbrahim,Bozdereli Berikol Göksu,Ozturan Ibrahim Ulas Performance of emergency triage prediction of an open access natural language processing based chatbot application (ChatGPT): A preliminary, scenario based cross sectional study. (2023): 156 - 161. 10.4103/tjem.tjem_79_23 |
| MLA | Sarbay İbrahim,Bozdereli Berikol Göksu,Ozturan Ibrahim Ulas Performance of emergency triage prediction of an open access natural language processing based chatbot application (ChatGPT): A preliminary, scenario based cross sectional study. , 2023, ss.156 - 161. 10.4103/tjem.tjem_79_23 |
| AMA | Sarbay İ,Bozdereli Berikol G,Ozturan I Performance of emergency triage prediction of an open access natural language processing based chatbot application (ChatGPT): A preliminary, scenario based cross sectional study. . 2023; 156 - 161. 10.4103/tjem.tjem_79_23 |
| Vancouver | Sarbay İ,Bozdereli Berikol G,Ozturan I Performance of emergency triage prediction of an open access natural language processing based chatbot application (ChatGPT): A preliminary, scenario based cross sectional study. . 2023; 156 - 161. 10.4103/tjem.tjem_79_23 |
| IEEE | Sarbay İ,Bozdereli Berikol G,Ozturan I "Performance of emergency triage prediction of an open access natural language processing based chatbot application (ChatGPT): A preliminary, scenario based cross sectional study." , ss.156 - 161, 2023. 10.4103/tjem.tjem_79_23 |
| ISNAD | Sarbay, İbrahim vd. "Performance of emergency triage prediction of an open access natural language processing based chatbot application (ChatGPT): A preliminary, scenario based cross sectional study". (2023), 156-161. https://doi.org/10.4103/tjem.tjem_79_23 |
| APA | Sarbay İ, Bozdereli Berikol G, Ozturan I (2023). Performance of emergency triage prediction of an open access natural language processing based chatbot application (ChatGPT): A preliminary, scenario based cross sectional study. Turkish journal of emergency medicine (Online), 23(3), 156 - 161. 10.4103/tjem.tjem_79_23 |
| Chicago | Sarbay İbrahim,Bozdereli Berikol Göksu,Ozturan Ibrahim Ulas Performance of emergency triage prediction of an open access natural language processing based chatbot application (ChatGPT): A preliminary, scenario based cross sectional study. Turkish journal of emergency medicine (Online) 23, no.3 (2023): 156 - 161. 10.4103/tjem.tjem_79_23 |
| MLA | Sarbay İbrahim,Bozdereli Berikol Göksu,Ozturan Ibrahim Ulas Performance of emergency triage prediction of an open access natural language processing based chatbot application (ChatGPT): A preliminary, scenario based cross sectional study. Turkish journal of emergency medicine (Online), vol.23, no.3, 2023, ss.156 - 161. 10.4103/tjem.tjem_79_23 |
| AMA | Sarbay İ,Bozdereli Berikol G,Ozturan I Performance of emergency triage prediction of an open access natural language processing based chatbot application (ChatGPT): A preliminary, scenario based cross sectional study. Turkish journal of emergency medicine (Online). 2023; 23(3): 156 - 161. 10.4103/tjem.tjem_79_23 |
| Vancouver | Sarbay İ,Bozdereli Berikol G,Ozturan I Performance of emergency triage prediction of an open access natural language processing based chatbot application (ChatGPT): A preliminary, scenario based cross sectional study. Turkish journal of emergency medicine (Online). 2023; 23(3): 156 - 161. 10.4103/tjem.tjem_79_23 |
| IEEE | Sarbay İ,Bozdereli Berikol G,Ozturan I "Performance of emergency triage prediction of an open access natural language processing based chatbot application (ChatGPT): A preliminary, scenario based cross sectional study." Turkish journal of emergency medicine (Online), 23, ss.156 - 161, 2023. 10.4103/tjem.tjem_79_23 |
| ISNAD | Sarbay, İbrahim vd. "Performance of emergency triage prediction of an open access natural language processing based chatbot application (ChatGPT): A preliminary, scenario based cross sectional study". Turkish journal of emergency medicine (Online) 23/3 (2023), 156-161. https://doi.org/10.4103/tjem.tjem_79_23 |
