Accuracy Analysis of Text-to-Speech (TTS) on Chatbot GPT for Arabic Conversation and Grammatical Mapping
DOI:
https://doi.org/10.58230/edutech.v4i2.144Keywords:
Accuracy Text-To-Speech, Chatbot GPT, Arabic ConversationAbstract
This qualitative study evaluates ChatGPT-4o's spoken Arabic capabilities and grammatical mapping accuracy across two distinct devices using a single foundational Arabic sentence prompt ("اللغة العربية هي لغة الجنة"). The evaluation criteria encompassed diacritical precision (harakat), phonetic articulation (makhraj), syntactic comprehension (nahw), and I'rab parsing. We cross-verified data generated through structured dialogues against Al-Nahw Al-Wadih and validated it with an Arabic linguistics expert. The empirical findings indicate that ChatGPT-4o achieved uniform excellence, scoring 4 out of 4 in diacritical rendering and phonetic articulation across both devices. However, in systematic syntactic parsing (I'rab), the model achieved measured grammatical accuracy rates of 78.4% on Device I using an Arabic interface and 74.5% on Device II using an English interface. These findings show that while AI can support Arabic conversational practice and basic linguistic inquiries, its structural parsing performance remains constrained by default system language configurations, hardware variations, and overall sentence complexity.
Downloads
References
Aidah Novianti Putri and Moh. Abdul Kholiq Hasan. 2022. ‘Penerapan Kecerdasan Buatan sebagai Media Pembelajaran Bahasa Arab di Era Society 5.0’. Tarling : Journal of Language Education 7(1):69–80. doi:10.24090/tarling.v7i1.8501.
Alfurqan, M. Hadli, & Wirdati Wirdati. (2024). ‘Analisis Penggunaan Website Berbasis AI dalam Mengerjakan Tugas Akademik pada Mahasiswa Pendidikan Agama Islam Universitas Negeri Padang’. ISLAMIKA 6(4):1727–39. doi:10.36088/islamika.v6i4.5309.
Alhuwaydi, A. M. (2024). ‘Exploring the Role of Artificial Intelligence in Mental Healthcare: Current Trends and Future Directions – A Narrative Review for a Comprehensive Insight’. Risk Management and Healthcare Policy 17:1339–48.
Allaithy, A., & Mai Zaki. (2025). ‘Evaluation of AI-Generated Reading Comprehension Materials for Arabic Language Teaching’. Computer Assisted Language Learning 0(0):1–33. doi:10.1080/09588221.2025.2474037.
Alruqi, T. N., & Salha M. Alzahrani. (2023). ‘Evaluation of an Arabic Chatbot Based on Extractive Question-Answering Transfer Learning and Language Transformers’. AI 4(3):667–91. doi:10.3390/ai4030035.
Al-Sabahi, K., Amer Fael Mohammed Balhaf, M. Moheb Zagloul Al Shamy, Majjed Al-Qatf, Kang Yang, & Abdulrahman Al-Badwi. (2026). ‘Benchmarking Large Language Models on Arabic Parsing’. International Journal of Information Management Data Insights 6(1):100404. doi:10.1016/j.jjimei.2026.100404.
Badawi, S. A., Maen Takruri, K. Salameh, Mohammad Al-Badawi, Nowar Alani, Isam ElBadawi, Aws Al-Qaisi, & Ghaleb Aldoboni. (2026). ‘AI-Based Framework for Arabic Language Proficiency Assessment: A Deep Learning ASR Model with Enhanced Similarity Measures’. Future Internet, 18(5), 251. doi:10.3390/fi18050251.
Behzad, S., Amir Zeldes, & Nathan Schneider. (2024). ‘To Ask LLMs about English Grammaticality, Prompt Them in a Different Language’. Pp. 15622–34 in Findings of the Association for Computational Linguistics: EMNLP 2024, edited by Y. Al-Onaizan, M. Bansal, and Y.-N. Chen. Miami, Florida, USA: Association for Computational Linguistics.
Boluk, Pinar, & Hana’a Maratouq. (2026). ‘An AI-Based Security Architecture for Fraud Detection in Cloud Call Centers for Low-Resource Languages: Arabic as a Use Case’. Electronics 15(8):1718. doi:10.3390/electronics15081718.
Chen, D.-L., Kirsi Aaltonen, H. Lampela, & Jaakko Kujala. (2024). ‘The Design and Implementation of an Educational Chatbot with Personalized Adaptive Learning Features for Project Management Training’. Technology, Knowledge and Learning. doi:10.1007/s10758-024-09807-5.
Creswell, John W., and J. David Creswell. 2018. Research Design: Qualitative, Quantitative, and Mixed Methods Approaches. 5th edn. Los Angeles: Sage.
Dokukina, I., & Julia Gumanova. (2020). ‘The Rise of Chatbots – New Personal Assistants in Foreign Language Learning’. Procedia Computer Science 169:542–46. doi:https://doi.org/10.1016/j.procs.2020.02.212.
Elgamal, S., Ossama Obeid, M. Kabbani, Go Inoue, & Nizar Habash. (2024). ‘Arabic Diacritics in the Wild: Exploiting Opportunities for Improved Diacritization’. Pp. 14815–29 in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), edited by L.-W. Ku, A. Martins, and V. Srikumar. Bangkok, Thailand: Association for Computational Linguistics.
Favirotus Siyam, Fera, Rahmat Hidayat, Cecep Sobar Rochmat, Rosendah Dwi Maulaya, Annisa Avilya, & Muhammad Bahaudin Maulidi. (2024). ‘Accuracy Analysis of Artificial Intelligence in Arabic Language Translation and Grammatical Rule Mapping’. Jurnal Al Bayan: Jurnal Jurusan Pendidikan Bahasa Arab 16(2):558. doi:10.24042/albayan.v16i2.24588.
Fitri, W. A., & Muqita Hanifah Hasanah Dilia. (2024). ‘Optimalisasi Teknologi AI dalam Meningkatkan Efektivitas Pembelajaran’. 5(10).
Ghamrawi, N., T. Shal, & N. A. R. Ghamrawi. (2024). ‘Exploring the Impact of AI on Teacher Leadership: Regressing or Expanding?’ Education and Information ….
Hong, X., & Li Guo. (2025). ‘Effects of AI-Enhanced Multi-Display Language Teaching Systems on Learning Motivation, Cognitive Load Management, and Learner Autonomy’. Education and Information Technologies 30(12):17155–89. doi:10.1007/s10639-025-13472-1.
Jeon, J., Seongyong Lee, & Hohsung Choe. (2023). ‘Beyond ChatGPT: A Conceptual Framework and Systematic Review of Speech-Recognition Chatbots for Language Learning’. Computers & Education, 206:104898. doi:https://doi.org/10.1016/j.compedu.2023.104898.
Karataş, F., Faramarz Yaşar Abedi, F. Özek Günyel, Derya Karadeniz, & Yasemin Kuzgun. (2024). ‘Incorporating AI in Foreign Language Education: An Investigation into ChatGPT’s Effect on Foreign Language Learners’. Education and Information Technologies 29(15):19343–66. doi:10.1007/s10639-024-12574-6.
Karim, A. A., & Gheith Abandah. (2021). ‘On the Training of Deep Neural Networks for Automatic Arabic-Text Diacritization’. International Journal of Advanced Computer Science and Applications (IJACSA) 12(8). doi:10.14569/IJACSA.2021.0120832.
Khondaker, M. T. I., Abdul Waheed, E. Moatez Billah Nagoudi, & Muhammad Abdul-Mageed. (2023). ‘GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLP’. Pp. 220–47 in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, edited by H. Bouamor, J. Pino, and K. Bali. Singapore: Association for Computational Linguistics.
Kwon, S., Gagan Bhatia, E. M. B. Nagoudi, & Muhammad Abdul-Mageed. (2023). ‘Beyond English: Evaluating LLMs for Arabic Grammatical Error Correction’. Pp. 101–19 in Proceedings of ArabicNLP 2023, edited by H. Sawaf, S. El-Beltagy, W. Zaghouani, W. Magdy, A. Abdelali, N. Tomeh, I. Abu Farha, N. Habash, S. Khalifa, A. Keleg, H. Haddad, I. Zitouni, K. Mrini, and R. Almatham. Singapore (Hybrid): Association for Computational Linguistics.
Maulaya, R. D., Annisa Avilya, I. Syafa’atun Nur, Fasya Asy Shifaauzzahra, S. Fadhillah Y., & Fera Favirotus Siyam. (2024). ‘Design Character Education Curriculum in Modern Pesantren Darussalam Gontor to Strengthen the Profile of Santri in Facing the VUCA Era’. INTEGRATIA 2(2):1–16.
Mubarak, H., Majd Hawasly, & Abubakr Mohamed. (2026). ‘Nahw: A Comprehensive Benchmark of Arabic Grammar Understanding, Error Detection, Correction, and Explanation’. Pp. 6310–28 in Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), edited by V. Demberg, K. Inui, and L. Marquez. Rabat, Morocco: Association for Computational Linguistics.
Musthofa, A., Alfian, F. F. S., Alif Cahya Setiyadi, Y. Kusuma W., & Siti Aisyah. (2024). ‘Efektivitas Media Pembelajaran Kotak Kartu Misterius (KoKaMi) untuk Meningkatkan Hasil Belajar Siswa pada Pelajaran Muthola’ah’. Mantiqu Tayr: Journal of Arabic Language 4(2):517–33. doi:10.25217/mantiqutayr.v4i2.4671.
Niyu, D. D., Azalia Gerungan, & Herman Purba. (2024). ‘Penggunaan ChatGPT di Kalangan Mahasiswa dan Dosen Perguruan Tinggi di Indonesia’. CoverAge: Journal of Strategic Communication 14(2):130–45. doi:10.35814/coverage.v14i2.6058.
Qin, H., Guimin Chen, Y. Tian, & Yan Song. (2021). ‘Improving Arabic Diacritization with Regularized Decoding and Adversarial Training’. Pp. 534–42 in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), edited by C. Zong, F. Xia, W. Li, and R. Navigli. Online: Association for Computational Linguistics.
Regina Dwi Aulia, Shine Quinn Firdaus, Zaizafun Naura, & Nur Aini Rakhmawati. (2024). ‘Analisis Pengaruh Penggunaan AI ChatGPT terhadap Minat Baca Mahasiswa Sistem Informasi IT’S Jurnal Pendidikan, Bahasa dan Budaya 3(3):01–11. doi:10.55606/jpbb.v3i3.3196.
Seyidov, R., & Ahmet Çitil. (2024). ‘Arabic Speech Recognition and Artificial Intelligence Technologies’. Journal of Ecohumanism 3(4):937–42. doi:10.62754/joe.v3i4.3640.
Suariqi Diantama. (2023). ‘Pemanfaatan Artificial Intelligence (AI) dalam Dunia Pendidikan’. DEWANTECH Jurnal Teknologi Pendidikan 1(1):8–14. doi:10.61434/dewantech.v1i1.8.
Wang, Y., Gai-Jing Wang, L.-J. Y., Jing Gao, Cong F., & Hai-Mei Ren. (2024). ‘Qualitative Research on the Flipped Classroom Cognition of Undergraduate Nursing Students’. BMC Medical Education 24(1):1460. doi:10.1186/s12909-024-06426-7.
Yuan, Y. (2024). ‘An Empirical Study of the Efficacy of AI Chatbots for English as a Foreign Language Learning in Primary Education’. Interactive Learning Environments 32(10):6774–89. doi:10.1080/10494820.2023.2282112.
Yulianti, Grace, Ngadi Permana, and Fitri Ayu Kusuma Wijayanti. (2023). ‘Transformasi Pendidikan Indonesia: Memanfaatkan Potensi Kecerdasan Buatan (AI)’. 02(06).
Zbib, M. Bilal, Hasan Abed Al Kader Hammoud, Ammar Mohanna, Nadine Rizk, Fatima Karnib, Sina Moukaled, & Bernard Ghanem. (2026). ‘AraLingBench: A Human-Annotated Benchmark for Evaluating Arabic Linguistic Capabilities of Large Language Models’. Pp. 385–93 in Proceedings of the 2nd Workshop on NLP for Languages Using Arabic Script, edited by M. El-Haj, P. Rayson, M. Jarrar, I. Ezeani, S. Ezzini, S. Ahmadi, A. Haddad Haddad, C. Amol, A. Abdelali, and S. Abudalfa. Rabat, Morocco: Association for Computational Linguistics.
Zhang, Z., & Xiaomeng Huang. (2024). ‘The Impact of Chatbots Based on Large Language Models on Second Language Vocabulary Acquisition’. Heliyon 10(3):e25370. doi:10.1016/j.heliyon.2024.e25370.
عتيق, الدكتور عبد العزيز. n.d. علم النحو والصرف. Vol. الأول. الأولى. بوروت: مكنبة منيمنه.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Fasya Asy Shyifaauzzahra, Moh. Ismail, Fairuz Subakir Ahmad, Rosendah Dwi Maulaya, Ulfa Muna Kamila, Fera Favirotus Siyam, Annisa Avilya
This work is licensed under a Creative Commons Attribution 4.0 International License.
