Automatic Sentiment Annotation Using Grok AI for Opinion Mining in a University Learning Management System
DOI:
https://doi.org/10.64878/jistics.v1i3.42Keywords:
Chat gpt, Grok ai, Labeling automation, Learning management system, Naive Bayes, Sentiment analysis, Text classification, Word cloud visualizationAbstract
Sentiment analysis has become an essential tool in evaluating user feedback on digital learning platforms. Understanding student sentiments toward Learning Management Systems (LMS) in higher education can offer critical insights for system development and service improvement. This study aims to evaluate the effectiveness of AI-assisted sentiment labeling using Grok AI and ChatGPT compared to manual labeling for sentiment classification of student opinions on LMS at Institut Teknologi Garut. The research involved distributing an online questionnaire to 96 students across four academic levels, collecting open-ended responses regarding their LMS usage experiences. These responses were preprocessed through case folding, cleaning, tokenization, stopword removal, and stemming. The sentiment labels were assigned using Grok AI, ChatGPT, and manual annotation, and the resulting datasets were used to build classification models using the Naïve Bayes algorithm in Altair RapidMiner with 10-Fold Cross Validation. The performance evaluation shows that manual labeling yielded the highest accuracy (52.22%) and Cohen's Kappa (0.137), followed by ChatGPT (50.11%, 0.119) and Grok AI (48.00%, 0.087). Word cloud visualizations further revealed the dominant themes within each sentiment class, indicating that positive opinions emphasized helpfulness and ease of use, while negative ones focused on access issues and system lags. This research suggests that AI-assisted labeling methods can be viable alternatives, although manual labeling still offers slightly better accuracy.
Downloads
References
A. Maritsa, U. Hanifah Salsabila, M. Wafiq, P. Rahma Anindya, and M. Azhar Ma’shum, “Pengaruh Teknologi Dalam Dunia Pendidikan,” Al-Mutharahah J. Penelit. dan Kaji. Sos. Keagamaan, vol. 18, no. 2, pp. 91–100, 2021, doi: 10.46781/al-mutharahah.v18i2.303. DOI: https://doi.org/10.46781/al-mutharahah.v18i2.303
Y. Arnas, A. Ahmad, and O. Fajarianto, “The Effectiveness of Learning Management System in Higher Education: A Case Study of Social Sciences Course Content,” IJESS Int. J. Educ. Soc. Sci., vol. 4, no. 2, pp. 114–118, 2023, doi: 10.56371/ijess.v4i2.185. DOI: https://doi.org/10.56371/ijess.v4i2.185
L. Sanchez, J. Penarreta, and X. Soria Poma, “Learning management systems for higher education: a brief comparison,” Discov. Educ., vol. 3, no. 1, 2024, doi: 10.1007/s44217-024-00143-5. DOI: https://doi.org/10.1007/s44217-024-00143-5
A. Haleem, M. Javaid, M. A. Qadri, and R. Suman, “Understanding the role of digital technologies in education: A review,” Sustain. Oper. Comput., vol. 3, pp. 275–285, 2022, doi: https://doi.org/10.1016/j.susoc.2022.05.004. DOI: https://doi.org/10.1016/j.susoc.2022.05.004
A. Sidhiq, Q. K. Rini, and U. Majorsy, “Evaluasi Pengalaman Pengguna Pada Aplikasi E-Learning Di Perguruan Tinggi,” J. Ilm. Inform. Komput., vol. 27, no. 3, pp. 206–215, 2022, doi: 10.35760/ik.2022.v27i3.7640. DOI: https://doi.org/10.35760/ik.2022.v27i3.7640
D. Prismala and I. K. D. Nuryana, “Analisis Opinion Mining Pada Topik Chatgpt di Aplikasi X Dengan Pendekatan Algoritma SVM Berbasis Lexicon,” J. Emerg. Inf. Syst. Bus. Intell., vol. 5, no. 3 SE-Articles, pp. 62–75, Jul. 2024, [Online]. Available: https://ejournal.unesa.ac.id/index.php/JEISBI/article/view/61575 DOI: https://doi.org/10.26740/jeisbi.v5i3.61575
M. Y. Pratama, U. A. Putri, P. A. D. Angraini, D. Puspita, and F. Kurniawan, “Analisis Sentimen Chat GPT sebagai Masa Depan Pekerja pada Media Sosial Youtube menggunakan Algoritma Klasifikasi Naive Bayes,” Explor. J. Sist. Inf. dan Telemat., vol. 14, no. 2, p. 193, 2023, doi: 10.36448/jsit.v14i2.3391. DOI: https://doi.org/10.36448/jsit.v14i2.3391
S. Joses, D. Yulvida, and S. Rochimah, “Pendekatan Metode Ensemble Learning untuk Prakiraan Cuaca menggunakan Soft Voting Classifier,” J. Appl. Comput. Sci. Technol., vol. 5, no. 1 SE-Articles, Jun. 2024, doi: 10.52158/jacost.v5i1.741. DOI: https://doi.org/10.52158/jacost.v5i1.741
Z. Ashktorab et al., “AI-Assisted Human Labeling: Batching for Efficiency without Overreliance,” Proc. ACM Human-Computer Interact., vol. 5, pp. 1–27, Apr. 2021, doi: 10.1145/3449163. DOI: https://doi.org/10.1145/3449163
I. T. Julianto, D. Kurniadi, B. B. B. Jr, and F. Rohman, “A Comparative Study of Alternative Automatic Labeling Using AI Assistant,” Sink. J. dan Penelit. Tek. Inform., vol. 8, no. 4, pp. 2125–2133, 2024, doi: https://doi.org/10.33395/sinkron.v8i4.13950. DOI: https://doi.org/10.33395/sinkron.v8i4.13950
Y. Asri and M. Fajri, “Sentiment Analysis of PLN Mobile Review Data Using Lexicon Vader and Naive Bayes Classification,” 2023 Int. Conf. Networking, Electr. Eng. Comput. Sci. Technol., pp. 132–137, 2023, [Online]. Available: https://api.semanticscholar.org/CorpusID:265484466 DOI: https://doi.org/10.1109/IConNECT56593.2023.10327064
T. Hidayat, R. Cahyana, and I. T. Julianto, “Analisis Sentimen Layanan Sistem Informasi Akademik Mahasiswa Menggunakan Algoritma Naive Bayes,” J. Algoritm., vol. 21, no. 1, pp. 119–130, 2024, doi: 10.33364/algoritma/v.21-1.1514. DOI: https://doi.org/10.33364/algoritma/v.21-1.1514
I. T. Julianto, “Analisis Sentimen Terhadap Sistem Informasi Akademik Institut Teknologi Garut,” J. Algoritm., vol. 19, no. 1, pp. 449–456, 2022, doi: 10.33364/algoritma/v.19-1.1112. DOI: https://doi.org/10.33364/algoritma/v.19-1.1112
A. Ardiyanto and E. I. Sela, “Analisis Sentimen Opini Mahasiswa Terhadap Aplikasi Portal Mahasiswa UTY Menggunakan Metode Naïve Bayes Classifier,” J. Indones. Manaj. Inform. dan Komun., vol. 5, no. 1, pp. 525–540, 2024, doi: 10.35870/jimik.v5i1.508. DOI: https://doi.org/10.35870/jimik.v5i1.508
A. Diahwahyuningtyas and A. N. Dzulfaroh, “Elon Musk Luncurkan Chatbot Grok-3, Diklaim Jadi AI Paling Canggih,” Kompas.com, 2025. https://www.kompas.com/tren/read/2025/02/19/203000065/elon-musk-luncurkan-chatbot-grok-3-diklaim-jadi-ai-paling-canggih
A. F. N. Azizah and V. P. Ramadhan, “Komparasi Naïve Bayes dan K-NN Dalam Analisis Sentimen di Twitter Terhadap Kemenangan Paslon 02,” J-INTECH (Journal Inf. Technol., no. 204, pp. 228–237, 2024. DOI: https://doi.org/10.32664/j-intech.v12i02.1305
D. R. P. Jaya and S. Lestari, “Analisis Sentimen Naturalisasi Tim Nasional Indonesia U-23 di Era Shin Tae-yong Menggunakan Algoritma Naïve Bayes,” J. Indones. Manaj. Inform. dan Komun., vol. 5, no. 3, pp. 3262–3277, 2024, doi: https://doi.org./10.35870/jimik.v5i3.1024. DOI: https://doi.org/10.35870/jimik.v5i3.1024
D. S. Nugroho, I. F. Hanif, M. A. Hasbi, F. Fredianto, A. M. Saputra, and R. Zildjian, “Analisis Sentimen Dugaan Pelanggaran Pemilu 2024 Berdasarkan Tweet Menggunakan Algoritma Naïve Bayes Classifier,” MALCOM Indones. J. Mach. Learn. Comput. Sci., vol. 4, no. 3, pp. 1169–1176, 2024, doi: 10.57152/malcom.v4i3.1496. DOI: https://doi.org/10.57152/malcom.v4i3.1496
A. Rifa’I, R. Ardhani, D. Pratama, and F. Fatihanursari, “Analisis Sentimen Terhadap Layanan Aplikasi Grab Indonesia Menggunakan Metode Naïve Bayes,” JATI (Jurnal Mhs. Tek. Inform., vol. 8, no. 1, pp. 303–309, 2024, doi: 10.36040/jati.v8i1.8425. DOI: https://doi.org/10.36040/jati.v8i1.8425
I. Amelia, A. Mutiara, and I. Santoso, “Analisis Sentimen Opini Publik Terhadap Pengambil Alihan Tmii Oleh Pemerintah Dengan Algoritma Naïve Bayes,” J. IKRAITH-INFORMATIKA, vol. 7, no. 2, pp. 142–148, 2023, [Online]. Available: https://journals.upi-yai.ac.id/index.php/ikraith-informatika/issue/archive
H. Berliana and R. Yusuf, “Analisis Sentimen Terhadap Penggunaan Donasi Korban Penyiraman Air Keras Pada Media Sosial X.com,” J. Sci. Soc. Res., vol. 8, no. 2, pp. 1134–1142, 2025, doi: https://doi.org/10.54314/jssr.v8i2.3078.
A. A. Patel and A. U. Arasanipalai, Applied Natural Language Processing in the Enterprise: Teaching Machines to Read, Write, and Understand, 1st ed., O’Reilly Media, 2021, p. 333.
E. Hvitfeldt and J. Silge, Supervised Machine Learning for Text Analysis in R, 1st ed., CRC Press, 2022. DOI: https://doi.org/10.1201/9781003093459
D. D. Lewis, “Naive (Bayes) at forty: The independence assumption in information retrieval,” in Machine Learning: ECML-98, C. Nédellec and C. Rouveirol, Eds., Berlin, Heidelberg: Springer, 1998, pp. 4–15. DOI: https://doi.org/10.1007/BFb0026666
H. D. Wijaya and S. Dwiasnati, “Implementasi Data Mining dengan Algoritma Naïve Bayes pada Penjualan Obat,” J. Inform., vol. 7, no. 1, pp. 1–7, 2020, doi: 10.31311/ji.v7i1.6203. DOI: https://doi.org/10.31311/ji.v7i1.6203
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Indri Tri Julianto, Muhammad Affan Al Sidqi
This work is licensed under a Creative Commons Attribution 4.0 International License.
License:
This is an open-access article distributed under the terms of the Creative Commons Attribution 4.0 International License (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.
