Review Article

Systematic analysis of generative AI tools integration in academic research and peer review

Husain Abdulrasool Salman 1 * , Muhammad Aliif Ahmad 2 , Roliana Ibrahim 2 , Jamilah Mahmood 2
More Detail
1 University of Bahrain, Zallaq, BAHRAIN2 Universiti Teknologi Malaysia, Johor Bahru, MALAYSIA* Corresponding Author
Online Journal of Communication and Media Technologies, 15(1), January 2025, e202502, https://doi.org/10.30935/ojcmt/15832
Published: 06 January 2025
OPEN ACCESS   172 Views   77 Downloads
Download Full Text (PDF)

ABSTRACT

While sparking a big debate among academics, generative artificial intelligence (GAI) tools are becoming integral to academic research, holding the potential to transform traditional research and peer review methods. This systematic literature review investigates the emergent role of GAI tools in academic research workflow and scholarly publications by analyzing 44 articles. The process of identifying the most relevant publications was done following the preferred reporting items for systematic reviews and meta-analyses method. The findings provide a thorough understanding of how GAI is currently being utilized in the various aspects of academic research workflow and peer review process, including concerns, limitations, and proactive measures to better employ these tools effectively. Our review suggests the need for more research to develop appropriate policies and guidelines, enhance researchers’ artificial intelligence literacy through targeted training, and ensure ethical use of these tools to boost research productivity and quality.
Keywords: generative AI, chatbot, ChatGPT, academic research, scientific writing, peer review

CITATION (APA)

Salman, H. A., Ahmad, M. A., Ibrahim, R., & Mahmood, J. (2025). Systematic analysis of generative AI tools integration in academic research and peer review. Online Journal of Communication and Media Technologies, 15(1), e202502. https://doi.org/10.30935/ojcmt/15832

REFERENCES

  1. Abdelhafiz, A. S., Ali, A., Maaly, A. M., Ziady, H. H., Sultan, E. A., & Mahgoub, M. A. (2024). Knowledge, perceptions and attitude of researchers towards using ChatGPT in research. Journal of Medical Systems, 48, Article 26. https://doi.org/10.1007/s10916-024-02044-4
  2. Abuyaman, O. (2023). Strengths and weaknesses of ChatGPT models for scientific writing about medical vitamin B12: Mixed methods study. JMIR Formative Research, 7, e49459. https://doi.org/10.2196/49459
  3. Aiumtrakul, N., Thongprayoon, C., Suppadungsuk, S., Krisanapan, P., Miao, J., Qureshi, F., & Cheungpasitporn, W. (2023). Navigating the landscape of personalized medicine: The relevance of ChatGPT, BingChat, and Bard AI in nephrology literature searches. Journal of Personalized Medicine, 13(10), Article 1457. https://doi.org/10.3390/jpm13101457
  4. Alkaissi, H., & McFarlane, S. I. (2023). Artificial hallucinations in ChatGPT: Implications in scientific writing. Cureus, 15(2), Article e35179. https://doi.org/10.7759/cureus.35179
  5. Alshami, A., Elsayed, M., Ali, E., Eltoukhy, A. E., & Zayed, T. (2023). Harnessing the power of ChatGPT for automating systematic review process: Methodology, case study, limitations, and future directions. Systems, 11, Article 351. https://doi.org/10.3390/systems11070351
  6. Alyasiri, O. M., Salman, A. M., & Salisu, S. (2024). ChatGPT revisited: Using ChatGPT-4 for finding references and editing language in medical scientific articles. Journal of Stomatology, Oral and Maxillofacial Surgery, 125(5), Article 101842. https://doi.org/10.1016/j.jormas.2024.101842
  7. Anghelescu, A., Ciobanu, I., Munteanu, C., Anghelescu, L. A. M., & Onose, G. (2023). ChatGPT: “To be or not to be” … in academic research. the human mind’s analytical rigor and capacity to discriminate between ai bots’ truths and hallucinations. Balneo and PRM Research Journal, 14, Article 4. https://doi.org/10.12680/balneo.2023.614
  8. Ariyaratne, S., Iyengar, K. P., Nischal, N., Chitti Babu, N., & Botchu, R. (2023). A comparison of ChatGPT-generated articles with human-written articles. Skeletal Radiology, 52, 1755–1758. https://doi.org/10.1007/s00256-023-04340-5
  9. Athaluri, S. A., Manthena, S. V., Kesapragada, V. S. R. K. M., Yarlagadda, V., Dave, T., & Duddumpudi, R. T. S. (2023). Exploring the boundaries of reality: Investigating the phenomenon of artificial intelligence hallucination in scientific writing through ChatGPT references. Cureus, 15(4), Article e37432. https://doi.org/10.7759/cureus.37432
  10. Aydin, A., Yürük, S. E., Reisoğlu, İ., & Goktas, Y. (2022). Main barriers and possible enablers of academicians while publishing. Scientometrics, 128, 623–650. https://doi.org/10.1007/s11192-022-04528-x
  11. Behera, R. K., Bala, P. K., & Dhir, A. (2019). The emerging role of cognitive computing in healthcare: A systematic literature review. International Journal of Medical Informatics, 129, 154–166. https://doi.org/10.1016/j.ijmedinf.2019.04.024
  12. Bekker, M. (2024). Large language models and academic writing: Five tiers of engagement. South African Journal of Science, 120(1/2). https://doi.org/10.17159/sajs.2024/17147
  13. Biswas, S., Dobaria, D., & Cohen, H. L. (2023). ChatGPT and the future of journal reviews: A feasibility study. Yale Journal of Biology and Medicine, 96, 415–420. https://doi.org/10.59249/SKDH9286
  14. Bond, A., Cilliers, D., Retief, F., Alberts, R., Roos, C., & Moolman, J. (2024). Using an artificial intelligence chatbot to critically review the scientific literature on the use of artificial intelligence in environmental impact assessment. Impact Assessment and Project Appraisal, 42(2), 189–199. https://doi.org/10.1080/14615517.2024.2320591
  15. Burger, B., Kanbach, D. K., Kraus, S., Breier, M., & Corvello, V. (2023). On the use of AI-based tools like ChatGPT to support management research. European Journal of Innovation Management, 26, 233–241. https://doi.org/10.1108/EJIM-02-2023-0156
  16. Carabantes, D., González-Geraldo, J. L., & Jover, G. (2023). ChatGPT could be the reviewer of your next scientific paper. Evidence on the limits of AI-assisted academic reviews. El Profesional de la Informacion, 32(5). https://doi.org/10.3145/epi.2023.sep.16
  17. Checco, A., Bracciale, L., Loreti, P., Pinfield, S., & Bianchi, G. (2021). AI-assisted peer review. Humanities and Social Sciences Communications, 8, Article 25. https://doi.org/10.1057/s41599-020-00703-8
  18. Chen, T., Gascó-Hernandez, M., & Esteve, M. (2024). The adoption and implementation of artificial intelligence chatbots in public organizations: Evidence from US state governments. The American Review of Public Administration, 54, 255–270. https://doi.org/10.1177/02750740231200522
  19. Crawford, J., Cowling, M., Ashton-Hay, S., Kelder, J.-A., Middleton, R., & Wilson, G. (2023). Artificial intelligence and authorship editor policy: ChatGPT, Bard Bing AI, and beyond. Journal of University Teaching and Learning Practice, 20(5). https://doi.org/10.53761/1.20.5.01
  20. Dashti, M., Londono, J., Ghasemi, S., & Moghaddasi, N. (2023). How much can we rely on artificial intelligence chatbots such as the ChatGPT software program to assist with scientific writing? The Journal of Prosthetic Dentistry. https://doi.org/10.1016/j.prosdent.2023.05.023
  21. Datt, M., Sharma, H., Aggarwal, N., & Sharma, S. (2023). Role of ChatGPT-4 for medical researchers. Annals of Biomedical Engineering, 52, 1534–1536. https://doi.org/10.1007/s10439-023-03336-5
  22. Dengel, A., Gehrlein, R., Fernes, D., Görlich, S., Maurer, J., Pham, H. H., Großmann, G., & Eisermann, N. D. G. (2023). Qualitative research methods for large language models: Conducting semi-structured interviews with ChatGPT and BARD on computer science education. Informatics, 10(4), Article 78. https://doi.org/10.3390/informatics10040078
  23. Dergaa, I., Chamari, K., Zmijewski, P., & Ben Saad, H. (2023). From human writing to artificial intelligence generated text: Examining the prospects and potential threats of ChatGPT in academic writing. Biology of Sport, 40, 615–622. https://doi.org/10.5114/biolsport.2023.125623
  24. Dhanvijay, A. K. D., Pinjar, M. J., Dhokane, N., Sorte, S. R., Kumari, A., & Mondal, H. (2023). Performance of large language models (ChatGPT, Bing Search, and Google Bard) in solving case vignettes in physiology. Cureus, 15(8), Article e42972. https://doi.org/10.7759/cureus.42972
  25. Ebert, C., & Louridas, P. (2023). Generative AI for software practitioners. IEEE Software, 40, 30–38. https://doi.org/10.1109/MS.2023.3265877
  26. Eigenmann, P., Akenroye, A., Markovic, M. A., Candotti, F., Ebisawa, M., Genuneit, J., Kalayci, Ö., Kollmann, D., Leung, A. S. Y., & Peters, R. L. (2023). Pediatric allergy and immunology (PAI) is for polishing with artificial intelligence, but careful use. Pediatric Allergy and Immunology, 34(9), Article e14023. https://doi.org/10.1111/pai.14023
  27. Firat, M. (2023). What ChatGPT means for universities: Perceptions of scholars and students. Journal of Applied Learning & Teaching, 6(1), 1–22. https://doi.org/10.37074/jalt.2023.6.1.22
  28. Gupta, R., Park, J. B., Bisht, C., Herzog, I., Weisberger, J., Chao, J., Chaiyasate, K., & Lee, E. S. (2023). Expanding cosmetic plastic surgery research with ChatGPT. Aesthetic Surgery Journal, 3, 930–937. https://doi.org/10.1093/asj/sjad069
  29. Hadi, M. U., Qureshi, R., Shah, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Wu, J., & Mirjalili, S. (2023). A survey on large language models: Applications, challenges, limitations, and practical usage. TechRxiv. https://doi.org/10.36227/techrxiv.23589741.v1
  30. Hake, J., Crowley, M., Coy, A., Shanks, D., Eoff, A., Kirmer-Voss, K., Dhanda, G., & Parente, D. J. (2024). Quality, accuracy, and bias in ChatGPT-based summarization of medical abstracts. Annals of Family Medicine, 22, 113–120. https://doi.org/10.1370/afm.3075
  31. Hamamah, Emaliana, I., Hapsari, Y., Degeng, P. D. D., & Fadillah, A. C. (2023). Using nominal group technique to explore publication challenges and the usefulness of AI-based writing technologies: Insights from Indonesian scholars. Theory and Practice in Language Studies, 13(8), 2038–2047. https://doi.org/10.17507/tpls.1308.20
  32. Hosseini, M., & Horbach, S. P. J. M. (2023). Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review. Research Integrity and Peer Review, 8, Article 4. https://doi.org/10.1186/s41073-023-00133-5
  33. Huang, J., & Tan, M. (2023). The role of ChatGPT in scientific communication: Writing better scientific review articles. American Journal of Cancer Research, 13, Article 1148.
  34. Imran, M., & Almusharraf, N. (2023). Analyzing the role of ChatGPT as a writing assistant at higher education level: A systematic review of the literature. Contemporary Educational Technology, 15(4), Article ep464. https://doi.org/10.30935/cedtech/13605
  35. Jarrah, A. M., Wardat, Y., & Fidalgo, P. (2023). Using ChatGPT in academic writing is (not) a form of plagiarism: What does the literature say? Online Journal of Communication and Media Technologies, 13(4), Article e202346. https://doi.org/10.30935/ojcmt/13572
  36. Jenko, N., Ariyaratne, S., Jeys, L., Evans, S., Iyengar, K., & Botchu, R. (2024). An evaluation of AI generated literature reviews in musculoskeletal radiology. The Surgeon, 22(3), 194–197. https://doi.org/10.1016/j.surge.2023.12.005
  37. Keele, S. (2007). Guidelines for performing systematic literature reviews in software engineering. EBSE.
  38. Khalifa, A. A., & Ibrahim, M. A. (2024). Artificial intelligence (AI) and ChatGPT involvement in scientific and medical writing, a new concern for researchers. A scoping review. Arab Gulf Journal of Scientific Research, 42(4), 1770–1787. https://doi.org/10.1108/AGJSR-09-2023-0423
  39. Khalifa, M., & Albadawy, M. (2024). Using artificial intelligence in academic writing and research: An essential productivity tool. Computer Methods and Programs in Biomedicine Update, 5, Article 100145. https://doi.org/10.1016/j.cmpbup.2024.100145
  40. Khlaif, Z. N., Mousa, A., Hattab, M. K., Itmazi, J., Hassan, A. A., Sanmugam, M., & Ayyoub, A. (2023). The potential and concerns of using AI in scientific research: ChatGPT performance evaluation. JMIR Medical Education, 9, Article e47049. https://doi.org/10.2196/47049
  41. Kraus, S., Breier, M., & Dasí-Rodríguez, S. (2020). The art of crafting a systematic literature review in entrepreneurship research. International Entrepreneurship and Management Journal, 16, 1023–1042. https://doi.org/10.1007/s11365-020-00635-4
  42. Leong, A. P. (2023). Clause complexing in research-article abstracts: Comparing human- and AI-generated texts. ExELL, 11(2), 9–132. https://doi.org/10.2478/exell-2023-0008
  43. Livberber, T. (2023). Toward non-human-centered design: Designing an academic article with ChatGPT. Profesional de la Informacion, 32(5). https://doi.org/10.3145/epi.2023.sep.12
  44. Lozić, E., & Štular, B. (2023). Fluent but not factual: A comparative analysis of ChatGPT and other AI chatbots’ proficiency and originality in scientific writing for humanities. Future Internet, 15(10), Article 336. https://doi.org/10.3390/fi15100336
  45. Mahyoob, M., Algaraady, J., & Alblwi, A. (2023). A proposed framework for human-like language processing of ChatGPT in academic writing. International Journal of Emerging Technologies in Learning, 18(14), 282–293. https://doi.org/10.3991/ijet.v18i14.41725
  46. Májovský, M., Černý, M., Kasal, M., Komarc, M., & Netuka, D. (2023). Artificial intelligence can generate fraudulent but authentic-looking scientific medical articles: Pandora’s box has been opened. Journal of Medical Internet Research, 25, Article e46924. https://doi.org/10.2196/46924
  47. Margetts, T. J., Karnik, S. J., Wang, H. S., Plotkin, L. I., Oblak, A. L., Fehrenbacher, J. C., Kacena, M. A., & Movila, A. (2024). Use of AI language engine ChatGPT 4.0 to write a scientific review article examining the intersection of Alzheimer’s disease and bone. Current Osteoporosis Reports, 22, 177–181. https://doi.org/10.1007/s11914-023-00853-z
  48. Mehta, J. (2023). Growing importance of academic research in education. EPRA International Journal of Research & Development, 8, 261–264.
  49. Mohamed Shaffril, H. A., Samsuddin, S. F., & Abu Samah, A. (2021). The ABC of systematic literature review: The basic methodological guidance for beginners. Quality & Quantity, 55, 1319–1346. https://doi.org/10.1007/s11135-020-01059-6
  50. Mohammed Alyasiri, O., Salman, A. M., Akhtom, D., & Salisu, S. (2024). ChatGPT revisited: Using ChatGPT-4 for finding references and editing language in medical scientific articles. Journal of Stomatology, Oral and Maxillofacial Surgery, 125(5, Supplement 2), Article 101842. https://doi.org/10.1016/j.jormas.2024.101842
  51. Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., Stewart, L. A., & Prisma-P Group. (2015). Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Systematic Reviews, 4(1), Article 1. https://doi.org/10.1186/2046-4053-4-1
  52. Mollaki, V. (2024). Death of a reviewer or death of peer review integrity? the challenges of using AI tools in peer reviewing and the need to go beyond publishing policies. Research Ethics, 20(2), 239–250. https://doi.org/10.1177/17470161231224552
  53. Mondal, H., & Mondal, S. (2023). ChatGPT in academic writing: Maximizing its benefits and minimizing the risks. Indian Journal of Ophthalmology, 71(12), 3600–3606. https://doi.org/10.4103/IJO.IJO_718_23
  54. Morocco-Clarke, A., Sodangi, F. A., & Momodu, F. (2023). The implications and effects of ChatGPT on academic scholarship and authorship: A death knell for original academic publications? Information & Communications Technology Law, 33(1), 21–41. https://doi.org/10.1080/13600834.2023.2239623
  55. Nakavachara, V., Potipiti, T., & Chaiwat, T. (2024). Experimenting with generative AI: Does ChatGPT really increase everyone’s productivity? ArXiv. https://doi.org/10.2139/ssrn.4746770
  56. Nazzal, M. K., Morris, A. J., Parker, R. S., White, F. A., Natoli, R. M., Fehrenbacher, J. C., & Kacena, M. A. (2024). Using AI to write a review article examining the role of the nervous system on skeletal homeostasis and fracture healing. Current Osteoporosis Reports, 22, 217–221. https://doi.org/10.1007/s11914-023-00854-y
  57. Nguyen, A., Hong, Y., Dang, B., & Huang, X. (2024). Human-AI collaboration patterns in AI-assisted academic writing. Studies in Higher Education, 49(5), 847–864. https://doi.org/10.1080/03075079.2024.2323593
  58. Olujimi, P. A., & Ade-Ibijola, A. (2023). NLP techniques for automating responses to customer queries: A systematic review. Discover Artificial Intelligence, 3, Article 20. https://doi.org/10.1007/s44163-023-00065-5
  59. Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J., Grimshaw, J. M., Hrobjartsson, A., Lalu, M. M., Li, T., Loder, E. W., Mayo-Wilson, E., McDonald, S., …, & Moher, D. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ, 372, Article n71. https://doi.org/10.1136/bmj.n71
  60. Patil, N. S., Huang, R. S., van der Pol, C. B., & Larocque, N. (2024). Comparative performance of ChatGPT and bard in a text-based radiology knowledge assessment. Canadian Association of Radiologists Journal, 75(2), 344–350. https://doi.org/10.1177/08465371231193716
  61. Pavlik, J. V. (2023). Collaborating with ChatGPT: Considering the implications of generative artificial intelligence for journalism and media education. Journalism & Mass Communication Educator, 78, 84–93. https://doi.org/10.1177/10776958221149577
  62. Pérez-Núñez, A. (2023). Exploring the potential of generative AI (ChatGPT) for foreign language instruction: Applications and challenges. Hispania, 106, 355–362. https://doi.org/10.1353/hpn.2023.a906568
  63. Perkins, M., & Roe, J. (2023). Academic publisher guidelines on AI usage: A ChatGPT supported thematic analysis. F1000Research, 12, Article 1398. https://doi.org/10.12688/f1000research.142411.2
  64. Pesovski, I., Santos, R., Henriques, R., & Trajkovik, V. (2024). Generative AI for customizable learning experiences. Sustainability, 16, 3034. https://doi.org/10.3390/su16073034
  65. Praveen, S. V., & Vajrobol, V. (2023). Understanding the perceptions of healthcare researchers regarding ChatGPT: A study based on bidirectional encoder representation from transformers (BERT) sentiment analysis and topic modeling. Annals of Biomedical Engineering, 51, 1654–1656. https://doi.org/10.1007/s10439-023-03222-0
  66. Rane, N., Choudhary, S., & Rane, J. (2024). Intelligent manufacturing through generative artificial intelligence, such as ChatGPT or Bard. SSRN. https://doi.org/10.2139/ssrn.4681747
  67. Rospigliosi, P. (2023). Artificial intelligence in teaching and learning: What questions should we ask of ChatGPT? Interactive Learning Environments, 31(1), 1–3. https://doi.org/10.1080/10494820.2023.2180191
  68. Saad, A., Jenko, N., Ariyaratne, S., Birch, N., Iyengar, K. P., Davies, A. M., Vaishya, R., & Botchu, R. (2024). Exploring the potential of ChatGPT in the peer review process: An observational study. Diabetes & Metabolic Syndrome: Clinical Research & Reviews, 18(2), Article 102946. https://doi.org/10.1016/j.dsx.2024.102946
  69. Salimi, A., & Saheb, H. (2023). Large language models in ophthalmology scientific writing: Ethical considerations blurred lines or not at all? American Journal of Ophthalmology, 254, 177–181. https://doi.org/10.1016/j.ajo.2023.06.004
  70. Salman, H., Al Mohsin, E., Al Rawi, A., & Shatnawi, S. (2022). Investigating HCI of the LMS blackboard ultra using WAMMI during COVID-19: Usability and design interactivity. In Proceedings of the 2022 International Conference on Innovation and Intelligence for Informatics, Computing, and Technologies (pp. 519–525). IEEE. https://doi.org/10.1109/3ICT56508.2022.9990650
  71. Schmidt, L., Piazza, A., & Wiedenhöft, C. (2023). “Augmented brainstorming with AI”–Research approach for identifying design criteria for improved collaborative idea generation between humans and AI. In Frontiers in artificial intelligence and applications (pp. 410–412). IOS Press. https://doi.org/10.3233/FAIA230113
  72. Semrl, N., Feigl, S., Taumberger, N., Bracic, T., Fluhr, H., Blockeel, C., & Kollmann, M. (2023). AI language models in human reproduction research: Exploring ChatGPT’s potential to assist academic writing. Human Reproduction, 38(12), 2281–2288. https://doi.org/10.1093/humrep/dead207
  73. Seth, I., Sinkjær Kenney, P., Bulloch, G., Hunter-Smith, D. J., Bo Thomsen, J., & Rozen, W. M. (2023). Artificial or augmented authorship? A conversation with a chatbot on base of thumb arthritis. Plastic & Reconstructive Surgery-Global Open, 11(5), Article e4999. https://doi.org/10.1097/GOX.0000000000004999
  74. Victor, B. G., Sokol, R. L., Goldkind, L., & Perron, B. E. (2023). Recommendations for social work researchers and journal editors on the use of generative AI and large language models. Journal of the Society for Social Work and Research, 14, 563–577. https://doi.org/10.1086/726021
  75. Watermeyer, R., Phipps, L., Lanclos, D., & Knight, C. (2023). Generative AI and the automating of academia. Postdigital Science and Education, 6, 446–466. https://doi.org/10.1007/s42438-023-00440-6
  76. Wu, R. T., & Dang, R. R. (2023). ChatGPT in head and neck scientific writing: A precautionary anecdote. American Journal of Otolaryngology, 44(6), Article 103980. https://doi.org/10.1016/j.amjoto.2023.103980
  77. Yan, L., Echeverria, V., Nieto, G. F., Jin, Y., Swiecki, Z., Zhao, L., Gašević, D., & Martinez-Maldonado, R. (2023). Human-AI collaboration in thematic analysis using ChatGPT: A user study and design recommendations. ArXiv. https://doi.org/10.1145/3613905.3650732
  78. Zhang, P., & Kamel Boulos, M. N. (2023). Generative AI in medicine and healthcare: Promises, opportunities and challenges. Future Internet, 15(9), Article 286. https://doi.org/10.3390/fi15090286
  79. Zheng, Z., Zhang, O., Borgs, C., Chayes, J. T., & Yaghi, O. M. (2023). ChatGPT chemistry assistant for text mining and the prediction of MOF synthesis. Journal of the American Chemical Society, 145, 18048–18062. https://doi.org/10.1021/jacs.3c05819
  80. Zohouri, M., Sabzali, M., & Golmohammadi, A. (2024). Ethical considerations of ChatGPT-assisted article writing. Synesis, 16(1), 94–113.