Behavioral Malware Detection via API Call Sequences: A Comparative Study of LSTM and Transformer Architectures Using NLP-Inspired Representations
DOI:
https://doi.org/10.62411/jcta.15811Keywords:
Behavioral malware detection, Cybersecurity, Deep learning, Natural language processing, Sequence modeling, Software security, Sustainable digital security, Vision transformerAbstract
The increasing sophistication of malware has rendered traditional signature-based detection methods insufficient, necessitating behavior-driven and adaptive analytical frameworks. This study presents a sequential deep learning framework that models system-level API call sequences as structured linguistic representations for behavioral malware detection. Unlike conventional comparative studies, this work systematically evaluates recurrent and attention-based architectures under controlled experimental conditions, with a particular focus on generalization performance and overfitting mitigation. Two neural architectures, a Long Short-Term Memory (LSTM) network and a Transformer-based attention model, are trained on publicly available API call sequence data for binary classification of malicious and benign executables. Beyond standard accuracy metrics, the study further examines model stability, convergence behavior, and the impact of long-range dependency modeling on detection robustness. Experimental results demonstrate that the Transformer architecture achieves superior performance, attaining 95.54% classification accuracy and consistent improvements in precision, recall, and F1-score, indicating a stronger ability to capture complex behavioral dependencies. These findings highlight the effectiveness of attention mechanisms in behavioral malware modeling and provide empirical evidence that NLP-inspired architectures offer a robust and scalable approach for real-world cybersecurity applications.References
A. Olivera, “Malware Analysis Datasets: API Call Sequences,” Kaggle.com, 2020. https://www.kaggle.com/datasets/ang3loliveira/malware-analysis-datasets-api-call-sequences
F. O. Catak, A. F. Yazı, O. Elezaj, and J. Ahmed, “Deep learning based Sequential model for malware analysis using Windows exe API Calls,” PeerJ Comput. Sci., vol. 6, p. e285, Jul. 2020, doi: 10.7717/peerj-cs.285.
M. S. Masari, M. A. Danladi, I. L. Onyinye, and L. K. Tohomdet, “Android Malware Detection Using Machine Learning with SMOTE-Tomek Data Balancing,” J. Comput. Theor. Appl., vol. 3, no. 3, pp. 302–313, Jan. 2026, doi: 10.62411/jcta.15084.
F. O. Catak and A. F. Yazı, “A Benchmark API Call Dataset for Windows PE Malware Classification,” ArXiv. Feb. 21, 2021. [Online]. Available: http://arxiv.org/abs/1905.01999
S. Aggarwal and F. Di Troia, “Malware Classification Using Dynamically Extracted API Call Embeddings,” Appl. Sci., vol. 14, no. 13, p. 5731, Jun. 2024, doi: 10.3390/app14135731.
A. H. Alhazmi, “A robust and dynamic malware detection and classification model using behavioral-based analysis and BERT technique,” PLoS One, vol. 20, no. 9, p. e0327604, Sep. 2025, doi: 10.1371/journal.pone.0327604.
A. S. Kale, F. Di Troia, and M. Stamp, “Malware Classification with Word Embedding Features,” ArXiv. Mar. 03, 2021. [Online]. Available: http://arxiv.org/abs/2103.02711
P. H. Hussan and S. M. Mangj, “BERTPHIURL: A Teacher-Student Learning Approach Using DistilRoBERTa and RoBERTa for Detecting Phishing Cyber URLs,” J. Futur. Artif. Intell. Technol., vol. 1, no. 4, pp. 417–428, Feb. 2025, doi: 10.62411/faith.3048-3719-71.
X. Wang and S. M. Yiu, “A multi-task learning model for malware classification with useful file access pattern from API call sequence,” ArXiv. Oct. 19, 2016. [Online]. Available: http://arxiv.org/abs/1610.05945
B. Kolosnjaji, A. Zarras, G. Webster, and C. Eckert, “Deep Learning for Classification of Malware System Call Sequences,” in Lecture Notes in Computer Science, 2016, pp. 137–149. doi: 10.1007/978-3-319-50127-7_11.
B. K. Mamade and D. M. Dabala, “Exploring The Correlation between Cyber Security Awareness, Protection Measures and the State of Victimhood: The Case Study of Ambo University’s Academic Staffs,” J. Cyber Secur. Mobil., Jun. 2021, doi: 10.13052/jcsm2245-1439.1044.
M. A. Rahman, G. A. Francia, and H. Shahriar, “Leveraging GANs for Synthetic Data Generation to Improve Intrusion Detection Systems,” J. Futur. Artif. Intell. Technol., vol. 1, no. 4, pp. 429–439, Feb. 2025, doi: 10.62411/faith.3048-3719-52.
M. Ahmed, A. Qureshi, J. Ahmed Shamsi, and M. Marvi, “Sequential Embedding-based Attentive (SEA) classifier for malware classification,” in 2022 International Conference on Cyber Warfare and Security (ICCWS), Dec. 2022, pp. 28–35. doi: 10.1109/ICCWS56285.2022.9998431.
A. Cannarile, F. Carrera, S. Galantucci, A. Iannacone, and G. Pirlo, “A study on malware detection and classification using the analysis of API calls sequences through shallow learning and recurrent neural networks,” CEUR Workshop Proc., vol. 3260, pp. 124–134, 2022.
Z. Zhang, P. Qi, and W. Wang, “Dynamic Malware Analysis with Feature Engineering and Feature Learning,” Proc. AAAI Conf. Artif. Intell., vol. 34, no. 01, pp. 1210–1217, Apr. 2020, doi: 10.1609/aaai.v34i01.5474.
C. Avci, B. Tekinerdogan, and C. Catal, “Analyzing the performance of long short‐term memory architectures for malware detection models,” Concurr. Comput. Pract. Exp., vol. 35, no. 6, pp. 1–1, Mar. 2023, doi: 10.1002/cpe.7581.
T. Quertier, B. Marais, G. Barrué, S. Morucci, S. Azé, and S. Salladin, “A Lean Transformer Model for Dynamic Malware Analysis and Detection,” ArXiv. Aug. 05, 2024. [Online]. Available: http://arxiv.org/abs/2408.02313
B. Marais, T. Quertier, and G. Barrue, “Semantic Preprocessing for LLM-based Malware Analysis,” ArXiv. Oct. 03, 2025. [Online]. Available: http://arxiv.org/abs/2506.12113
Jobanpreet Kaur et al., “Comparative Analysis of Transformer and LSTM Architectures for Cybersecurity Threat Detection Using Machine Learning,” EAI Endorsed Trans. AI Robot., vol. 4, Sep. 2025, doi: 10.4108/airo.9759.
A. Rahali and M. A. Akhloufi, “MalBERT: Using Transformers for Cybersecurity and Malicious Software Detection,” ArXiv. Mar. 05, 2021. [Online]. Available: http://arxiv.org/abs/2103.03806
A. Walker and S. Sengupta, “Malware Family Fingerprinting Through Behavioral Analysis,” in 2020 IEEE International Conference on Intelligence and Security Informatics (ISI), Nov. 2020, pp. 1–5. doi: 10.1109/ISI49825.2020.9280529.
A. P. Binitie, S. I. Onyemenem, N. C. Anujeonye, A. A. Ojugo, F. A. Egbokhare, and T. C. Aghaunor, “A Graph-Augmented Isolation Forest Using Node2Vec and GraphSAGE for Mobile User Behavior Anomaly Detection,” J. Comput. Theor. Appl., vol. 3, no. 3, pp. 369–383, Feb. 2026, doi: 10.62411/jcta.15494.
F. Demirkıran, A. Çayır, U. Ünal, and H. Dağ, “An ensemble of pre-trained transformer models for imbalanced multiclass malware classification,” Comput. Secur., vol. 121, p. 102846, Oct. 2022, doi: 10.1016/j.cose.2022.102846.
P. Gysel, C. Wüest, K. Nwafor, O. Jašek, A. Ustyuzhanin, and D. M. Divakaran, “EAGLEEYE: Attention to Unveil Malicious Event Sequences From Provenance Graphs,” in 2024 APWG Symposium on Electronic Crime Research (eCrime), Sep. 2024, pp. 27–42. doi: 10.1109/eCrime66200.2024.00009.
M. Gao, P. Wu, and L. Pan, “MINES: Multi-perspective API Call Sequence Behavior Fusion Malware Classification,” in Lecture Notes in Computer Science, 2024, pp. 210–220. doi: 10.1007/978-981-97-5562-2_13.
S. Abbas, S. Amjad, S. Craß, and S. A. Moeinzadeh Mirhosseini, “Analysis of Blockchain-IoT Connection Patterns based on Clients Type,” in 2024 IEEE International Conferences on Internet of Things (iThings) and IEEE Green Computing & Communications (GreenCom) and IEEE Cyber, Physical & Social Computing (CPSCom) and IEEE Smart Data (SmartData) and IEEE Congress on Cybermatics, Aug. 2024, pp. 579–586. doi: 10.1109/iThings-GreenCom-CPSCom-SmartData-Cybermatics62450.2024.00107.
R. Kharsae, F. Kurugollu, A. Anjum, A. Amira, and A. Bouridane, “Malware Family Classification with Explainable BERT (xBERT) Using API Calls,” in 2024 IEEE/ACM International Conference on Big Data Computing, Applications and Technologies (BDCAT), Dec. 2024, pp. 324–333. doi: 10.1109/BDCAT63179.2024.00057.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Anusree K J, Narottam Das Patel, Saravanan D, Adarsh Patel
This work is licensed under a Creative Commons Attribution 4.0 International License.
