Multimodal Deep Learning for Pneumonia Detection Using Wearable Sensors: Toward an Edge-Cloud Framework

Emmanuel Onwako Ibam; Johnson Bisi Oluwagbemi

doi:10.62411/jcta.14944

Authors

Emmanuel Onwako Ibam Federal University of Technology
Johnson Bisi Oluwagbemi McPherson University

DOI:

https://doi.org/10.62411/jcta.14944

Keywords:

Deep Learning, Edge–Cloud Computing, Explainable AI, Multimodal Learning, Pneumonia Detection, Remote Health Monitoring, Wearable Sensors, Wireless Health Systems

Abstract

Pneumonia remains a leading cause of morbidity and mortality worldwide, particularly in resource-limited settings and among elderly populations, where timely diagnosis and continuous monitoring are often constrained by limited clinical infrastructure. This study presents an edge–cloud–integrated framework for early pneumonia risk monitoring, leveraging multimodal wearable sensors and deep learning to support continuous short-duration monitoring. The proposed system is designed to operate in near real time under simulated deployment conditions, continuously acquiring and analyzing physiological signals (respiratory rate, heart rate, SpO₂, and body temperature) alongside event-driven acoustic biomarkers (cough sounds) within a distributed architecture. A lightweight edge module performs local signal preprocessing and anomaly triage, selectively transmitting salient information to a cloud-based multimodal deep learning model for refined risk estimation and interpretability analysis. The framework was evaluated using a multi-source dataset comprising public repositories (MIMIC-III and Coswara) and a clinically supervised wearable study conducted in two Nigerian hospitals, resulting in 718 hours of quality-controlled multimodal monitoring data. In a pooled multi-source evaluation, the system achieved an AUC of 0.95, while in a clinically realistic local-only evaluation, the AUC was 0.86, reflecting a consistent but preliminary diagnostic signal. These results highlight the importance of local data adaptation for real-world applicability and suggest that multimodal AI can provide meaningful early risk indicators under resource constraints. Beyond predictive performance, this work demonstrates the feasibility of integrating multimodal learning, edge–cloud computation, and explainable analytics into a deployment-aware, privacy-preserving monitoring framework for low-resource healthcare environments.

Author Biographies

Emmanuel Onwako Ibam, Federal University of Technology

Department of Information Technology, Federal University of Technology, Akure 110001, Ondo State, Nigeria

Johnson Bisi Oluwagbemi, McPherson University

Department of Computer Science, McPherson University, Seriki-Sotayo 340001, Ogun State, Nigeria

References

World Health Organization (WHO), “Pneumonia,” WHO Fact Sheets, 2021. https://www.who.int/health-topics/pneumonia#tab=tab_1

D. Bhattacharya et al., “Coswara: A respiratory sounds and symptoms dataset for remote screening of SARS-CoV-2 infection,” Sci. Data, vol. 10, no. 1, p. 397, Jun. 2023, doi: 10.1038/s41597-023-02266-0.

P. O. Adebayo, F. Basaky, and E. Osaghae, “Leveraging Variational Quantum-Classical Algorithms for Enhanced Lung Cancer Prediction,” J. Comput. Theor. Appl., vol. 2, no. 3, pp. 307–323, Dec. 2024, doi: 10.62411/jcta.10424.

C. Brown et al., “Exploring Automatic Diagnosis of COVID-19 from Crowdsourced Respiratory Sound Data,” in Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, Aug. 2020, pp. 3474–3484. doi: 10.1145/3394486.3412865.

A. B. Rashid, J. Asma, K. Barua, and D. Das, “An Enhanced Deep Learning Framework for Pneumonia Detection in Chest X-rays,” SN Comput. Sci., vol. 6, no. 5, p. 472, May 2025, doi: 10.1007/s42979-025-04017-x.

Z. Jia, H. Huth, W. Q. Teoh, S. Xu, B. Wood, and Z. T. H. Tse, “State of the Art Review of Wearable Devices for Respiratory Monitoring,” IEEE Access, vol. 13, pp. 18178–18190, 2025, doi: 10.1109/ACCESS.2025.3529437.

Y. Li et al., “Integrated wearable smart sensor system for real-time multi-parameter respiration health monitoring,” Cell Reports Phys. Sci., vol. 4, no. 1, p. 101191, Jan. 2023, doi: 10.1016/j.xcrp.2022.101191.

M. B. Teferi and L. A. Akinyemi, “Deep Learning-Based Cross-Cancer Morphological Analysis: Identifying Histopathological Patterns in Breast and Lung Cancer,” J. Futur. Artif. Intell. Technol., vol. 1, no. 3, pp. 235–248, Oct. 2024, doi: 10.62411/faith.3048-3719-36.

D. Lin, Y. F. Ji, J. A. A. McArt, and J. Li, “SynLS: A novel diffusion-transformer framework for generating high-quality wearable sensor time series data to enhance health monitoring,” biorxiv. May 15, 2025. doi: 10.1101/2025.05.11.653212.

S. R. Stahlschmidt, B. Ulfenborg, and J. Synnergren, “Multimodal deep learning for biomedical data fusion: a review,” Brief. Bioinform., vol. 23, no. 2, Mar. 2022, doi: 10.1093/bib/bbab569.

A. Kline et al., “Multimodal machine learning in precision health: A scoping review,” npj Digit. Med., vol. 5, no. 1, p. 171, Nov. 2022, doi: 10.1038/s41746-022-00712-8.

N. Fadul, M. F. Alaskar, K. B. Jillahi, and D. B. El-Khaled, “Generative AI in Healthcare: An Analytical Review of Models, Clinical Applications, and Decision-Support Implications,” J. Futur. Artif. Intell. Technol., vol. 2, no. 4, pp. 587–615, Dec. 2025, doi: 10.62411/faith.3048-3719-298.

J. B. Oluwagbemi, A. E. Mesioye, and R. S. Akinbo, “Depress-HybridNet: A Linguistic-Behavioral Hybrid Framework for Early and Accurate Depression Detection on Social Media,” J. Futur. Artif. Intell. Technol., vol. 2, no. 3, pp. 432–444, Sep. 2025, doi: 10.62411/faith.3048-3719-266.

F. Mohsen, H. Ali, N. El Hajj, and Z. Shah, “Artificial intelligence-based methods for fusion of electronic health records and imaging data,” Sci. Rep., vol. 12, no. 1, p. 17981, Oct. 2022, doi: 10.1038/s41598-022-22514-4.

M. Salvi et al., “Multi-modality approaches for medical support systems: A systematic review of the last decade,” Inf. Fusion, vol. 103, p. 102134, Mar. 2024, doi: 10.1016/j.inffus.2023.102134.

J. N. Acosta, G. J. Falcone, P. Rajpurkar, and E. J. Topol, “Multimodal biomedical AI,” Nat. Med., vol. 28, no. 9, pp. 1773–1784, Sep. 2022, doi: 10.1038/s41591-022-01981-2.

M. Liu, S. Suh, J. F. Vargas, B. Zhou, A. Grünerbl, and P. Lukowicz, “A Wearable Multi-modal Edge-Computing System for Real-Time Kitchen Activity Recognition,” in Communications in Computer and Information Science, 2025, pp. 132–145. doi: 10.1007/978-981-97-9003-6_9.

M. D. Toruner et al., “Artificial Intelligence-Driven Wireless Sensing for Health Management,” Bioengineering, vol. 12, no. 3, p. 244, Feb. 2025, doi: 10.3390/bioengineering12030244.

M. C. Kelvin-Agwu, B. O. Tomoh, and A. Y. Forkuo, “Development of AI-Assisted Wearable Devices for Early Detection of Respiratory Diseases,” J. Front. Multidiscip. Res., vol. 6, no. 1, pp. 64–72, 2025, doi: 10.54660/.IJFMR.2025.6.1.64-72.

H. Ren et al., “CheXMed: A multimodal learning algorithm for pneumonia detection in the elderly,” Inf. Sci. (Ny)., vol. 654, p. 119854, Jan. 2024, doi: 10.1016/j.ins.2023.119854.

H. Wang et al., “Developing and Validation of a Multimodal-Based Machine Learning Model for Diagnosis of Usual Interstitial Pneumonia,” Chest, Sep. 2025, doi: 10.1016/j.chest.2025.08.034.

M. S. Sunarjo, H. Gan, and D. R. I. M. Setiadi, “High-Performance Convolutional Neural Network Model to Identify COVID-19 in Medical Images,” J. Comput. Theor. Appl., vol. 1, no. 1, pp. 19–30, Aug. 2023, doi: 10.33633/jcta.v1i1.8936.

K. Pyar, “Segmentation Performance Analysis of Transfer Learning Models on X-Ray Pneumonia Images,” J. Futur. Artif. Intell. Technol., vol. 1, no. 1, pp. 64–74, Jun. 2024, doi: 10.62411/faith.2024-10.

F. S. Gomiasti, W. Warto, E. Kartikadarma, J. Gondohanindijo, and D. R. I. M. Setiadi, “Enhancing Lung Cancer Classification Effectiveness Through Hyperparameter-Tuned Support Vector Machine,” J. Comput. Theor. Appl., vol. 1, no. 4, pp. 396–406, Mar. 2024, doi: 10.62411/jcta.10106.

J. Li et al., “Diagnostic assistance method for RR-TB/MDR-TB patients under treatment based on CNN-LSTM,” Sci. Rep., vol. 15, no. 1, p. 38035, Oct. 2025, doi: 10.1038/s41598-025-21955-x.

J. Colin and N. Surantha, “Interpretable Deep Learning for Pneumonia Detection Using Chest X-Ray Images,” Information, vol. 16, no. 1, p. 53, Jan. 2025, doi: 10.3390/info16010053.

J. Xu and Y. Wang, “FMT:A Multimodal Pneumonia Detection Model Based on Stacking MOE Framework,” in 2025 8th International Conference on Information and Computer Technologies (ICICT), Mar. 2025, pp. 517–521. doi: 10.1109/ICICT64582.2025.00087.

S. Sharma and K. Guleria, “A Deep Learning based model for the Detection of Pneumonia from Chest X-Ray Images using VGG-16 and Neural Networks,” Procedia Comput. Sci., vol. 218, pp. 357–366, 2023, doi: 10.1016/j.procs.2023.01.018.

A. Rancea, I. Anghel, and T. Cioara, “Edge Computing in Healthcare: Innovations, Opportunities, and Challenges,” Futur. Internet, vol. 16, no. 9, p. 329, Sep. 2024, doi: 10.3390/fi16090329.

A. Ali, “Exploring the Transformative Potential of Technology in Overcoming Educational Disparities,” Int. J. Multidiscip. Sci. Arts, vol. 2, no. 1, Jul. 2023, doi: 10.47709/ijmdsa.v2i1.2559.

I. Sanches et al., “MIMIC-BP: A curated dataset for blood pressure estimation,” Sci. Data, vol. 11, no. 1, p. 1233, Nov. 2024, doi: 10.1038/s41597-024-04041-1.

N. Sharma et al., “Coswara -- A Database of Breathing, Cough, and Voice Sounds for COVID-19 Diagnosis,” arXiv. Aug. 11, 2020. doi: 10.21437/Interspeech.2020-2768.

Y. Song, Z. Liu, J. Wang, R. Tang, G. Duan, and J. Tan, “Multiscale Adversarial and Weighted Gradient Domain Adaptive Network for Data Scarcity Surface Defect Detection,” IEEE Trans. Instrum. Meas., vol. 70, pp. 1–10, 2021, doi: 10.1109/TIM.2021.3096284.

H. Lim, B. Kim, J. Choo, and S. Choi, “TTN: A Domain-Shift Aware Batch Normalization in Test-Time Adaptation,” arXiv. Feb. 18, 2023. [Online]. Available: http://arxiv.org/abs/2302.05155

B. Isgor and M. Koklu, “Lightweight Hybrid Model for Bone Fracture Detection Using MobileNetV2 Feature Extraction and Ensemble Learning,” J. Futur. Artif. Intell. Technol., vol. 2, no. 3, pp. 521–533, Dec. 2025, doi: 10.62411/faith.3048-3719-284.

A. O. Eboka et al., “Resolving Data Imbalance Using a Bi-Directional Long-Short Term Memory for Enhanced Diabetes Mellitus Detection,” J. Futur. Artif. Intell. Technol., vol. 2, no. 1, pp. 95–109, May 2025, doi: 10.62411/faith.3048-3719-73.

T. R. Noviandy, G. M. Idroes, and I. Hardi, “An Interpretable Machine Learning Strategy for Antimalarial Drug Discovery with LightGBM and SHAP,” J. Futur. Artif. Intell. Technol., vol. 1, no. 2, pp. 84–95, Aug. 2024, doi: 10.62411/faith.2024-16.

J. B. Oluwagbemi, O. V. Oyetayo, and E. O. Ibam, “SysFungiNet: A Multi-Omics Data Fusion Framework with Explainable AI for Bioactive Prioritization,” J. Futur. Artif. Intell. Technol., vol. 2, no. 4, pp. 661–679, Jan. 2026, doi: 10.62411/faith.3048-3719-304.

A. Hamza, W. Hussain, H. Iftikhar, A. Ahmad, and A. M. Shamim, “Evaluating Open-Source Machine Learning Project Quality Using SMOTE-Enhanced and Explainable ML/DL Models,” J. Comput. Theor. Appl., vol. 3, no. 2, pp. 206–222, Nov. 2025, doi: 10.62411/jcta.14793.