When Similar Models Disagree: Model Multiplicity and Geographic Robustness in School Meal Utilization Prediction
DOI:
https://doi.org/10.62411/jcta.18209Keywords:
Geographic Robustness, Grouped Validation, Model Multiplicity, Model-Choice Robustness, Probability Calibration, Scenario Analysis, School Feeding, School Meal UtilizationAbstract
Competing predictive models can achieve similar aggregate performance while producing different classifications and downstream conclusions for the same individuals. This model disagreement raises an important question about whether conventional performance metrics adequately capture the reliability of model-based analyses. This study investigates model-choice disagreement across multiple analytical layers using voluntary-response survey data from 8,802 respondents who reported receiving meals through Indonesia’s Free Nutritious Meals program. Logistic Regression (LR), Random Forest, and XGBoost were evaluated using a nested school-disjoint validation framework. All three models were compared in terms of discrimination and probability calibration, while the competitive models were further examined through high-risk classification, predictive attribution, model-based scenario analysis, and province-holdout robustness. LR and XGBoost achieved nearly identical PR-AUC values (0.7270 and 0.7277, respectively), with similarly close pooled PR-AUC under province holdout (0.7288 and 0.7285). Despite this aggregate similarity, the models assigned different high-risk classifications to 5.41% of respondents. Their scenario analyses also produced different threshold-crossing rates: 8.87% of eligible respondents under LR (382 of 4,307) and 6.66% under XGBoost. Random Forest exhibited weaker probability calibration in the fitted configuration and was not retained for downstream comparison. Overall, the findings reveal that model agreement at the population level does not necessarily extend to individual classifications and scenario outputs. LR was selected as the reference model through documented judgment considering predictive performance, calibration, robustness, parsimony, and interpretability. These results demonstrate the importance of evaluating model-choice robustness beyond conventional predictive performance metrics, particularly when model outputs inform downstream interpretation or classification.References
C. T. Marx, F. du P. Calmon, and B. Ustun, “Predictive Multiplicity in Classification,” in Proceedings of the 37th International Conference on Machine Learning, Sep. 2020, vol. 119, pp. 6765–6774. [Online]. Available: https://proceedings.mlr.press/v119/marx20a.html
A. D’Amour et al., “Underspecification Presents Challenges for Credibility in Modern Machine Learning,” J. Mach. Learn. Res., vol. 23, no. 226, pp. 1–61, Nov. 2022, [Online]. Available: https://www.jmlr.org/papers/v23/20-1335.html
E. Christodoulou, J. Ma, G. S. Collins, E. W. Steyerberg, J. Y. Verbakel, and B. Van Calster, “A systematic review shows no per-formance benefit of machine learning over logistic regression for clinical prediction models,” J. Clin. Epidemiol., vol. 110, pp. 12–22, Jun. 2019, doi: 10.1016/j.jclinepi.2019.02.004.
C. Rudin, “Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead,” Nat. Mach. Intell., vol. 1, no. 5, pp. 206–215, May 2019, doi: 10.1038/s42256-019-0048-x.
J. S. Paige et al., “Variability Among Breast Cancer Risk Classification Models When Applied at the Level of the Individual Woman,” J. Gen. Intern. Med., vol. 38, Feb. 2023, doi: 10.1007/s11606-023-08043-4.
J. Zhang and T. Li, “Discrimination stability and calibration of cardiovascular risk prediction models in the Framingham baseline cohort,” Sci. Rep., vol. 16, no. 1, p. 23965, May 2026, doi: 10.1038/s41598-026-54869-3.
M. Pawelczyk, K. Broelemann, and G. Kasneci, “On Counterfactual Explanations under Predictive Multiplicity,” Proceedings of the 36th Conference on Uncertainty in Artificial Intelligence, vol. 124. pp. 809–818, Jun. 23, 2020. doi: 10.48550/arXiv.2006.13132.
S. Saeb, L. Lonini, A. Jayaraman, D. C. Mohr, and K. P. Kording, “The need to approximate the use-case in clinical machine learning,” Gigascience, vol. 6, no. 5, p. gix019, May 2017, doi: 10.1093/gigascience/gix019.
B. Van Calster, D. J. McLernon, M. van Smeden, L. Wynants, and E. W. Steyerberg, “Calibration: the Achilles heel of predictive analytics,” BMC Med., vol. 17, no. 1, p. 230, Dec. 2019, doi: 10.1186/s12916-019-1466-7.
T. Takada et al., “Internal-external cross-validation helped to evaluate the generalizability of prediction models in large clustered datasets,” J. Clin. Epidemiol., vol. 137, pp. 83–91, Sep. 2021, doi: 10.1016/j.jclinepi.2021.03.025.
E. W. Steyerberg and F. E. Harrell, “Prediction models need appropriate internal, internal–external, and external validation,” J. Clin. Epidemiol., vol. 69, pp. 245–247, Jan. 2016, doi: 10.1016/j.jclinepi.2015.04.005.
R. Saptati, “Pemerintah Salurkan Makan Bergizi Gratis (MBG), Ini Sasaran Utama Penerimanya,” Media Keuangan Kemenkeu, Feb. 2025. https://mediakeuangan.kemenkeu.go.id/article/show/pemerintah-salurkan-makan-bergizi-gratis-mbg-ini-sasaran-utama-penerimanya
C. Sobek et al., “Purchasing Behavior, Setting, Pricing, Family: Determinants of School Lunch Participation,” Nutrients, vol. 13, no. 12, p. 4209, Nov. 2021, doi: 10.3390/nu13124209.
L. A. Spruance and T. Vo, “Race/Ethnicity Predicts School Meal Participation,” J. Sch. Health, vol. 93, no. 4, pp. 305–312, Apr. 2023, doi: 10.1111/josh.13270.
L. A. Spruance, D. Santamaria, and S. Hall, “Parent Perceptions of School Meals as Predictors of Participation Among Pacific Is-lander Families in the United States,” J. Sch. Health, vol. 96, no. 6, p. e70169, Jun. 2026, doi: 10.1111/josh.70169.
M. D. Zuercher et al., “Student Perceptions of School Lunch and Their Association With Self-Reported Participation in School Meal Programs,” J. Nutr. Educ. Behav., vol. 58, no. 4, pp. 336–345, Apr. 2026, doi: 10.1016/j.jneb.2025.12.012.
S. A. Santana, S. A. Batista, D. da Costa Maynard, V. C. Ginani, R. P. Zandonadi, and R. B. A. Botelho, “Acceptability of School Menus: A Systematic Review of Assessment Methods,” Int. J. Environ. Res. Public Health, vol. 20, no. 3, p. 2242, Jan. 2023, doi: 10.3390/ijerph20032242.
N. S. Guimarães et al., “Plate Food Waste in Food Services: A Systematic Review and Meta-Analysis,” Nutrients, vol. 16, no. 10, p. 1429, May 2024, doi: 10.3390/nu16101429.
V. T. Siy Van et al., “Predicting undernutrition among elementary schoolchildren in the Philippines using machine learning algo-rithms,” Nutrition, vol. 96, p. 111571, Apr. 2022, doi: 10.1016/j.nut.2021.111571.
B. H. Sgura, M. S. M. Saes, C. V. C. Fonseca, and Y. P. B. Adorno, “Heterogeneidade regional no cumprimento das diretrizes do Programa Nacional de Alimentação Escolar: uma análise comparativa entre Nordeste e Sul do Brasil com aprendizado de máquina,” Cad. Saude Publica, vol. 42, p. e00056125, 2026, doi: 10.1590/0102-311xpt056125.
H. Surendra, A. Sujagat, and G. Wangge, “Mitigating food safety risks in Indonesia’s free school meals programme,” Lancet Reg. Heal. - Southeast Asia, vol. 46, p. 100734, Mar. 2026, doi: 10.1016/j.lansea.2026.100734.
M. A. Wardana and I. Ismail, “Model Explainable Machine Learning untuk Prediksi Kecukupan Gizi Menu MBG,” J. Ilm. Sist. Inf. dan Tek. Inform., vol. 9, no. 1, pp. 99–110, Apr. 2026, doi: 10.57093/jisti.v9i1.410.
T. B. Sedayu, N. Saputri, T. P. Akhmaliyah, M. Hanif, and D. Apdillah, “Analisis Prediksi Konsumsi Gizi pada Program Makan Bergizi Gratis (MBG) Menggunakan Metode Machine Learning,” J. Sci. Soc. Res., vol. 9, no. 3, pp. 4283–4291, 2026, doi: 10.54314/jssr.v9i3.6644.
I. M. Rachmawati, A. Nurhindarto, and M. T. Sulistyono, “Prediksi Konsumsi Harian Program Makan Bergizi,” Rabit J. Teknol. dan Sist. Inf. Univrab, vol. 11, no. 1, pp. 2029–2039, Feb. 2026, doi: 10.36341/rabit.v11i1.7609.
L. Breiman, “Random Forests,” Mach. Learn., vol. 45, no. 1, pp. 5–32, Oct. 2001, doi: 10.1023/A:1010933404324.
T. Chen and C. Guestrin, “XGBoost: A Scalable Tree Boosting System,” in Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Aug. 2016, pp. 785–794. doi: 10.1145/2939672.2939785.
H. Meyer, C. Reudenbach, S. Wöllauer, and T. Nauss, “Importance of spatial predictor variable selection in machine learning ap-plications – Moving from data reproduction to spatial prediction,” Ecol. Modell., vol. 411, p. 108815, Nov. 2019, doi: 10.1016/j.ecolmodel.2019.108815.
P. Ploton et al., “Spatial validation reveals poor predictive performance of large-scale ecological mapping models,” Nat. Commun., vol. 11, no. 1, p. 4540, Sep. 2020, doi: 10.1038/s41467-020-18321-y.
G. C. Cawley and N. L. C. Talbot, “On Over-Fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation,” J. Mach. Learn. Res., vol. 11, pp. 2079–2107, 2010, [Online]. Available: https://jmlr.org/papers/v11/cawley10a.html
S. M. Lundberg and S.-I. Lee, “A Unified Approach to Interpreting Model Predictions,” in NIPS’17: Proceedings of the 31st International Conference on Neural Information Processing Systems, Nov. 2017, pp. 4768–4777. [Online]. Available: https://dl.acm.org/doi/10.5555/3295222.3295230
S. M. Lundberg et al., “From local explanations to global understanding with explainable AI for trees,” Nat. Mach. Intell., vol. 2, no. 1, pp. 56–67, Jan. 2020, doi: 10.1038/s42256-019-0138-9.
I. E. Kumar, S. Venkatasubramanian, C. Scheidegger, and S. Friedler, “Problems With Shapley-Value-Based Explanations as Feature Importance Measures,” in Proceedings of the 37th International Conference on Machine Learning, Jun. 2020, vol. 119, pp. 5491–5500. [Online]. Available: https://dl.acm.org/doi/10.5555/3524938.3525447
D. Janzing, L. Minorics, and P. Blobaum, “Feature Relevance Quantification in Explainable AI: A Causal Problem,” in Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics, 2020, vol. 108, pp. 2907–2916. [Online]. Available: https://proceedings.mlr.press/v108/janzing20a.html
G. Shmueli, “To Explain or to Predict?,” Stat. Sci., vol. 25, no. 3, pp. 289–310, Aug. 2010, doi: 10.1214/10-STS330.
S. Wachter, B. Mittelstadt, and C. Russell, “Counterfactual Explanations Without Opening the Black Box: Automated Decisions and the GDPR,” SSRN Electron. J., 2017, doi: 10.2139/ssrn.3063289.
B. Ustun, A. Spangher, and Y. Liu, “Actionable Recourse in Linear Classification,” in Proceedings of the Conference on Fairness, Ac-countability, and Transparency, Jan. 2019, pp. 10–19. doi: 10.1145/3287560.3287566.
R. Poyiadzi, K. Sokol, R. Santos-Rodriguez, T. De Bie, and P. Flach, “FACE: Feasible and Actionable Counterfactual Explanations,” in Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, Feb. 2020, pp. 344–350. doi: 10.1145/3375627.3375850.
T. Laugel, M.-J. Lesot, C. Marsala, X. Renard, and M. Detyniecki, “The Dangers of Post-hoc Interpretability: Unjustified Counter-factual Explanations,” in Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, Aug. 2019, pp. 2801–2807. doi: 10.24963/ijcai.2019/388.
A.-H. Karimi, G. Barthe, B. Schölkopf, and I. Valera, “A Survey of Algorithmic Recourse: Contrastive Explanations and Conse-quential Recommendations,” ACM Comput. Surv., vol. 55, no. 5, pp. 1–29, May 2023, doi: 10.1145/3527848.
A.-H. Karimi, B. Schölkopf, and I. Valera, “Algorithmic Recourse,” in Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, Mar. 2021, pp. 353–362. doi: 10.1145/3442188.3445899.
N. Hidayat, L. Afuan, and H. R. Jannah, “Prescriptive Learning Analytics for Student Dropout: Integrating Temporal Velocity and Counterfactual Explanations in Longitudinal Data,” J. Comput. Theor. Appl., vol. 3, no. 4, pp. 627–646, May 2026, doi: 10.62411/jcta.15920.
J. Kleinberg, J. Ludwig, S. Mullainathan, and Z. Obermeyer, “Prediction Policy Problems,” Am. Econ. Rev., vol. 105, no. 5, pp. 491–495, May 2015, doi: 10.1257/aer.p20151023.
M. R. Maulana, P. Sudira, P. Priyanto, Y. A. Fadlullah, D. Nurdiana, and A. H. Bin Sofhi @ Subhi, “Explainable Machine Learning for Predicting Indonesian Vocational School Accreditation: Geographic Validation, Probability Calibration, and Subgroup Audit-ing,” J. Comput. Theor. Appl., vol. 4, no. 2, pp. 502–516, Sep. 2026, doi: 10.62411/jcta.17721.
F. Pedregosa et al., “Scikit-learn: Machine Learning in Python,” J. Mach. Learn. Res., vol. 12, pp. 2825–2830, 2011, doi: 10.1289/EHP4713.
D. R. Roberts et al., “Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure,” Ecography (Cop.)., vol. 40, no. 8, pp. 913–929, 2017, doi: 10.1111/ecog.02881.
T. Saito and M. Rehmsmeier, “The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets,” PLoS One, vol. 10, no. 3, p. e0118432, Mar. 2015, doi: 10.1371/journal.pone.0118432.
E. W. Steyerberg et al., “Assessing the Performance of Prediction Models,” Epidemiology, vol. 21, no. 1, pp. 128–138, Jan. 2010, doi: 10.1097/EDE.0b013e3181c30fb2.
M. Pakdaman Naeini, G. Cooper, and M. Hauskrecht, “Obtaining Well Calibrated Probabilities Using Bayesian Binning,” Proc. AAAI Conf. Artif. Intell., vol. 29, no. 1, Feb. 2015, doi: 10.1609/aaai.v29i1.9602.
C. A. Field and A. H. Welsh, “Bootstrapping Clustered Data,” J. R. Stat. Soc. Ser. B Stat. Methodol., vol. 69, no. 3, pp. 369–390, Jun. 2007, doi: 10.1111/j.1467-9868.2007.00593.x.
Y. Vergouwe, E. W. Steyerberg, M. J. C. Eijkemans, and J. D. F. Habbema, “Substantial effective sample sizes were required for external validation studies of predictive logistic regression models,” J. Clin. Epidemiol., vol. 58, no. 5, pp. 475–483, May 2005, doi: 10.1016/j.jclinepi.2004.06.017.
C. J. Van Rijsbergen, “Evaluation,” in Information Retrieval, 2nd ed., vol. 30, no. 6, London, UK: Butterworths, 1979, pp. 374–375. [Online]. Available: https://www.dcs.gla.ac.uk/Keith/Chapter.7/Ch.7.html
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Amanatullah Pandu Zenklinov, Andriano, Wisnowan Hendy Saputra, Marcella Ariani, Nabilla Rida Tri Nisa, Sabrina Julietta Arisanty
This work is licensed under a Creative Commons Attribution 4.0 International License.
