TRIASE-SLO: Risk-Gated and SLO-Aware Routing Between FP16 and INT4 Language-Model Workers for Radiology Impression Serving

Authors

  • Junta Zeniarja Universitas Dian Nuswantoro
  • Guruh Fajar Shidik Universitas Dian Nuswantoro
  • Heru Suhartanto Universitas Indonesia
  • Aris Marjuni Universitas Dian Nuswantoro

DOI:

https://doi.org/10.62411/jcta.18198

Keywords:

FP16–INT4 Routing, LLM Serving, Mixed Precision, Multi-Instance GPU, Quantization, Radiology Impression Generation, Request Routing, Service-Level Objective

Abstract

Language models can draft radiology impressions, but a 4-bit copy of a model does not always reproduce the output of its full-precision version. TRIASE-SLO routes each request between FP16 and AWQ INT4 copies of one LoRA-adapted Qwen2.5-3B-Instruct model. A frozen rule-based hazard score computed from the FINDINGS text sends every request scoring at least 0.5 to FP16, and a secondary latency-aware score places the remaining requests. We evaluated the design in a locked confirmation in which every arm reserved the same two A100 MIG slices: four arms, seven loads and 12 paired deployment blocks of 450 previously inspected test reports (151,200 requests, none failed). The gate improved agreement with FP16. With full TRIASE-SLO routing, agreement with finding-bearing FP16 references was 99.11–99.49%, compared with 66.15–82.31% under hazard-blind routing, while 34–48% of requests still used INT4. A consistent serving-efficiency advantage was not demonstrated: full routing did not improve tail latency or throughput over two FP16 workers (median p95 change +0.33%; slower at C = 2 after Holm correction), and a consistent speed advantage of INT4 appeared only under hazard-blind routing. The demonstrated benefits of the INT4 copy are a smaller checkpoint and a larger KV-cache capacity. The incremental value of the score remains uncertain, because its prespecified C = 8 throughput benefit over gate-only routing (+1.86%, 95% CI −3.27% to +7.27%) was not confirmed. No INT4-using arm met the zero-drift engineering floor. All quality measures are rule-based FP16-consistency proxies from one public dataset, not evidence of clinical correctness or safety.

Author Biographies

Junta Zeniarja, Universitas Dian Nuswantoro

Research Center for Intelligent Distributed Surveillance and Security (IDSS), Faculty of Intelligent Technology, Universitas Dian Nuswantoro, Semarang 50131, Central Java, Indonesia; Faculty of Intelligent Technology, Universitas Dian Nuswantoro, Semarang 50131, Central Java, Indonesia

Guruh Fajar Shidik, Universitas Dian Nuswantoro

Research Center for Intelligent Distributed Surveillance and Security (IDSS), Faculty of Intelligent Technology, Universitas Dian Nuswantoro, Semarang 50131, Central Java, Indonesia; Faculty of Intelligent Technology, Universitas Dian Nuswantoro, Semarang 50131, Central Java, Indonesia

Heru Suhartanto, Universitas Indonesia

Faculty of Computer Science, Universitas Indonesia, Depok 16424, West Java, Indonesia

Aris Marjuni, Universitas Dian Nuswantoro

Research Center for Intelligent Distributed Surveillance and Security (IDSS), Faculty of Intelligent Technology, Universitas Dian Nuswantoro, Semarang 50131, Central Java, Indonesia; Faculty of Intelligent Technology, Universitas Dian Nuswantoro, Semarang 50131, Central Java, Indonesia

References

Y. Zhang, D. Y. Ding, T. Qian, C. D. Manning, and C. P. Langlotz, “Learning to summarize radiology findings,” in Proc. 9th Int. Workshop on Health Text Mining and Information Analysis (LOUHI), Brussels, Belgium, 2018, pp. 204–213, doi: 10.18653/v1/W18-5623.

D. Van Veen et al., “Adapted large language models can outperform medical experts in clinical text summarization,” Nat. Med., vol. 30, no. 4, pp. 1134–1142, Apr. 2024, doi: 10.1038/s41591-024-02855-5.

J. Dean and L. A. Barroso, “The tail at scale,” Commun. ACM, vol. 56, no. 2, pp. 74–80, Feb. 2013, doi: 10.1145/2408776.2408794.

E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, “GPTQ: Accurate post-training quantization for generative pre-trained transformers,” in Proc. 11th Int. Conf. Learning Representations (ICLR), Kigali, Rwanda, 2023, doi: 10.48550/arXiv.2210.17323.

G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han, “SmoothQuant: Accurate and efficient post-training quantization for large language models,” in Proc. 40th Int. Conf. Machine Learning (ICML), PMLR, vol. 202, Honolulu, HI, USA, 2023, pp. 38087–38099, doi: 10.48550/arXiv.2211.10438.

J. Lin et al., “AWQ: Activation-aware weight quantization for on-device LLM compression and acceleration,” in Proc. Machine Learning and Systems (MLSys), vol. 6, Santa Clara, CA, USA, 2024, pp. 87–100, doi: 10.48550/arXiv.2306.00978.

Y. Lin et al., “QServe: W4A8KV4 quantization and system co-design for efficient LLM serving,” in Proc. Machine Learning and Systems (MLSys), vol. 7, Santa Clara, CA, USA, 2025, doi: 10.48550/arXiv.2405.04532.

S. Li et al., “Evaluating quantized large language models,” in Proc. 41st Int. Conf. Machine Learning (ICML), PMLR, vol. 235, Vienna, Austria, 2024, pp. 28480–28524. [Online]. Available: https://proceedings.mlr.press/v235/li24bb.html

D. Crankshaw, X. Wang, G. Zhou, M. J. Franklin, J. E. Gonzalez, and I. Stoica, “Clipper: A low-latency online prediction serving system,” in Proc. 14th USENIX Symp. Networked Systems Design and Implementation (NSDI), Boston, MA, USA, 2017, pp. 613–627. [Online]. Available: https://www.usenix.org/conference/nsdi17/technical-sessions/presentation/crankshaw

A. Gujarati et al., “Serving DNNs like Clockwork: Performance predictability from the bottom up,” in Proc. 14th USENIX Symp. Operating Systems Design and Implementation (OSDI), 2020, pp. 443–462. [Online]. Available: https://www.usenix.org/conference/osdi20/presentation/gujarati

F. Romero, Q. Li, N. J. Yadwadkar, and C. Kozyrakis, “INFaaS: Automated model-less inference serving,” in Proc. 2021 USENIX Annual Technical Conf. (ATC), 2021, pp. 397–411. [Online]. Available: https://www.usenix.org/conference/atc21/presentation/romero

K. Hong et al., “SOLA: Optimizing SLO attainment for large language model serving with state-aware scheduling,” in Proc. Machine Learning and Systems (MLSys), vol. 7, Santa Clara, CA, USA, 2025. [Online]. Available: https://proceedings.mlsys.org/paper_files/paper/2025/hash/bc82dbfbfa43232be85b8d9838f49c3e-Abstract-Conference.html

Y. Zhong et al., “DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving,” in Proc. 18th USENIX Symp. Operating Systems Design and Implementation (OSDI), Santa Clara, CA, USA, 2024, pp. 193–210. [Online]. Available: https://www.usenix.org/conference/osdi24/presentation/zhong-yinmin

A. Agrawal et al., “Taming throughput-latency tradeoff in LLM inference with Sarathi-Serve,” in Proc. 18th USENIX Symp. Operating Systems Design and Implementation (OSDI), Santa Clara, CA, USA, 2024, pp. 117–134. [Online]. Available: https://www.usenix.org/conference/osdi24/presentation/agrawal

L. Chen, M. Zaharia, and J. Zou, “FrugalGPT: How to use large language models while reducing cost and improving performance,” arXiv preprint arXiv:2305.05176, 2023, doi: 10.48550/arXiv.2305.05176.

D. Ding et al., “Hybrid LLM: Cost-efficient and quality-aware query routing,” in Proc. 12th Int. Conf. Learning Representations (ICLR), Vienna, Austria, 2024, doi: 10.48550/arXiv.2404.14618.

I. Ong et al., “RouteLLM: Learning to route LLMs with preference data,” in Proc. 13th Int. Conf. Learning Representations (ICLR), Singapore, 2025, doi: 10.48550/arXiv.2406.18665.

Y. Li, “Rethinking predictive modeling for LLM routing: When simple kNN beats complex learned routers,” arXiv preprint arXiv:2505.12601, 2025, doi: 10.48550/arXiv.2505.12601.

NVIDIA Corporation, “NVIDIA Multi-Instance GPU user guide.” NVIDIA Docs. [Online]. Available: https://docs.nvidia.com/datacenter/tesla/mig-user-guide/ (accessed Oct. 7, 2026).

D. Demner-Fushman et al., “Preparing a collection of radiology examinations for distribution and retrieval,” J. Am. Med. Inform. Assoc., vol. 23, no. 2, pp. 304–310, Mar. 2016, doi: 10.1093/jamia/ocv080.

Z. Chen, Y. Song, T.-H. Chang, and X. Wan, “Generating radiology reports via memory-driven transformer,” in Proc. 2020 Conf. Empirical Methods in Natural Language Processing (EMNLP), Online, 2020, pp. 1439–1449, doi: 10.18653/v1/2020.emnlp-main.112.

C.-Y. Lin, “ROUGE: A package for automatic evaluation of summaries,” in Text Summarization Branches Out, Barcelona, Spain, 2004, pp. 74–81. [Online]. Available: https://aclanthology.org/W04-1013

J. Irvin et al., “CheXpert: A large chest radiograph dataset with uncertainty labels and expert comparison,” in Proc. AAAI Conf. Artificial Intelligence, vol. 33, no. 01, 2019, pp. 590–597, doi: 10.1609/aaai.v33i01.3301590.

W. W. Chapman, W. Bridewell, P. Hanbury, G. F. Cooper, and B. G. Buchanan, “A simple algorithm for identifying negated findings and diseases in discharge summaries,” J. Biomed. Inform., vol. 34, no. 5, pp. 301–310, Oct. 2001, doi: 10.1006/jbin.2001.1029.

Y. Peng, X. Wang, L. Lu, M. Bagheri, R. Summers, and Z. Lu, “NegBio: A high-performance tool for negation and uncertainty detection in radiology reports,” in AMIA Jt. Summits Transl. Sci. Proc., 2018, pp. 188–196. [Online]. Available: https://pmc.ncbi.nlm.nih.gov/articles/PMC5961822

A. Smit, S. Jain, P. Rajpurkar, A. Pareek, A. Y. Ng, and M. P. Lungren, “Combining automatic labelers and expert annotations for accurate radiology report labeling using BERT,” in Proc. 2020 Conf. Empirical Methods in Natural Language Processing (EMNLP), Online, 2020, pp. 1500–1519, doi: 10.18653/v1/2020.emnlp-main.117.

S. Jain et al., “RadGraph: Extracting clinical entities and relations from radiology reports,” in Proc. 35th Conf. Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2021, doi: 10.48550/arXiv.2106.14463.

S. Ostmeier et al., “GREEN: Generative radiology report evaluation and error notation,” in Findings Assoc. Comput. Linguistics: EMNLP 2024, Miami, FL, USA, 2024, pp. 374–390, doi: 10.18653/v1/2024.findings-emnlp.21.

F. Yu et al., “Evaluating progress in automatic chest X-ray radiology report generation,” Patterns, vol. 4, no. 9, Art. no. 100802, Sep. 2023, doi: 10.1016/j.patter.2023.100802.

E. Winarno, I. M. Nur, A. Karim, S. Amri, I. E. Wirdati, and P. W. Adi, “An explainable multimodal framework for chest X-ray alert classification using radiology reports and images,” J. Comput. Theor. Appl., vol. 3, no. 4, pp. 647–666, May 2026, doi: 10.62411/jcta.16023.

Qwen Team, “Qwen2.5 technical report,” arXiv preprint arXiv:2412.15115, 2024, doi: 10.48550/arXiv.2412.15115.

A. Q. Jiang et al., “Mistral 7B,” arXiv preprint arXiv:2310.06825, 2023, doi: 10.48550/arXiv.2310.06825.

E. J. Hu et al., “LoRA: Low-rank adaptation of large language models,” in Proc. 10th Int. Conf. Learning Representations (ICLR), 2022, doi: 10.48550/arXiv.2106.09685.

W. Kwon et al., “Efficient memory management for large language model serving with PagedAttention,” in Proc. 29th ACM Symp. Operating Systems Principles (SOSP), Koblenz, Germany, 2023, pp. 611–626, doi: 10.1145/3600006.3613165.

G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim, and B.-G. Chun, “Orca: A distributed serving system for transformer-based generative models,” in Proc. 16th USENIX Symp. Operating Systems Design and Implementation (OSDI), Carlsbad, CA, USA, 2022, pp. 521–538. [Online]. Available: https://www.usenix.org/conference/osdi22/presentation/yu

M. Mitzenmacher, “The power of two choices in randomized load balancing,” IEEE Trans. Parallel Distrib. Syst., vol. 12, no. 10, pp. 1094–1104, Oct. 2001, doi: 10.1109/71.963420.

N. I. Sitlong, A. E. Evwiekpaefe, and M. E. Irhebhude, “Hybrid dynamic programming healthcare cloud-based quality of service optimization,” J. Comput. Theor. Appl., vol. 3, no. 2, pp. 115–131, Sep. 2025, doi: 10.62411/jcta.14455.

R. Schwartz, J. Dodge, N. A. Smith, and O. Etzioni, “Green AI,” Commun. ACM, vol. 63, no. 12, pp. 54–63, Dec. 2020, doi: 10.1145/3381831.

A. S. Tejani et al., “Checklist for artificial intelligence in medical imaging (CLAIM): 2024 update,” Radiol. Artif. Intell., vol. 6, no. 4, Art. no. e240300, Jul. 2024, doi: 10.1148/ryai.240300.

G. S. Collins et al., “TRIPOD+AI statement: Updated guidance for reporting clinical prediction models that use regression or machine learning methods,” BMJ, vol. 385, Art. no. e078378, Apr. 2024, doi: 10.1136/bmj-2023-078378.

S. Holm, “A simple sequentially rejective multiple test procedure,” Scand. J. Stat., vol. 6, no. 2, pp. 65–70, 1979, doi: 10.2307/4615733.

R. Bender and S. Lange, “Adjusting for multiple testing—when and how?,” J. Clin. Epidemiol., vol. 54, no. 4, pp. 343–349, Apr. 2001, doi: 10.1016/S0895-4356(00)00314-0.

B. Efron, “Bootstrap methods: Another look at the jackknife,” Ann. Statist., vol. 7, no. 1, pp. 1–26, Jan. 1979, doi: 10.1214/aos/1176344552.

F. Wilcoxon, “Individual comparisons by ranking methods,” Biometrics Bull., vol. 1, no. 6, pp. 80–83, Dec. 1945, doi: 10.2307/3001968.

Q. McNemar, “Note on the sampling error of the difference between correlated proportions or percentages,” Psychometrika, vol. 12, no. 2, pp. 153–157, Jun. 1947, doi: 10.1007/BF02295996.

Downloads

Published

2026-11-19

How to Cite

Zeniarja, J., Shidik, G. F., Suhartanto, H., & Marjuni, A. (2026). TRIASE-SLO: Risk-Gated and SLO-Aware Routing Between FP16 and INT4 Language-Model Workers for Radiology Impression Serving. Journal of Computing Theories and Applications, 4(2). https://doi.org/10.62411/jcta.18198