notesum.ai

Published at October 18

Revisiting SLO and Goodput Metrics in LLM Serving

cs.AI
cs.CL

Released Date: October 18, 2024

Authors: Zhibin Wang1, Shipeng Li1, Yuhang Zhou1, Xue Li2, Rong Gu1, Nguyen Cam-Tu1, Chen Tian1, Sheng Zhong1

Aff.: 1State Key Laboratory for Novel Software Technology, Nanjing University; 2Alibaba Group

Arxiv: https://arxiv.org/abs/2410.14257v1