notesum.ai

Published at December 6

GUIDE: A Global Unified Inference Engine for Deploying Large Language Models in Heterogeneous Environments

cs.AI

Released Date: December 6, 2024

Authors: Yanyu Chen¹, Ganhong Huang¹

Aff.: ¹Sun Yat-sen University

Arxiv: http://arxiv.org/pdf/2412.04788v1

Refer to caption

Error Metric	vLLM (%)	FastGen (%)
Batch Latency	33.04	32.74
TTFT	33.31	41.43
Decode Throughput	51.43	54.94