notesum.ai

Published at November 21

InstCache: A Predictive Cache for LLM Serving

cs.CL

cs.DC

Released Date: November 21, 2024

Authors: Longwei Zou¹, Tingfeng Liu¹, Kai Chen², Jiangang Kong², Yangdong Deng¹

Aff.: ¹Tsinghua University; ²DiDi Global Inc.

Arxiv: http://arxiv.org/abs/2411.13820v1

Refer to caption