notesum.ai

Published at October 31

Kernel Looping: Eliminating Synchronization Boundaries for Peak Inference Performance

cs.CL

cs.AI

cs.AR

D.3.4; C.1.3

Released Date: October 31, 2024

Authors: David Koeplinger¹, Darshan Gandhi¹, Pushkar Nandkar, Nathan Sheeley¹, Matheen Musaddiq¹, Leon Zhang, Reid Goodbar¹, Matthew Shaffer¹, Han Wang, Angela Wang¹, Mingran Wang¹, Raghu Prabhakar

Aff.: ¹SambaNova Systems, Inc

Arxiv: http://arxiv.org/abs/2410.23668v1

$Const$	$\|A_{i}\|=1$
	Exactly one unique argument across all calls.
$In$	$\|Reads(p_{i})\|>0$
	Parameter is read at least once.
$Local$	$In(p_{i})\cap\exists!j:i\neq j,\forall k\in[0,N-1),a_{j,k}=a_{i,k+1}$
	Read parameter: written by $p_{j}$ in previous call.
	$Out(p_{i})\cap\exists!j:i\neq j,\forall k\in[0,N-1),a_{i,k}=a_{j,k+1}$
	$\cap$ $\forall k\in[0,N-1),h\neq j,a_{i,k}\neq a_{h,k+1}$
	$\cap$ $\cup WriteSpace(p_{j})\geq\cup ReadSpace(p_{j})$
	Written parameter: read only by $p_{j}$ in next call
	and write space entirely covers read space.
$Live$	$Out(p_{i})\cap\exists j\in[0,N-1):a_{i,j}\in LiveAfter(c_{N-1})$
	At least one stored argument is live after last call.
$Out$	$\|Writes(p_{i})\|>0$
	Parameter is written at least once.