notesum.ai

Published at November 6

Can Custom Models Learn In-Context? An Exploration of Hybrid Architecture Performance on In-Context Learning Tasks

cs.LG

cs.AI

Released Date: November 6, 2024

Authors: Ryan Campbell¹, Nelson Lojo¹, Kesava Viswanadha¹, Christoffer Grondal Tryggestad¹, Derrick Han Sun¹, Sriteja Vijapurapu¹, August Rolfsen¹, Anant Sahai¹

Aff.: ¹UC Berkeley

Arxiv: http://arxiv.org/abs/2411.03945v1

Task	dim $(d)$	points ( $N$ )	$x$ distribution	$y$ calculation / parameter distribution	Task-specific
Linear Regression	20	41	$\mathcal{N}(0,I_{d})$	$w\sim\mathcal{N}(0,I_{d})$	–
Sparse Linear	20	41	$\mathcal{N}(0,I_{d})$	$w\sim\mathcal{N}(0,I_{d})$ , $\texttt{sparsity}(w)\leftarrow k$	$k=3$
2-Layer MLP	20	101	$\mathcal{N}(0,I_{d})$	$W^{(1)}_{ij},W^{(2)}_{ij}\sim\mathcal{N}(0,1)$	$\mathrm{width}=100$
Decision Tree	20	101	$\mathcal{N}(0,I_{d})$	$\mathrm{leaf}\sim\mathcal{N}(0,1),\mathrm{non\_leaf}\sim\{1,...,d\}$	$\mathrm{depth}=4$
Sparse Parity	10	140	$\{-1,1\}^{d}$	$y=\prod_{j\in I}x[j]$	$k=2$
Vector MQAR	20	128	$\mathrm{Unif}(\mathcal{S}^{d-1})$	$y\sim\mathrm{Unif}(\mathcal{S}^{d-1})$	–