notesum.ai

Published at October 21

Bayesian scaling laws for in-context learning

cs.CL

cs.AI

cs.FL

cs.LG

I.2.7

Released Date: October 21, 2024

Authors: Aryaman Arora¹, Dan Jurafsky¹, Christopher Potts¹, Noah D. Goodman¹

Aff.: ¹Stanford University

Arxiv: https://arxiv.org/abs/2410.16531v1

Refer to caption

Hyperparameter	Setting
hidden_size	$768$
max_position_embeddings	$1024$
num_hidden_layers	$[4,8,12]$
num_attention_heads	$12$
vocab_size	$50$
intermediate_size	$3072$
tie_word_embeddings	True