notesum.ai

Published at December 10

cs.CL

Released Date: December 10, 2024

Authors: Matthieu Meeus¹, Anthony Rathé², François Remy³, Pieter Delobelle⁴, Jens-Joris Decorte³, Thomas Demeester³

Aff.: ¹Imperial College London; ²Cavell; ³Ghent University; ⁴Aleph Alpha

	Total	Batch	Total	Gradient	Total	Wall time
Model	samples	size	steps	acc. steps	Params	(days)
ChocoLlama-2-7B	$7.95$ M	$128$	$62.1$ k	$8$	$544$ M	$18.4$
ChocoLlama-2-7B-tokentrans	$5.61$ M	$128$	$43.8$ k	$6$	$839$ M	$13.1$
Llama-3-ChocoLlama	3.69M	512	57.6k	8	1072M	$14.0$