notesum.ai

Published at October 22

PLDR-LLM: Large Language Model from Power Law Decoder Representations

cs.CL

cs.AI

Released Date: October 22, 2024

Authors: Burc Gokden¹

Aff.: ¹Fromthesky Research Labs LLC, Oregon, USA

Arxiv: https://arxiv.org/abs/2410.16703v1

Refer to caption

Model	$\lambda_{DAG1}$	$\lambda_{DAG2}$	$\lambda_{DAG3}$	DAG Loss at Inference			$DLR({\bm{\mathsfit{A}}}_{P},{\bm{\mathsfit{G}}}_{LM})$
				DL( ${\bm{\mathsfit{A}}}_{LM})$	DL( ${\bm{\mathsfit{A}}}_{P})$	DL( ${\bm{\mathsfit{G}}}_{LM})$
PLDRv5-1	NA	NA	NA	$6.77\times 10^{-7}$	$\nearrow$	$\nearrow$	NA
PLDRv5-2	NA	NA	NA	$7.51\times 10^{-7}$	$\nearrow$	$\nearrow$	NA
PLDRv5-3	NA	NA	NA	$8.01\times 10^{-5}$	$\nearrow$	$\nearrow$	NA
PLDRv5-4	NA	NA	NA	$1.78\times 10^{-6}$	$\nearrow$	$\nearrow$	NA
PLDRv9-1	NA	NA	NA	$2.03\times 10^{-6}$	$\nearrow$	$\nearrow$	NA
PLDRv9-2	NA	NA	NA	$1.47\times 10^{-6}$	$\nearrow$	$\nearrow$	NA
PLDRv5-DAG-1	0.05	0.05	0.05	0	$8.61\times 10^{-3}$	$1.43\times 10^{-2}$	$1.15\times 10^{-3}$
PLDRv5-DAG-2	0.02	0.02	0.02	0	$2.22\times 10^{-2}$	$4.03\times 10^{-2}$	$1.25\times 10^{-3}$
PLDRv5-DAG-3	0.01	0.01	0.01	0	$4.36\times 10^{-2}$	$8.03\times 10^{-2}$	$1.24\times 10^{-3}$
PLDRv5-DAG-4	0.005	0.005	0.005	0	$8.51\times 10^{-2}$	$1.94\times 10^{-1}$	$1.39\times 10^{-3}$
PLDRv5-DAG-5	1	0.005	0.005	0	$8.19\times 10^{-2}$	$2.15\times 10^{-1}$	$1.48\times 10^{-3}$
PLDRv5-DAG-6	0.005	NA	NA	$5.09\times 10^{-7}$	$\nearrow$	$\nearrow$	NA
PLDRv5-DAG-7	0.001	0.005	0.005	0	$8.31\times 10^{-2}$	$2.07\times 10^{-1}$	$1.45\times 10^{-3}$
PLDRv5-tab-1	NA	NA	NA	$4.24\times 10^{-3}$	$\nearrow$	$\nearrow$	NA
PLDRv5-tab-2	NA	NA	NA	$2.58\times 10^{-3}$	$\nearrow$	$\nearrow$	NA