notesum.ai

Published at October 20

Causality for Large Language Models

cs.DS

cs.CC

Released Date: October 20, 2024

Authors: Anpeng Wu¹, Kun Kuang¹, Minqin Zhu¹, Yingrong Wang¹, Yujia Zheng², Kairong Han¹, Baohong Li¹, Guangyi Chen³, Fei Wu¹, Kun Zhang⁴

Aff.: ¹Department of Computer Science and Technology, Zhejiang University; ²Carnegie Mellon University; ³Mohamed bin Zayed University of Artificial Intelligence; ⁴Mohamed bin Zayed University of Artificial Intelligence, Carnegie Mellon University

Arxiv: https://arxiv.org/abs/2410.15319v1

LLM Stages	Causality-based Techniques
Pre-Training	Debiased Token Embedding: [27, 28, 29, 30, 31]
	Counterfactual Training Corpus: [32, 33, 34, 35]
	Causal Foundation Model: [36, 37, 38, 39]
Fine-Tuning	Debiased Token Embedding: [27, 28, 29, 30, 31]
	Counterfactual Training Corpus: [32, 33, 34, 35]
	SFT in Specific Tasks: [31, 32, 33, 40, 41, 42, 43, 44, 45]
Alignment	Causal RLHF [46]
	Counterfactual DPO [47]
	Causal Preference Optimization [48]
Inference	Causal Discovery: [49, 50, 51, 52, 53]
	Causal Effect: [25, 53, 54, 55, 56]
	Counterfactual Reasoning: [23, 32, 33, 43, 57]
	Other Debiasing Tasks: [40, 58, 59, 60, 61, 62, 63]
Evaluation	Benchmark [23, 25, 64, 65, 66, 46, 67, 52, 68, 53, 60, 69, 70, 71, 72, 73, 74]