notesum.ai

Published at October 18

REEF: Representation Encoding Fingerprints for Large Language Models

cs.CV

cs.AI

Released Date: October 18, 2024

Authors: Jie Zhang¹, Dongrui Liu², Chen Qian³, Linfeng Zhang⁴, Yong Liu⁵, Yu Qiao², Jing Shao²

Aff.: ¹Shanghai Artificial Intelligence Laboratory, University of Chinese Academy of Sciences; ²Shanghai Artificial Intelligence Laboratory; ³Shanghai Artificial Intelligence Laboratory, Renmin University of China; ⁴Shanghai Jiaotong University; ⁵Renmin University of China

Arxiv: https://arxiv.org/abs/2410.14273v1

Model Fine-tuning

Llama-2-finance-7b

(5M Tokens)

Vicuna-1.5-7b

(370M Tokens)

Wizardmath-7b

(1.8B Tokens)

Chinesellama-2-7b

(13B Tokens)

Codellama-7b

(500B Tokens)

Llemma-7b

(700B Tokens)

PCS

0.9979

0.9985

0.0250

0.0127

0.0105

0.0098

ICS

0.9952

0.9949

0.9994

0.4996

0.2550

0.2257

Logits

0.9999

0.7033

0.7833

0.6367

REEF

0.9950

0.9985

0.9979

0.9974

0.9947

0.9962

Structured Pruning

Sheared-llama-

1.3b-pruned

Sheared-llama-

1.3b

Sheared-llama-

1.3b-sharegpt

Sheared-llama-

2.7b-pruned

Sheared-llama-

2.7b

Sheared-llama-

2.7b-sharegpt

PCS

0.0000

ICS

0.4927

0.3512

0.3510

0.6055

0.4580

0.4548

Logits

0.9967

0.9999

0.9967

0.9999

REEF

0.9368

0.9676

0.9710

0.9278

0.9701

0.9991

Unstructured Pruning

Distribution Merging (Fusechat-7b)

Sparse-llama-

2-7b

Wanda-llama-

2-7b

GBLM-llama-

2-7b

Internlm2-chat-

20b

Mixtral-8x7b-

instruct

Qwen-1.5-chat-

72b

PCS

0.9560

0.9620

0.9616

0.0000

ICS

0.9468

0.9478

0.1772

0.0105

0.0635

Logits

0.9999

0.0000

REEF

0.9985

0.9986

0.9991

0.9278

0.9701

0.9991

Weight Merging (Evollm-jp-7b)

Distribution Merging(Fusellm-7b)

Shisa-gamma-7b-v1

Wizardmath-7b-1.1

Abel-7b-002

Llama-2-7b

Openllama-2-7b

Mpt-7b

PCS

0.9992

0.9990

0.9989

0.9997

0.0194

0.0000

ICS

0.9992

0.9988

0.1043

0.2478

0.1014

Logits

0.9933

0.9999

0.0100

0.0000

REEF

0.9635

0.9526

0.9374

0.9996

0.6713

0.6200

Permutation

Scaling Transformation

Llama-2-7b

Mistral-7b

Qwen-1.5-7b

Llama-2-7b

Mistral-7b

Qwen-1.5-7b

PCS

0.0000

0.9999

0.9989

0.9999

ICS

0.1918

0.9847

0.9912

0.9999

0.9998

Logits

0.0000

0.9999

REEF

1.0000