notesum.ai
Published at December 3Personalized Multimodal Large Language Models: A Survey
cs.CV
cs.AI
cs.CL
cs.IR
Released Date: December 3, 2024
Authors: Junda Wu1, Hanjia Lyu, Yu Xia, Zhehao Zhang, Joe Barrow, Ishita Kumar, Mehrnoosh Mirtaheri, Hongjie Chen, Ryan A. Rossi, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, Jiuxiang Gu, Nesreen K. Ahmed, Yu Wang, Xiang Chen, Hanieh Deilamsalehy, Namyong Park, Sungchul Kim, Huanrui Yang, Subrata Mitra, Zhengmian Hu, Nedim Lipka, Dang Nguyen, Yue Zhao, Jiebo Luo, Julian McAuley
Aff.: 1University of California, San Diego
| Category | General Mechanism | Example Models and Methods |
| Personalized MLLM Text Generation | Instruction (Sec. 3.1) | CGSMP Yong et al. (2023), ModICT Li et al. (2024c) |
| (Section 3) | Alignment (Sec. 3.2) | MPDialog Agrawal et al. (2023), Athena 3.0 Fan et al. (2023) |
| Generation (Sec. 3.3) | Wu et al. (2024b), PTSCG Wang et al. (2024a) | |
| Fine-tuning (Sec. 3.4) | Wang et al. (2023), PVIT Pi et al. (2024) | |
| Personalized MLLM Image Generation | Instruction (Sec. 4.1) | MuDI Jang et al. (2024), Zhong et al. (2024) |
| (Section 4) | Alignment (Sec. 4.2) | -ECLIPSE Patel et al. (2024), MoMA Song et al. (2024) |
| Generation (Sec. 4.3) | Layout-and-Retouch Kim et al. (2024), Instantbooth Shi et al. (2024a) | |
| Fine-tuning (Sec. 4.4) | MS-Diffusion Wang et al. (2024d), UNIMO-G Li et al. (2024a) | |
| Personalized MLLM Recommendation | Instruction (Sec. 5.1) | InteraRec Karra and Tulabandhula (2024), X-Reflect Lyu et al. (2024b) |
| (Section 5) | Alignment (Sec. 5.2) | PMG Shen et al. (2024), MMREC Tian et al. (2024) |
| Generation (Sec. 5.3) | RA-Rec Yu et al. (2024),Wei et al. (2024a) | |
| Fine-tuning (Sec. 5.4) | GPT4RecZhang et al. (2024),MMSSL Wei et al. (2023) | |
| Personalized MLLM Retrieval | Instruction (Sec. 6.1) | ConCon-Chi Rosasco et al. (2024), Med-PMC Liu et al. (2024a) |
| (Section 6) | Alignment (Sec. 6.2) | AlignBot Chen et al. (2024c), Xu et al. (2024) |
| Generation (Sec. 6.3) | Ye et al. (2024a),Yo’LLaVA Nguyen et al. (2024) | |
| Fine-tuning (Sec. 6.4) | FedPAM Feng et al. (2024), VITR Gong et al. (2023) |