notesum.ai
Published at November 12New Emerged Security and Privacy of Pre-trained Model: a Survey and Outlook
cs.AI
Released Date: November 12, 2024
Authors: Meng Yang1, Tianqing Zhu2, Chi Liu2, WanLei Zhou2, Shui Yu1, Philip S. Yu3
Aff.: 1University of Technology Sydney, Australia; 2City University of Macao, China; 3University of Illinois at Chicago, United States
| Model | Available | Modal | Size | Base Model | Release Time |
|---|---|---|---|---|---|
| GPT-1 (Radford, 2018) | open-source | text | - | Transformer (decoder) | Jun-2018 |
| BERT (Devlin et al., 2019) | open-source | text | 330M | Transformer (encoder) | Oct-2018 |
| GPT-2 (Radford et al., 2019) | open-source | text | 1.5B | GPT | Feb-2019 |
| RoBERTa (Liu, 2019) | open-source | text | 355M | BERT | Aug-2019 |
| ALBERT (Lan, 2019) | open-source | text | 235M | BERT | Sep-2019 |
| BART (Lewis, 2019) | open-source | text | 400M | BERT | Oct-2019 |
| T5 (Raffel et al., 2020) | open-source | text | 3B/11B | Transformer | Oct-2019 |
| GPT-J (Wang and Komatsuzaki, 2021) | open-source | text | 6B | GPT-2 | May-2021 |
| CLIP (Radford et al., 2021) | open-source | multi | 400M | ResNet/Transformer(encoder) | Jan-2021 |
| BLIP (Li et al., 2022) | open-source | multi | 400M | Transformer | Mar-2022 |
| OFA (Wang et al., 2022b) | open-source | multi | 33M-930M | ResNet | Jun-2022 |
| GPT-3 (Brown et al., 2020) | close-source | text | 6B/175B | GPT-2 | May-2020 |
| GPT-3.5 (Brown et al., 2020) | close-source | text | - | GPT-3 | Mar-2022 |
| Instruct-GPT (Ouyang et al., 2022) | close-source | text | 6B/175B | GPT-3 | Mar-2022 |
| PaLM (Chowdhery et al., 2023) | close-source | text | 8B/62B/540B | Transformer (decoder) | Apr-2022 |
| OPT (Zhang et al., 2022b) | open-source | text | 125M-175B | Transformer (decoder) | May-2022 |
| ChatGPT (Brown et al., 2020) | close-source | text | Application | GPT-3.5 | Nov-2022 |
| LLaMA (Touvron et al., 2023a) | open-source | text | 7B/13B/33B/65B | Transformer (decoder) | Feb-2023 |
| Alpaca (Taori et al., 2023) | open-source | text | 7B/13B | LLaMA | Mar-2023 |
| Vicuna (Chiang et al., 2023) | open-source | text | 7B/13B | LLaMA | Mar-2023 |
| Claude | close-source | text | Application | - | Mar-2023 |
| Dolly (Conover et al., 2023) | open-source | text | 7B/12B | GPT-3 | Mar-2023 |
| Bard (Gemini) | close-source | text | Application | PaLM | Mar-2023 |
| PaLM-2 (Anil et al., 2023) | close-source | text | 16B | PaLM | May-2023 |
| LLaMA-2 (Touvron et al., 2023b) | open-source | text | 7B/13B/70B | - | Jul-2023 |
| Claude-2 | close-source | text | Application | - | Jul-2023 |
| Falcon (Almazrouei et al., 2023) | open-source | text | 7B/40B/180B | GPT-3 | Nov-2023 |
| LLaMA-3 | open-source | text | 8B/70B | - | Apr-2024 |
| New Bing | close-source | - | Application | GPT-4 | - |
| Flamingo (Alayrac et al., 2022) | close-source | multi | 3B/9B/80B | CLIP+Transformer (decoder) | Apr-2022 |
| BLIP-2 (Li et al., 2023b) | open-source | multi | 3B-12B | CLIP+OPT/T5 | Feb-2023 |
| GPT-4 (Achiam et al., 2023) | close-source | multi | - | GPT-3.5 | Mar-2023 |
| MiniGPT-4 (Zhu et al., 2023) | open-source | multi | 7B/13B | Vicuna/LLaMA | Apr-2023 |
| Instruct BLIP (Dai et al., 2023) | open-source | multi | 7B/13B | BLIP-2 | Jun-2023 |
| OpenFlamingo (Awadalla et al., 2023) | open-source | multi | 3B/9B | CLIP+LLaMA | Aug-2023 |
| LLaVA (Liu et al., 2023c) | open-source | multi | 7B/13B | Vicuna | Dec-2023 |
| LLaVA-NeXT (Liu et al., 2024a) | open-source | multi | 34B | LLaVA | Jan-2024 |
| GPT-4o | close-source | multi | - | GPT-4 | May-2024 |
| Claude-3 | close-source | multi | Application | - | Mar-2024 |
| Claude-3.5 | close-source | multi | Application | - | Jun-2024 |
| LLaMA-3.2 | open-source | multi | 11B/90B | - | Sep-2024 |