GPT-J 사용법

//-------------------------------------
eleuther.ai
* GPT-Neo (2021-03)
https://github.com/EleutherAI/gpt-neo


* GPT-J (2021-06) - 60억 (6B) - OpenAI GPT 모델 중 Curie급
GPT-3의 오픈소스 버전
https://huggingface.co/EleutherAI/gpt-j-6B

- 사용법
https://huggingface.co/docs/transformers/model_doc/gptj

https://github.com/kingoflolz/mesh-transformer-jax


* GPT-NeoX (2022-01) - 200억 (20B)
https://github.com/EleutherAI/gpt-neox
NLP Cloud 에서 API 서비스

 

//-----------------------------------------------------------------------------
사용법 예제 샘플 코드 소스

import torch
import gc
import os
import transformers
import time
from transformers import GPTJForCausalLM, AutoTokenizer

def gptj(text, dev):
    device = torch.device(dev)
    #config = transformers.GPTJConfig.from_pretrained("EleutherAI/gpt-j-6B")

    model = GPTJForCausalLM.from_pretrained(
        "EleutherAI/gpt-j-6B",
        revision="float16",
        torch_dtype=torch.float16,
        low_cpu_mem_usage=True,
        use_cache=True,
        gradient_checkpointing=True,
    )

    model.to(device)

    tokenizer = transformers.AutoTokenizer.from_pretrained(
        "EleutherAI/gpt-j-6B", pad_token='<|endoftext|>', eos_token='<|endoftext|>', truncation_side='left')

    prompt = tokenizer(text, return_tensors='pt',
                       truncation=True, max_length=2048)
    prompt = {key: value.to(dev) for key, value in prompt.items()}
    out = model.generate(**prompt,
                         # n=1,
                         min_length=16,
                         max_new_tokens=75,
                         do_sample=True,
                         top_k=35,
                         top_p=0.9,
                         # batch_size=512,
                         temperature=0.75,
                         no_repeat_ngram_size=4,
                         # clean_up_tokenization_spaces=True,
                         use_cache=True,
                         pad_token_id=tokenizer.eos_token_id
                         )
    res = tokenizer.decode(out[0])
    return res


#
text = "The Belgian national football team "
print("generated_text", gptj(text, "cuda"))

GPU에서 14초 정도 걸림

 

참고

https://www.reddit.com/r/huggingface/comments/zgz879/reducing_latency_for_gptj/

//-----------------------------------------------------------------------------

< 참고 >
노트북 프로젝트 파일
https://github.com/NielsRogge/Transformers-Tutorials/blob/master/GPT-J-6B/Inference_with_GPT_J_6B.ipynb

pip install -q git+https://github.com/huggingface/transformers.git
pip install accelerate

- CPU에서는 작동 성공, CUDA에서 에러

model = GPTJForCausalLM.from_pretrained("EleutherAI/gpt-j-6B", revision="float16")  # , low_cpu_mem_usage=True)
    model.to(device)  <==== 에러

OutOfMemoryError: CUDA out of memory. Tried to allocate 64.00 MiB (GPU 0; 15.99 GiB total capacity; 15.10 GiB already allocated; 0 bytes free; 15.10 GiB      
reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory   
Management and PYTORCH_CUDA_ALLOC_CONF

해결방법
GPTJForCausalLM.from_pretrained() 에 옵션으로 torch_dtype=torch.float16,가 있어야 한다.

model = GPTJForCausalLM.from_pretrained(
        "EleutherAI/gpt-j-6B",
        revision="float16",
        torch_dtype=torch.float16,
        low_cpu_mem_usage=True,
        use_cache=True,
        gradient_checkpointing=True,
    )

 

//-----------------------------------------------------------------------------

< 참고 >

 

OpenAI
* GPT-3 (2020-06)
* GPT-3.5 (2022-03)

//-------------------------------------
eleuther.ai
* GPT-Neo (2021-03)
https://github.com/EleutherAI/gpt-neo


* GPT-J (2021-06) - 60억 (6B) - OpenAI Curie급
GPT-3의 오픈소스 버전
https://huggingface.co/EleutherAI/gpt-j-6B

- 사용법
https://huggingface.co/docs/transformers/model_doc/gptj

https://github.com/kingoflolz/mesh-transformer-jax



* GPT-NeoX (2022-01) - 200억 (20B)
https://github.com/EleutherAI/gpt-neox
NLP Cloud 에서 API 서비스


* GPT-NeoX-20B (2022-02)
EleutherAI/gpt-neox-20b 
https://huggingface.co/EleutherAI/gpt-neox-20b