# NLP

🔍 DeBERTa: Decoding-enhanced BERT with Disentangled Attention

BERT에서 Word Emb, Pos(Relative) Emb를 쪼개 두 벡터로 각각 계산하자!

Posted on Fri, Jun 25, 2021 NLP 논문리뷰

📎 FairFil: Contrastive Neural Debiasing Method for Pretrained Text Encoders

ICLR2021, PLM(BERT)에 추가 모듈 붙이고, Contrastive learning + Regualizer로 Debiased된 output 추출하는 방법론.

Posted on Mon, May 24, 2021 NLP TLDR논문리뷰

🤗 Transformers Trainer 뜯어보기

Huggingface Transformers 학습 Wrapper, Trainer가 어떻게 동작하는지 알아보자!

Posted on Sat, May 22, 2021 NLP MLDL Framework

🐳 Docker + DeepSpeed + MultiGPU 사용 중 NCCL posix_fallocate failed: No space left on device 에러 대응하기

도커 컨테이너 위에서 DeepSpeed + MultiGPU 사용시, NCCL No Space left on device 에러가 발생하는 경우의 해결책

Posted on Thu, May 20, 2021 NLP MLDL Framework

🤗 Transformers와 DeepSpeed로 새 BERT모델 굽기

Transformers run_mlm.py 와 DeepSpeed, ZeRO-2/ZeRO-3으로 새 BERT 굽기

Posted on Mon, May 17, 2021 NLP MLDL Framework

🤗 Huggingface + DeepSpeed + FairScale

Huggingface로 '큰' 모델 학습하기

Posted on Sun, May 16, 2021 NLP MLDL Framework

🤗 Transformers 새 모델 만들기

🤗Huggingface Transformers에 새로운 모델 구조를 만들어보자!

Posted on Fri, May 14, 2021 NLP MLDL Framework

🔎 DExperts: On-the-Fly Controlled Text Generation with Experts and Anti-Experts

Language Model Finetune 통해 Detoxify & Sentiment Controlled Generation 하기

Posted on Fri, May 14, 2021 NLP 논문리뷰

Train Language Model on TPU

TPU로 Language Model 학습해 보자! 🔥

Posted on Tue, May 11, 2021 NLP

🔎 GeDi: Generative Discriminator Guided Sequence Generation

GPT 110M으로 GPT-2(XL, 1.2B), GPT-3(175B) Generation Guide하기

Posted on Sat, May 1, 2021 NLP 논문리뷰

🔎 Longformer

BERT max len 512를 넘어 4096까지, Sequence length에 O(n)인 Attention Transformer

Posted on Sat, Mar 27, 2021 NLP 논문리뷰

🔎 exBERT: Extending Pre-trained Models with Domain-specific Vocabulary Under Constrained Training Resources

기존 BERT에 새로운 Vocab & (상대적으로)작은, 병렬 BERT모델을 붙여서 학습시, Domain Adaptation(DAPT)가 아주 잘 된다! (약 5-6%p의 균일한 성능 향상을 보임)

Posted on Fri, Mar 19, 2021 NLP 논문리뷰