1. 핵심 요약
Simple Contrastive Sentence Embedding Framework
Unsupervised method
- input을 받고 그 input 자체를 contrastive objective로 예측
⇒ 같은 문장을 Dropout을 applying 한 후 두 번 반복해서 넣음 → 이렇게 나온 2개의 임베딩 = positive pairs
⇒ 다른 sentence(negatives)를 넣고 positive를 다시한번 predict
- dropout만 사용
⇒ minimal data augmentation 효과
⇒ dropout을 제거하면 representation collapse 생김
Supervised method
NLI 데이터 사용
- entailment pairs : Positive
- contradiction pairs : Negative
성능 증명
(1) alignment between semantically-related positive pairs
(2) improve uniformity
- contrastive learning objective “flattens” the singular value distribution of the sentence embedding space
Evaluation
- STS data로 성능평가
- 기존 best results보다 4.2% and 2.2% improvement
Conclusion
- Contrastive Learning Objective가 pre-trained embedding의 anisotropic spacer가 uniform하게 함 (*uniformity: 임베딩에서 얼마나 넓은 공간을 활용하고 있는지)
- aligns positive pairs ← supervised일 때
2. 논문 링크
https://arxiv.org/abs/2104.08821
SimCSE: Simple Contrastive Learning of Sentence Embeddings
This paper presents SimCSE, a simple contrastive learning framework that greatly advances the state-of-the-art sentence embeddings. We first describe an unsupervised approach, which takes an input sentence and predicts itself in a contrastive objective, wi
arxiv.org
3. 설명 링크
SimCSE: Simple Contrastive Learning of Sentence Embeddings
1. Abstract & Introduction
coding-moomin.notion.site