And How Does That Make You Feel? Towards More Robust Emotion Representations in Few-Shot Text Classification with BERT

Although contextualized word embeddings have shown great improvements in various NLP tasks, earlier research has shown that they do not contain a sufficient amount of emotion information to represent emotion associations in real-world concepts. Additionally, classes in emotion detection tasks may show subtle differences and cause confusion in language models. Previous attempts to tackle these issues include learning emotion labels during fine-tuning. For example, Xiong (2021), and Alhuzali & Ananiadou (2021) prepended emotion class labels to the input text in multi-class and multi-label experiments, respectively: “[CLS] label1, label2, … labeln [SEP] token1, token2, … tokenn [EOS]”. This simple approach allows BERT's internal mechanisms to compute attention between the words in the input text and the words in the labels during the forward pass, therefore amplifying the relationship between tokens and emotion classes. In our work, we investigate the effect of contrastive learning on emotion representation quality in a few-shot setting, making it even more difficult to learn emotion representations due to a lack of labeled data. Since the goal of contrastive learning is to push away dissimilar instances in the embedding space, and group similar instances closer together, we hypothesize that BERT will be able to distinguish emotion classes more effectively. We will compare our method to the aforementioned label attention approach.

References:

Hassan Alhuzali and Sophia Ananiadou. 2021. SpanEmo: Casting Multi-label Emotion Classification as Span-prediction. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1573-1584, Online. Association for Computational Linguistics.

Yijin Xiong, Yukun Feng, Hao Wu, Hidetaka Kamigaito, and Manabu Okumura. 2021. Fusing Label Embedding into BERT: An Efficient Improvement for Text Classification. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1743–1750, Online. Association for Computational Linguistics.