Zero-shot Emotion Detection with RoBERTa as an Alternative for Large Language Models

Large Language Models (LLMs) such as GPT3 and ChatGPT have shown promising results in zero-shot learning. This type of learning can be useful in low-resource projects where it is necessary to avoid high annotation costs and the need to re-train a model when including new labels (or performing a different task altogether). However, LLMs, as their name suggests, are computationally expensive, and often not open source, free of charge, nor stable (since model versions can change, resulting in reproducibility issues).

As an alternative, we propose to use a lighter RoBERTa model that is specialized in one type of task as opposed to a general-purpose zero-shot model that achieves acceptable performance on a multitude of tasks. The advantages of such a specialized model are that it can be fine-tuned with less computational resources and that it can be made accessible more easily than LLMs, while maintaining competitive performance. We apply our case to emotion detection, use predictions of ChatGPT and RoBERTa fine-tuned on the end task as upper bounds, and explore various fine-tuning methods to develop a zero-shot classifier.

As a first baseline, we fine-tune RoBERTa on a Natural Language Inference (NLI) task to create a general-purpose zero-shot classifier and use it to predict emotions. As a second baseline, we use the approach proposed in Mao et al. (2022), who use out-of-the-box pre-trained LMs to predict the masked word in the hypothesis “I feel [MASK].” which precedes the relevant text. Existing emotion lexica are then used to map the predicted word to a class label, which is used as final prediction.

In this work we will attempt to develop a method that leads to a specialized zero-shot classifier that performs significantly better than the baselines, while keeping the computational costs substantially lower than those of LLMs. Evaluation will be performed on various benchmark datasets.

References:

R. Mao, Q. Liu, K. He, W. Li and E. Cambria, "The Biases of Pre-Trained Language Models: An Empirical Study on Prompt-Based Sentiment Analysis and Emotion Detection," in IEEE Transactions on Affective Computing, 2022, doi: 10.1109/TAFFC.2022.3204972.