Evaluation of zero-shot NLI-based hate speech detection

Conventional supervised methods for hate speech detection depend on the availability of a sufficient number of annotated examples. However, researchers often face challenges in accessing such an amount of data. In addition, hate speech samples are hard to find, which leads to highly unbalanced data. In such cases, the zero-shot approaches can be a great alternative to the supervised methods. The goal of our work is to compare these approaches and highlight advantages and disadvantages of them. In this paper, we experimented with four hate speech dataset (FRENK, HateCheck, CAD and OLID) and three models (flan-t5-large, bart-large-mnli and XLM-RoBERTa-large-XNLI-ANLI) pre-trained on general natural language inference (NLI) tasks. NLI approaches require a hypothesis - a statement that is being evaluated for its logical relationship with the target sentence. The results show that the NLI based approaches are competitive with supervised ones. However, the NLI models are sensitive to the choice of a hypothesis and even small paraphrasing can change F1-scores substantially. In addition, our experiments indicate that the results for a set of hypotheses for different model-data pairs are positively correlated, and that the correlation for different datasets when using the same model is higher than the correlation for different models when using the same dataset. These results suggest that if we find a hypothesis that works well for a specific model and domain or for a specific type of hate speech, we can use that hypothesis with the same model also for a different domain. Similarly, another model might require different suitable hypotheses in order to demonstrate high performance.