Hate Speech Detection: A Solved Problem? The Challenging Case of Long Tail on Twitter 리뷰

Miss classification 에 대한 원인을 세분화하여 분석한 논문

Abstract

1. our analysis of the language in the typical datasets shows that hate speech lacks unique, discriminative features and therefore is found in the ‘long tail’ in a dataset that is difficult to discover.
2. We then propose Deep Neural Network structures serving as feature extractors that are particularly effective for capturing the semantics of hate speech.
3. are shown to be able to outperform the best performing method by up to 5 percentage points in macro-average F1, or 8 percentage points in the more challenging case of identifying hateful content.

5.4 Error Analysis

we manually analysed a sample of 200 tweets covering all classes to identify ones that are incorrectly classified by all methods.

We generally split these errors into four categories.

⇒ hate인데 non-hate 으로 판단한 경우와

⇒non-hate인데 hate 으로 판단한 경우를 모두 포함

(뭐든 본래 class가 아닌 다른class로 분류된 건 다 고려)

1. Implicitness (46%) 암시성 - 문장에 암시적인 면이 있는(ex사회문화정보알아야해석가능한?)경우

• the largest group of errors
• tweet does not contain explicit lexical or syntactic patterns as useful classification features.
• requires complicated reasoning and cultural and social background knowledge.(사회문화적인 의미를 담은 hate expression)
• false negative 예시( 정답이 hate 인데 non-hate이라고 분류한 경우)
• ex) .. these same girls ... didn’t cook that well and aren’t very nice = 이 여자들은 요리를 못하고 친절하지 않다.
• ex) expecting gender equality is the same as genocide = 성평등을 바라는 것은 대량학살이다.

2. Non-discriminative features (24%) hate키워드가 포함되어서 무조건 hate라고 잘못판단하는 경우

• classifiers were confused by certain features that are frequent, seemingly indicative of hate speech but in fact, can be found in both hate and non-hate tweets = hate speech 지표가 되는 단어들이 많이 사용되지만 실은 아닐 수도 있는 문장들
• white trash = trash 라는 단어가 hate의 강한 지표 → 아래 예시들을 모두 hate 으로 판단하는 경향있음
• ex1 ) White bus drivers are all white trash... [hate]
• ex2) ... I’m a piece of white trash I say it proudly [non-hate]

3. contextual information (18%) - 추가정보(url: 사진, 영상,웹사이트,다른 트윗 등)를 함께보고 해석해야하는 트윗인 경우

• URL the language itself does not imply hate. However, when it is combined with the video content referenced by the link, the tweet incites hatred towards particular religious groups.
• ex)what they tell you is their intention is not their intention. https://t.co/8cmfoOZwxz
• The content referenced by URLs can be videos, images, websites, or even other tweets.

4. disputable annotations (12%) - 애초에 annotator가 왜 때문에 hate으로 분류한건지 해석조차 안되는 것인 경우

• 왜 hate 인지 모르겠음 데이터 분류부터 잘못된것들?
• [Sexism] ex) He got one serve, not two. Had to defend the doubles lines also
• [Sexism] ex)@XXX Picwhatting? And you have quoted none of the tweets. What are you trying to say ...?’ is questioning a point raised in another tweet which we consider as sexism, but this tweet itself has been annotated as sexism.

---

논문 출처

 

Zhang, Ziqi, and Lei Luo. "Hate speech detection: A solved problem? the challenging case of long tail on twitter." Semantic Web 10.5 (2019): 925-945.