Task 6: Automatic cyberbullying detection
Task definition
Although the problem of humiliating and slandering people through the Internet has existed almost as long as communication via the Internet between people, the appearance of new devices, such as smartphones and tablet computers, which allow using this medium not only at home, work or school but also in motion, has further exacerbated the problem. Especially recent decade, during which Social Networking Services (SNS), such as Facebook and Twitter, rapidly grew in popularity, has brought to light the problem of unethical behaviors in Internet environments, which has been greatly impairing public mental health in adults and, for the most, in younger users and children. It is the problem of cyberbullying (CB), defined as exploitation of open online means of communication, such as Internet forum boards, or SNS to convey harmful and disturbing information about private individuals, often children and students.
To deal with the problem, researchers around the world have started studying the problem of cyberbullying with a goal to automatically detect Internet entries containing harmful information, and report them to SNS service providers for further analysis and deletion. After ten years of research [1], a sufficient knowledge base on this problem has been collected for languages of well-developed countries, such as the US, or Japan. Unfortunately, still close to nothing in this matter has been done for the Polish language. With this task, we aim at filling this gap.
In this pilot task, the contestants will determine whether an Internet entry is classifiable as part of cyberbullying narration or not. The entries will contain tweets collected from openly available Twitter discussions. Since much of the problem of automatic cyberbullying detection often relies on feature selection and feature engineering [2], the tweets will be provided as such, with minimal preprocessing. The preprocessing, if used, will be applied mostly for cases when information about a private person is revealed to the public.
The goal of the contestants will be to classify the tweets into cyberbullying/harmful and non-cyberbullying/non-harmful with the highest possible Precision, Recall, balanced F-score and Accuracy. In an additional sub-task, the contestants will differentiate between various types of cyberbullying, i.e., revealing of private information, personal threats, blackmails, ridiculing, gossip/insinuations, or accumulation of vulgarities and profane language.
Examples of tweets
Template: “contents”,class (1=harmful, 0=non harmful, etc.):
"Ja mam dla ciebie lepszą propozycję : powieś się gdzieś pod lasem UB-ecka gnido .",1
"macie jej numer zdissujcie ją 8)",1
"Gosia się bardzo nudzi i chętnie z wami porozmawia. macie jej numer - [NUMER TEL.] dzwonić może każdy, ale sms tylko plus.",1
"huju jebany oddawaj server gnoju glubi frajezre kutasie oddawaj bo cie zajebie huju zzglosilem cie i tak nie będziesz miec konta hahahahahahahhahahahaahha",1
"Czerwone Gitary, Historia jednej znajomości... i parawany które istniały zawsze…",0
Task description
Task 6-1: Harmful vs non-harmful
In this task, the participants are to distinguish between normal/non-harmful tweets (class: 0) and tweets that contain any kind of harmful information (class: 1). This includes cyberbullying, hate speech and related phenomena. The data for the task is available now and can be downloaded from the link provided below.
Task 6-1 training data: DOWNLOAD
Evaluation
File for evaluation should contain only tags (results of classification), one per line, aligned in order corresponding to the order of sentences in test data (provided later). For evaluation use the attached Perl script in the following manner to calculate the results:
perl evaluate1.pl results.txt > output.txt
The Perl script calculates Precision, Recall, Balanced F-score and Accuracy. In choosing the winners we will look primarily at the balanced F-score. Moreover, in the case of equal results for F-score, the team with higher Accuracy will be chosen as the winner. Furthermore, in case of the same F-score and Accuracy, a priority will be given to the results as close as possible to BEP (break-even-point of Precision and Recall).
Task 6-2: Type of harmfulness
In this task, the participants shall distinguish between three classes of tweets: 0 (non-harmful), 1 (cyberbullying), 2 (hate-speech). There are various definitions of both cyberbullying and hate-speech, some of them even putting those two phenomena in the same group. The specific conditions on which we based our annotations for both cyberbullying and hate-speech, which have been worked out during ten years of research [1] will be summarized in an introductory paper for the task, however, the main and definitive condition to distinguish the two is whether the harmful action is addressed towards a private person(s) (cyberbullying), or a public person/entity/large group (hate-speech).
Task 6-2 training data: DOWNLOAD
Evaluation
File for evaluation should contain only tags (results of classification), one per line, aligned in the order corresponding to the order of sentences in test data (provided later). For evaluation use the attached Perl script in the following manner to calculate the results:
perl evaluate2.pl results.txt > output.txt
The Perl script calculates Micro-Average F-score (microF) and Macro-Average F-score (macroF). In choosing the winners we will look primarily microF to treat all instances equally since the number of instances is different for each class. Moreover, in the case of equal results for microF, the team with higher macroF will be chosen as the winner. The additional macroF, treating equally not all instances, but rather all classes, is used to provide additional insight into the results.
Test data
The test data is available here for DOWNLOAD.
Submission
The participating team should prepare two folders, one for each task, with naming such as “task01” and “task02”. If the team aims only at participating in one of the tasks, only one folder is required. In each folder, the team should copy the file containing results (all labels, one per line, provided by their method), and an output file with results calculated using the evaluation Perl scripts. The folder(s) should be compressed in one file and submitted to the PolEval Submission System.
References
[1] Michal E. Ptaszynski, Fumito Masui. (2018). “Automatic Cyberbullying Detection: Emerging Research and Opportunities”, IGI Global Publishing (November 2018), ISBN: 9781522552499.
[2] Michal Ptaszynski, Juuso Kalevi Kristian Eronen and Fumito Masui. (2017). "Learning Deep on Cyberbullying is Always Better Than Brute Force", IJCAI 2017 3rd Workshop on Linguistic and Cognitive Approaches to Dialogue Agents (LaCATODA 2017), Melbourne, Australia, August 19-25, 2017