Social sensing with big data: Detecting hate speech in social media

The internet's accessibility and social media platforms, like Facebook and Twitter, have accelerated the spread of hate speech and fake news, both of which can be detrimental to society's overall well-being. Identifying and tracking hate speech is becoming increasingly difficult for the public, private citizens, legislators, and academics. Despite efforts to leverage automatic detection and monitoring techniques, their performances are still far from satisfactory. This study employs Natural Language Processing (NLP) and Machine Learning (ML) approaches to detect hate speech for decision-making. The result showed that the Support Vector Machine (SVM) algorithm has the best performance with an accuracy of 0.86 compared to the Random Forest with 0.8 accuracy. The manual evaluation of the performance of our algorithm yielded an inter-annotator agreement Cronbach’s alpha (α = .775).


Introduction
Recent statistics have shown that Nigeria has about 90.48 million internet users (about 55.4% penetration) out of which 33 million are actively using social networking sites with WhatsApp, Facebook, and Twitter accounting for 95%, 89%, and 61% of the users [1].A growing number of government agencies and companies also use social media networks for public interaction.The emergence of social networking sites such as Twitter and Facebook has democratized information, allowing users to share content about themselves and their social surroundings at an unprecedented scale.These sites have resulted in large volumes (big data) of unsolicited geographically and socially relevant data being created in real-time.These new forms of big spatial data sources contain lots of information (digital footprints) that can be harnessed for studying the dynamics of the human social environment known as social sensing.Social sensing is an emerging and novel research field, where unsolicited observations of real-world events are spontaneously mentioned in cyberspace and the users of such platforms act as human sensors [2].In contrast to physical sensors that report events objectively, human sensors can recognize, summarize, and report perceptions of events differently.
However, the availability of the internet and social networking sites has propelled the spread of fake news and haterelated content in society.Hate speech includes any words, actions, writings, or displays that could incite others to engage in violent or discriminatory behavior [3].In other words, hate speech is a form of verbal abuse that targets specific groups or individuals based on their identity, beliefs, or characteristics [4].It can jeopardize social cohesiveness and democratic values, as well as negatively affect the victim's mental health, general well-being, and safety.Thus, it is crucial to identify and stop hate speech if one hopes to maintain social harmony and online safety.Opinion mining from social media content can be a very valuable tool in governance [5].No study of contemporary society can ignore the rich potential afforded by these new dimensions of social life.However, social media content is messy and unstructured and the task of extracting valuable insights from these emerging datasets presents a big computational research challenge [6].People are essential components in social sensing, therefore collection and analysis of these new forms of data sources can help us better understand society and make decisions.In this paper, we propose to use social sensing techniques to enhance the performance and explain ability of hate speech detection models.We will analyze large Twitter datasets using Natural Language Processing (NLP) algorithm in order to identify and reveal patterns of hate speech on social media.Additionally, utilizing machine learning (ML) techniques to automate the text classification process will yield findings that are less subjective and more accurate [7].NLP and ML are often combined to classify and detect hate speech in social media [8].
The rest of the paper is structured as follows: section two reviews the related literature, then material and method is described in section three.In section four, results and discussion are presented and conclusion is given in section five.

Review of Related Literature
The fact that hate speech is frequently subtle and context-dependent makes it challenging to identify and distinguish from acceptable expressions of humor or opinion.Furthermore, hate speech is subject to change throughout time, taking on new forms and tactics to evade detection through established means.Consequently, it might not be possible to fully capture the subtleties and dynamics of hate speech by depending only on linguistic elements.To overcome this difficulty, several researchers have suggested utilizing social sensing techniques to add user profile and social context data to the hate speech identification process.
Researchers have employed different approaches to quantify and detect hate-related speech especially those from social media.For example, Oriola and Kotzé [9] evaluated different machine-learning techniques on South African tweets to analyze and detect offensive and hate speech.They obtained the best results with a support vector machine (SVM).Mutanga, and Naicker [10] employed ensemble machine methods (decision trees and SVM) for automatic detection of hate speech in tweets.They found that these methods outperform classical machine learning methods that suffer from high variance.Plaza-Del-Arco, Molina-González [11] employed a combined approach using natural language processing and machine learning to detect hate speech.They found that a combination of NLP and ML helps to detect hate speech more accurately.Khanday, and Rabani [12] used manually annotated tweets collected during the COVID-19 pandemic and ensemble machine-learning methods to detect hate speech.They concluded that the Decision Tree classifier is the most effective compared to other ML methods.Haider, and Dipty [13] also explored different ML algorithms in their attempt to classify hate comments from social media texts.Their experimental results showed that Random Forest produced the highest accuracy compared to other learning methods such as logistic regression, SVM and Naive Bayes.
Finally, in a survey on hate speech detection, despite the introduction of deep learning, ensemble and transformer-based approaches NLP and ML remain relevant in hate speech detection tasks [14].Combining two or more machine learning algorithms can minimize variance and increase learning capacity greatly.This study will leverage from previous literature to develop NLP and ML models to automatically analyze and detect hateful comments on Twitter datasets.

Materials and Method
The proposed framework for this study is an improved classification model that aims to increase classification accuracy to assist decision-makers in the stock market sector.The data collection exercise for this model begins with financial tweets and stock prices.In the second phase, Twitter data is pre-processed to ensure that only essential data is retained, as well as all necessary formatting on the stock data.The polarity of Twitter data is determined in the analysis phase, and public emotions are classified as hateful, not hateful or abusive values using Natural Language Processing.In the analysis phase, we will explore relevant Machine Learning algorithms for hate speech detection.We will then evaluate the performance of the forecasting model.The proposed conceptual model for the research is presented in Figure 1.

Data
All the necessary credentials such as access key and access token will be obtained from the Twitter application programming interface (API).The API will then be utilized to stream data using Twitter handles of some pre-defined keywords within the Nigeria bounding box (top left 14°.5771N, 2°.6917E; lower right 4°.2405N, 15°.0000E).Streaming API can access real-time data of tweets using defined queries.Data streaming will be carried out for up to three months to acquire sufficient data for the study.Data has been described as an essential ingredient of the digital space [15].

Method of Data Analysis
Using social media texts to extract meaningful insights provides a challenging yet rich context for exploring computational models of natural language [16].Because Twitter data is highly unstructured, it necessitates extensive data cleaning to prepare it for further analysis.This process will involve enormous tasks starting with the removal of abbreviations, white spaces, stop words, URLs, hashtags and user names.The process is concluded with feature extraction to filter relevant and irrelevant tweets.In the R programming environment, the NLP technique will be used to classify sentiments automatically.We then employ the Amazon Mechanical Turk (AMT) for manual validation.a commercial Human Intelligence Task (HTI) crowdsourcing platform to validate a sample of tweets.AMT is widely used by researchers for SA validation [17][18][19][20][21][22].
Estimating the reliability and accuracy of human annotation is implemented by quantifying the degree of agreement that exists between the annotators called inter-rater reliability (IRR) [23][24][25][26][27]. IRR is a measure of consistency among observation ratings provided by multiple annotators commonly evaluated using the intraclass correlation coefficient (ICC) [28].ICC is a method used to measure the degree of correlation and agreement between raters [29].The range of ICC is between 0 and 1. ICC value of 0 indicates random agreement and 1 indicates perfect agreement while negative ICC indicates disagreement between raters' scores [28].ICC is usually expressed as Cronbach's alpha (α).Cronbach's alpha (α)provides a measure of the internal consistency of annotation scores [30], written as in Equation 1: Where α is a measure of consistency in agreement,   is the variance of scores in item i,   is the variance of test scores and n is the number of items.
Combining textual, social context, and user data to identify the subtleties and targets of hate speech on social media sites like Twitter is one approach to employing social sensing for hate speech identification.By taking into account the multi-modal data and the context-dependent nature of hate speech, this approach can increase the robustness and accuracy of hate speech identification [31].

Results and Discussion
In this study, we employ five different human annotators through Amazon Mechanical Turk (AMT), Overall the results obtained from AMT have been recognised to be of high quality, accurate and reliable [32].
In this study, we used irr Gamer, Lemon [33] and psych Revelle [34] R packages to measure Cronbach's () and Fleiss's (respectively, in conformity with the literature [35].Landis and Koch [36] provide a useful benchmark for interpreting Fleiss  and is adopted in this study (Table 1).2. Before applying exclusion criteria, accuracy and reliability were assessed between the manually classified data and algorithm classification to evaluate the extent to which the results will differ after the exclusion criteria are applied.This decision was to measure the quality of interannotator results and to make comparisons between human annotations and algorithm classification simpler.Overall, the performance of our algorithm compared to human annotation indicates good agreement (= .770)and also good agreement for the inter-annotator agreement (= .775).Our result is better compared to = .55value obtained in Provoost, Ruwaard [37] and = .73 reported in Sloan and Morgan [38].Nevertheless, the validation analysis in this study, indicates that the results of our hate speech algorithm is good and is potentially useful for further analysis.
Various machine learning and deep learning models have been used to tackle the problem of hate speech detection, as given in the literature review.Based on the analysis conducted, we have implemented the Naïve Bayes Classifier, Random Forrest (RF), Support Vector Machines (SVM) model and Recurrent Neural Network (RNN) model (Table 3).Our proposal is a methodology for identifying hate speech and its targets that integrates linguistic, social, and emotional characteristics.Our framework consists of an emotion classifier that uses two separate approaches: lexicon-based and machine learning, to predict the emotions exhibited by the targets and consumers of hate speech.A lexicon-based method is a strategy that uses a pre-established set of terms or expressions that are connected to particular feelings, such hateful, neutral, and not hateful.Machine learning methods are used to extract complicated patterns from data and generate predictions from them.

Conclusion
In this study, we used a straightforward yet innovative method to identify hate speech in tweets.To find the best working model for this data, a variety of machine learning models are trained and assessed.Evaluation metrics like accuracy, precision score, recall score, and f1 score are computed.With an accuracy of 83%, the Support Vector Machine (SVM) Classifier model produced the best results, according to the data.The algorithm's performance was successfully manually validated with the help of Amazon Mechanical Turk (AMT).
In summary, social sensing is a potential method for tracking and identifying hate speech on social media sites.Social sensing enhances the accuracy and resilience of the detection algorithms by utilizing the textual, social, and emotional characteristics of users and their posts to identify the subtleties and settings of hate speech.The primary targets and origins of hate speech, as well as the chronological and spatial patterns of its dissemination, can all be found with the aid of social sensing.However, there are several difficulties that social sensing must overcome, including a lack of data, moral dilemmas, and cross-domain adaptability.Thus, future studies ought to concentrate on creating more efficient and morally sound procedures for gathering, marking, and examining social media data to identify hate speech.

Figure 1
Figure 1 Conceptual model of the research proposal

Table 2
Inter-rater reliability test between human annotators and algorithm

Table 3
Machine learning algorithms implemented for hate speech detection