How to evade hate speech filters with "love"

It has often been suggested that text classification and machine learning techniques can be used to detect hate speech. Such tools could then be used for automatic content filtering, and perhaps even law enforcement. However, we show that existing datasets are too domain-specific, and the resulting models are easy to circumvent by applying simple automatic changes to the text.

 

The problem of hate speech and its detection

Hate speech is a major problem online, and a number of machine learning solutions have been suggested to combat it.

The vast majority of online material consists of natural language text. Along with its undeniable benefits, this is not without negative side-effects. Hate speech is rampant in discussion forums and blogs, and can have real world consequences. While individual websites can have specific filters to suit their own needs, the general task of hate speech detection involves many unanswered questions. For example, it is problematic where we should draw the line between hateful and merely offensive material.

Despite this, multiple studies have claimed success at detecting hate speech using state-of-the-art natural language processing (NLP) and machine learning (ML) methods. All these studies involve supervised learning, where a ML model is first trained with data labeled by humans, and then tested on new data which it has not seen during training. The more accurately the trained model manages to classify unseen data, the better it is considered.

While a number of labeled hate speech datasets exist, most of them are relatively small, containing just some thousands of hate speech examples. So far, the largest dataset has been drawn from Wikipedia edit comments, and contains around 13 000 hateful sentences. In the world of ML, these are still small numbers. Other datasets are typically taken from Twitter. They may also differ in the type of hate speech they focus on, such as racism, sexism, or personal attacks.

Google has also developed its own “toxic speech” detection system, called Perspective. While the training data and model architecture are unavailable to the public, Google provides black-box access via an online UI.

We wanted to compare existing solutions with one another, and test how resistant they are against possible attacks. Both issues are crucial for determining how well suggested models could actually fare in the real world.

Applying proposed classifiers to multiple datasets

Datasets are too small and specific for models to scale beyond their own training domain.

Most prior studies have not compared their approach with alternatives. In contrast, we wanted to test how all state-of-the-art models perform on all state-of-the-art datasets. We gathered up five datasets and five model architectures, seven combinations of which had been presented in resent academic research.

The model architectures differed in input features and the ML-algorithm. They either looked at characters and character sequences, or alternatively at entire words and/or word sequences. Some models used simple probabilistic ML algorithms (such as logistic regression or a multilayered perceptron network), while others used state-of-the-art deep neural networks (DNNs). More details can be found in our paper.

To begin with, we were interested in understanding how ML-algorithms differ in performance when trained and tested on the same type of data. This would give us some clue as to what kinds of properties might be most relevant for classifying something as hate speech. For example, if character frequencies were sufficient, simple character-based models would suffice. In contrast, if complex word-relations are needed, recurrent neural network DNN-models (like LSTMs or CNNs) would fare better.

We took all four two-class model architectures, and trained them on all four two-class datasets, yielding eight models in total. Next, we took the test sets from each dataset, and applied the models to those. The test sets were always distinct from the training set, but derived from the same dataset. We show the results in the two figures below (datasets used in the original studies are written in bold.)

Performance of ML-algorithms on different datasets (F1-score)

Performance of models with different test sets (F1-score)

Our results were surprising on two fronts. First, all models trained on the same dataset performed similarily. In particular, there was no major difference between using a simple probabilistic classifier (logistic regression: LR) that looked at character sequences, or using a complex deep neural network (DNN) that looked at long word sequences.

Second, no model performed well outside of its training domain: models performed well only on the test set that was taken from the same dataset type that they were trained on.

Our results indicate that training data was more important in determining performance than model architecture. We take the main relevance of this finding to be that focus should be on collecting and labeling better datasets, not only on refining the details of the learning algorithms. Without proper training data, even the most sophisticated algorithms can do very little.

Attacking the classifiers

Hate  speech classifiers can be fooled by simple automatic text transformation methods.

Hate speech can be considered an adversarial setting, with the speakers attacking the people their text targets. However, if automatic measures are used for filtering, these classifiers might also be attack targets. In particular, these could be circumvented by text transformation. We wanted to find out whether this is feasible in practice.

Text transformation involves changing words and/or characters in the text with the intent of altering some property while retaining everything else. For evading hate speech detection, this property is the classification the detector makes.

We experimented with three transformation types, and two variants of each. The types were:

1. word-internal changes:
   

   typos (I htae you) 

   leetspeak (1 h4te y0u)
2. word boundary changes:
   

   adding whitespace (I ha te you)

   deleting whitespace (Ihateyou)
3. word appending:
   

   random words (I hate you dog cat...)

   "non-hateful" words (I hate you good nice...)

Details of all our experiments are in our paper. Here, we summarize three main results.

• Character-based models were significantly more robust against word-internal and word boundary transformations.
• Deleting whitespace completely broke all word-based models, regardless of their complexity.
• Word appending systematically hindered the performance of all models. Further, while random words did not fare as well as “non-hateful” words from the training set, the difference was relatively minor. This indicates that the word appending attack is feasible even in a black-box setting.

The first two results are caused by a fundamental problem with all word-based NLP methods: the model must recognize the words based on which it does all further computation. If word recognition fails, so does everything else. Deleting all whitespaces makes the entire sentence look like a single unknown word, and the model can do nothing useful with that. Character-models, in contrast, retain the majority of original features, and thus take less damage.

Word appending attacks take advantage of the fact that all suggested approaches treat the detection task as classification. The classifier makes a probabilistic decision of whether the text is more dominantly hateful or non-hateful, and simply including irrelevant non-hateful material will bias this decision to the latter side. This is obviously not what we want from a hate speech classifier, as the status of hate speech should not be affected by such additions. Crucially, this problem will not go away simply by using more complex model architectures; it is built into the very nature of probabilistic classification. Avoiding it requires re-thinking the problem from a novel perspective, perhaps by re-conceptualizing hate speech detection as anomaly detection rather than classification.

The "love" attack 

Deleting whitespaces and adding "love" broke all word-based classifiers.

Based on our experimental results, we devised an attack that uses two of our most effective transformation techniques: whitespace deletion and word appending. Here, instead of adding multiple innocuous words, we only add one: “love”. This attack completely broke all word-based models we tested, as well as severely hindered character-based models. However, character-models were much more resistant against it, for reasons we discussed above. (Take a look at our paper for more details on models and datasets.)

The "love" attack applied to seven hate speech classifiers

We also applied the “love” attack to Google Perspective, with comparable results on all example sentences. This indicates that the model is word-based. To improve readability, the attacker can use alternative methods of indicating word boundaries, such as CamelFont.

Example of the "love" attack applied to Google Perspective

 

Conclusions: can the deficiencies be remedied?

Our study demonstrates that proposed solutions for hate speech detection are unsatisfactory in two ways. Here, we consider some possibilities for alleviating the situation with respect to both.

First, classifiers are too specific for particular text domains and hence do not scale across different datasets. The main reason behind this problem is the lack of sufficient training data. In contrast, model architecture had little to no effect on classification success. Alleviate this problem requires focusing further resources on the hard work of manually collecting and labeling more data.

Second, existing classifiers are vulnerable to text transformation attacks that can easily be applied automatically in a black box setting. Some of these attacks can be partially mitigated by pre-processing measures, such as automatic spell-checking. Others, however, are more difficult to detect. This is particularly true of word appending, as it is very hard to evaluate whether some word is "relevant" or simply added to the text for deceptive purposes. Like we mentioned above, avoiding the word appending attack ultimately requires re-thinking the detection process as something other than probabilistic classification.

Character-models have a far superior resistance to word boundary changes than word-models. This is as expected, since most character sequences are still retained even if word identities are destroyed. As character-models performed equally well to word-models in our comparative tests, we recommend using them to make hate speech detection more resistant to attacks.

Common sense applications of trusted hardware

Hardware security mechanisms like ARM TrustZone and Intel SGX have been widely available for a while and the academic research community is finally paying attention. Whether these mechanisms are safe and useful for ordinary people is hotly debated. In this post, we sketch two of our "common sense" applications of hardware-based trusted execution environments.

Hardware security mechanisms in commodity devices like smartphones (ARM TrustZone) and PCs (TPMs, Intel SGX) have been deployed for almost 15 years! But most of this time, ordinary app developers have not had the means to actually make use of these mechanisms. Intel made their SGX SDK publicly available two years ago. This has dramatically changed the awareness and interest in hardware security mechanisms among developers and researchers. When we published the first academic research paper on TrustZone almost a decade ago, most academics didn't even recognize the phrase "trusted execution environment (TEE)". This year NDSS had an entire session for TEEs and reportedly the most common word in the titles of the papers rejected at Usenix SEC was "SGX"!

Despite increased awareness, mentions of "trusted hardware" seem to elicit a general sense of mistrust among developers and researchers. One concern is that it requires having to trust the manufacturer, which is of course a reasonable concern. We often hear statements that are not exactly true, like "TEEs were designed for digital rights management (DRM)." For example, the genesis of ARM TrustZone, which dates back to the design of "Baseband 5 security architecture" in Nokia, was motivated not by DRM but by demands for technical mechanisms to comply with various requirements including regulatory requirements for type approval (note: the link points to a video). Of course, once TEEs were available, people did try to use them for DRM or other applications that can be perceived as unfairly limiting the freedom of users.

Nevertheless, hardware security mechanisms are strong enablers that can be used to improve the security and privacy of end users. In this post, we illustrate two such "common sense applications of TEEs" resulting from our recent work.

Private membership test

How can you check whether that new app you recently downloaded is actually malicious software?

At first glance, it may appear that there is a well-known solution for the above question - you simply install anti-malware software (often called "anti-virus") and use it to scan all the apps on your device. In the past, this anti-malware software would periodically download a database of known malware signatures, and compare your apps against this database. Of course this approach is not always 100% accurate (e.g. the database might not contain a complete list of all malware signatures), but nevertheless it can still be used to identify known malware.

However, performing this type of malware checking on a mobile device is very inefficient because every device would have to download the full malware database and expend its own energy to perform the comparisons. A much more efficient approach is to host the malware database in the cloud and allow users to upload the signatures of their apps to the cloud for checking. Statistics from Yahoo Aviate have shown that the average user installs approximately 95 apps, but according to G Data, the number of unique mobile malware samples is in the millions. Clearly it is far more efficient for each user to upload a list of 95 signatures than to download a database of millions. This scheme is also advantageous for the anti-malware vendors as they can update their databases in the cloud very frequently, and they need not reveal the database in its entirety to all users. But although it is very efficient, this approach raises serious privacy concerns! It has been shown that the selection of apps installed on a user's device reveals a lot of information about the user, some of which may be privacy-sensitive.

To overcome this challenge, we proposed a solution that makes use of a server-side TEE to perform a private-membership test. As shown in the diagram below, remote attestation is used to assure the user that she is communicating with our TEE. The user then uploads her app (or its hash) directly to the TEE, which checks if it is in the dictionary. One critical consideration in this approach is that the adversary can potentially monitor the memory access pattern of the TEE, so the TEE cannot just perform a normal search of the database. Instead we proposed a new design called a carousal, in which the entire dictionary is continuously circulated through the TEE. As we discovered through implementing this system, with the right data structures to represent the dictionary, this carousal approach can achieve surprisingly good throughput, even beating competing oblivious RAM (ORAM) schemes, because it can process batches of queries simultaneously.

The Carousal approach to prevent leaking information about queries from the TEE.

Also, as we point out in a recent PETS paper, private membership test (and more generally, private set intersection with unequal set sizes) has many other applications beyond cloud-assisted malware checking. One example we mentioned in the PETS paper is contacts discovery in a (mobile) messaging service: when a user installs a messaging client, it checks to see whether any contacts on the user's address book are already using the messaging service. As it happens, Signal recently announced their implementation of a private contacts discovery mechanism using SGX which essentially follows the design we described in our work. It is always gratifying when real world systems deploy innovations from research!

Protecting password-based authentication systems

How can you be sure your passwords are kept safe, even when sent to a compromised server?

Passwords are undoubtedly the most dominant user authentication mechanism on the web today. This is unlikely to change in the foreseeable future because password-based authentication is inexpensive to deploy; simple to use; well understood; compatible with virtually all devices; and does not require users to have any additional hardware or software.

However, password-based authentication faces various security concerns, including phishing, theft of password databases, and server compromise. For example, the adversary could attempt to trick users into entering their passwords into a phishing site, and then use these to impersonate the users. Alternatively, the adversary could steal a database of hashed passwords from a web server and use this to guess users' passwords. Most worryingly, a stealthy adversary might even compromise an honest server and siphon off passwords in transit as users login. Users' tendency to re-use passwords across different services further exacerbates these concerns.

Current approaches for protecting passwords are not fully satisfactory for various reasons: they typically address only one of the above challenges; they do not provide users with any assurance that they are in use; and they face deployability challenges in terms of cost and/or ease-of-use for service providers and end users alike.

To address these challenge, we have developed SafeKeeper, a comprehensive approach to protect the confidentiality of passwords on the web. Unlike previous approaches, SafeKeeper protects against very strong adversaries, including so-called rogue servers (i.e., either honest servers that have been compromised, or actively malicious servers), as well as sophisticated external phishers.

At its core, SafeKeeper uses a server-side TEE to perform a keyed one-way function on users' password, using a key that is only available to the TEE. This immediately prevents offline password guessing, and thus eliminates the threat of password database theft. Furthermore, using remote attestation, users can establish a secure channel directly from their browsers to the TEE, and receive strong assurance that they are indeed communicating with a legitimate SafeKeeper TEE. When used correctly, these powerful capabilities protect the user against password phishing and ensure that even if the server is compromised (or actively malicious), their passwords will be kept safe.

SafeKeeper: Protecting passwords even against an actively malicious web server.

We have implemented SafeKeeper's server-side password protection service using Intel SGX and integrated this with the PHPass password processing framework, which is used in multiple popular platforms, including WordPress and Joomla. We have developed the client-side component as a Google Chrome plugin. Our 86-participant user-study confirms that users can use this tool to protect their passwords on the web.

Epilogue: Enabling more common-sense applications with Intel SGX

In the process of designing and implementing these applications, we encountered a couple of challenges to be solved before the applications can be deployed at scale. The first is that SGX enclaves can only run in "release mode" (i.e., with full memory encryption etc.) if they have been signed by an authorized key, which requires a commercial use license from Intel. However, we expect that this would not be a challenge for a company to obtain. The second challenge is that we would like anyone to be able to verify the remote attestation of our enclaves. At the moment, verifying SGX attestation requires users to register their public keys with the Intel Attestation Service (IAS). It is probably not realistic to expect all our end users to do this, so for SafeKeeper we implemented a proxy that accepts attestation quotes from clients, sends them to IAS for verification (using our registered key), and returns the results to the client. Fortunately, IAS signs the verification result with a published Intel key, so clients can verify the response without having to trust our proxy. We are still investigating whether this is the optimal solution going forward.


- Andrew Paverd and N. Asokan

Disclaimer: By way of full disclosure, the authors are part of the Intel Collaborative Research Institute for Secure Computing which receives funding from Intel. But the views in this article reflect the opinion of the authors.