Colluding Adversaries in Machine Learning Pipelines

 TL;DR: Current research treats ML adversaries as independent actors with distinct objectives. However, real-world adversaries often collude with each by executing attacks in tandem. We present a framework to study this collusion; we can identify factors enabling it, and provide guidelines for predicting unexplored cases.

Link to Webpage: https://ssg-research.github.io/mlsec/interactions/
Code: https://github.com/ssg-research/sok-collusion
Paper Link: https://arxiv.org/abs/2606.10091

Problem Statement

Adversaries with different objectives in an ML pipeline can execute distinct attacks. Although these attacks are typically studied independently, collusion can arise when adversaries coordinate to use the outcomes of one attack to improve another. Such collusion is common in other domains, e.g., cryptocurrency pump-and-dump schemes and cybercrime markets, where adversaries share resources or monetize attack outcomes. Similar incentives exist in ML pipelines through attack information markets or cost sharing, motivating the need to investigate colluding adversaries.

However, existing work lacks a systematic framework to identify collusion opportunities, understand the role of adversary characteristics, or predict the potential for collusion. A unified framework may help identify new threats, design stronger red-teaming mechanisms, and develop effective defenses. This work presents the first framework for analyzing factors that enable collusion. This raises the following questions: What are the underlying factors that aid collusion among adversaries, and how can we predict the potential for collusion?

Our framework covers two collusion types identified from prior work: (i) between
train- and test-time adversaries (→ “train-inference collusion”), and (ii) among test-time adversaries (→ “test-time collusion”). We observe that in both types of collusion, a successful attack changes certain factors (such as overfitting), which can influence the effectiveness of other attacks relying on those factors. However, these factors differ between both collusion types (described below). By studying these factors, we show that we can not only explain collusion in prior work but also predict unexplored cases of collusion. 

Train-inference Collusion

In case of train-inference collusion, the first adversary is train-time adversary, typically executing poisoning attacks, while the second adversary is an inference-time adversary. For instance, Adversary-1 can execute poisoning during train-time to increase effectiveness of inference-time privacy attacks (e.g., membership/attribute/distribution inference, data reconstruction) executed by Adversary-2. For train-inference collusion, poisoning changes the factors underlying overfitting and memorization which can impact the effectiveness of inference-time attacks that rely on those factors (Figure 1). Here, the framework includes the factors underlying overfitting and memorization, which can be manipulated by the train-time adversary through poisoning.

Figure 1 (Train-inference Collusion): Adversary-1 is a train-time adversary that uses attacks such as poisoning to manipulate factors affecting overfitting and memorization. Since these factors influence inference-time attacks by Adversary-2, poisoning-induced changes can improve their effectiveness.

By understanding how poisoning changes key factors and how these factors affect inference-time attacks, we can predict collusion potential, i.e., whether poisoning can improve inference-time attack effectiveness. We identify three factors influencing overfitting and memorization that adversaries can control: (a) training data distribution tail length, and model observability differences across (b) datasets and (c) demographic subgroups. Figure 2 shows how poisoning affects these factors (left column) and how factor changes correlate with inference-time attack effectiveness (right column).

Figure 2 (Relation of Factors with Attacks): We identify how poisoning affects factors related to overfitting and memorization (left), and how changes in these factors influence inference-time attack effectiveness (right), with correlations represented as arrows.

To determine collusion potential, we evaluate whether poisoning can favorably alter factors that improve inference-time attack effectiveness. Specifically, by comparing the direction of changes between poisoning with factors, and change in factors with inference-time attacks, we identify whether poisoning can enhance inference-time attacks, indicating potential for collusion (<↑, ↑>).

Inference-time Collusion

In inference-time collusion, both the adversaries execute attacks during inference-time. For instance, the first adversary can find adversarial examples for evasion to increase effectiveness of privacy attacks executed by the second adversary. For inference-time collusion, knowledge inferred from the first attack is treated as an outcome that satisfies the prerequisite for the second attack. Here, factors related to adversary's knowledge (e.g., of training data, target model) are included as part of the framework. 

Figure 3 (Inference-time Collusion): Adversary-1 executes inference-time attacks to learn information about the target model and dataset, which serves as prerequisites for improving other inference-time attacks.

Hence, by identifying attack outcomes that serve as prerequisites for other attacks, we can predict the potential for collusion. We identify four key factors related to knowledge of the target model and its training data: (a) quality and overlap of the adversary's auxiliary data, (b) similarity between the auxiliary and training data distributions, (c) functional similarity between the shadow and target models, and (d) architectural similarity between the shadow and target models. While many inference-time attacks depend on these factors, successfully executing some attacks also reveals or improves them. This enhanced knowledge enables more accurate shadow models, increasing attack transferability and the potential for collusion.
  
Figure 4 (Relation of Factors with Attacks): We identify how executing one inference-time attack correlated with the change in factors (left column); and how change in factors correlate with the second attack effectiveness (right column).

To determine the potential for collusion among inference-time adversaries, we identify the outcomes of different attacks (Figure 4: left column) and the factors correlated with attack effectiveness (Figure 4: right column). Two adversaries executing different inference-time attacks have collusion potential if an outcome of the first satisfies a prerequisite of the second (indicated by <↑, ↑>).

Summary

We validate our guideline by (a) applying it to collusion cases reported in prior work to determine whether it explains the observed empirical results, and (b) comparing its predictions against empirical evaluations of five previously unexplored collusion scenarios. Our results show that the guideline both explains known collusions and accurately predicts new ones. Overall, by identifying factors shared across attacks, we provide a guideline for predicting collusion potential and demonstrate its ability to explain and predict collusion, while also highlighting the influence of adversary characteristics (see the paper for details).

Artifact for Evaluating Colluding Adversaries

Our codebase supports five empirical interactions: one for train-time inference and four for inference-time collusion, cross attacks including data poisoning, model extraction, distribution inference, data reconstruction, and membership inference. We find low collusion potential between poisoning and model extraction, but high potential between model extraction and distribution inference, and between data reconstruction and distribution inference, membership inference, and attribute inference.

The codebase can be extended into a library covering various attacks, enabling researchers and practitioners to systematically evaluate collusion potential, validate our guidelines, and identify worst-case attack combinations. We invite the community to build on this foundation to explore additional attacks, variants, and unexplored collusion scenarios.

Acknowledgements: This work was supported in part by Wallenberg Visiting Professor Program, the Natural Sciences and Engineering Research Council of Canada (grant number RGPIN-2026-04826), and the Government of Ontario (RE011-038).

Deployment Concerns for ML Systems: Unintended Interactions

Existing research focuses on defenses against individual machine learning (ML) risks. However, trustworthy ML deployment also requires addressing challenges that arise when mitigating multiple risks simultaneously, such as unintended interactions. We identify and explore three such unintended interactions in ML.

Overview

Current trustworthy ML research largely focuses identifying and mitigating risks within individual pillars of trust, namely, security, privacy, fairness, transparency, and safety. While essential, this is insufficient for trustworthy deployment, which also requires addressing additional problems that arise when mitigating multiple risks simultaneously. One such problem is unintended interactions in ML. Over the past two years, we have explored three types of unintended interaction: (a) defense against one risk may increase or decrease other unrelated risks; (b) conflicts among defenses can decrease their effectiveness when combined; and (c) potential for collusion among adversaries can enable executing an attack to amplify others. We show that such interactions can be understood and predicted through factors governing defense effectiveness and risk susceptibility, and propose frameworks to study such interactions.

Unintended Interactions among Defenses and Unrelated Risks

Defenses targeting one risk can inadvertently increase or decrease susceptibility to other unrelated risks. For example, adversarial training (against the risk of evasion) may increase susceptibility to membership inference, and discriminatory behavior. However, a systematic understanding of the underlying causes of such interactions is lacking in the literature. Identifying these causes can enable practitioners to predict unexplored interactions without expensive empirical evaluation. However, this is challenging due to the incomprehensible behavior of ML models. This raises the following research question: what are the factors underlying unintended interactions among defenses and unrelated risks, and how can we effectively predict unexplored ones?

We developed the first framework to systematically explore unintended interactions based on the conjecture that overfitting and memorization are the primary causes. This is because different risks depend on factors that influence overfitting and memorization, and an effective defense that changes these underlying factors, in turn affects the susceptibility to the risks. Our framework includes various factors that influence overfitting and memorization: characteristics of the training dataset (e.g., size, distribution), the model (e.g., capacity), and the objective function (e.g., distance to boundary, loss curvature). Using this framework, we propose a guideline to explain interactions previously explored in the literature, and conjecture about unexplored ones by examining how a defense changes a factor, and how a change in that factor correlates with a change in susceptibility to an unrelated risk. Using this guideline, we conjecture two previously unexplored interactions and empirically validate them. This work was published in IEEE Symposium on Security and Privacy (S&P) 2024, and received a distinguished paper award (see blog). This work also resulted in Amulet, an open-source library for evaluating unintended interactions, which received funding from Intel.

Combining ML Defenses without Conflicts

Real-world models must simultaneously protect against multiple risks to security, privacy, and fairness. Combining existing defenses is plausible, but conflicting objectives can reduce the effectiveness of the constituent defenses. This leads to asking the following: how can we effectively combine ML defenses while avoiding conflicts? We need an easy-to-use combination technique that should be (a) accurate (correctly identifies whether a combination is effective), (b) scalable (allows two or more defenses to be combined), (c) non-invasive (requires no changes to the defenses being combined), and (d) general (applicable to different types of defenses).

We propose Def\Con, the first combination technique that meets all requirements by effectively combine defenses based on whether the defenses will conflict with each other. Def\Con identifies potential conflicts based on where each defense is applied in the ML pipeline (pre-, in-, or post-training), and whether the underlying mechanisms of the defenses interfere (e.g., a later defense minimizes a risk used by an earlier defense). Since these steps do not depend on specific defenses, Def\Con is generally applicable to various defenses. This helps practitioners identify compatible defenses that can be effectively combined, i.e., their effectiveness remains comparable to applying each defense separately. We show that Def\Con is more accurate than prior baselines, and correctly predicts seven out of the eight previously explored combinations, and 27 out of 30 unexplored combinations that we evaluated empirically. We also show, for the first time in the literature, that it is possible to effectively combine three defenses (i.e., Def\Con is scalable). Finally, Def\Con combines existing defenses without modification (i.e., Def\Con is non-invasive). This work was published in the Transactions on Machine Learning Research (TMLR), 2025.

Colluding Adversaries in ML Pipelines

So far, while answering the above two unintended interactions, we assumed that adversaries are independent actors in the ML pipeline with distinct objectives targeting different risks. However, practitioners must consider the potential for collusion among adversaries within the ML pipeline, where an adversary executes an attack (by exploiting the vulnerability related to the corresponding risk) to amplify the effectiveness of attacks by other adversaries. For instance, poisoning to increase effectiveness of privacy attacks (e.g., membership/attribute/distribution inference, data reconstruction), or using adversarial examples for evasion to increase effectiveness of privacy attacks. Existing work lacks a systematic framework to explore potential colluding adversaries, and to study the implications of the adversaries' characteristics for collusion. A unified framework can help practitioners and researchers (i) identify new threats against ML systems, (ii) design strong attacks for auditing, and (iii) design effective defenses against such attacks. This raises the following question: what are the underlying factors that aid the potential for collusion among adversaries, and how can we predict the collusion potential?

We present the first framework to explore two collusion types: (i) between train- and inference-time adversaries (or "train-inference collusion"), and (ii) among inference-time adversaries (or "inference-time collusion"). In both cases, we observe that a successful attack changes certain factors, which can influence the effectiveness of other attacks relying on those factors. However, these factors differ between both collusion types. For train-inference collusion, poisoning changes the factors underlying overfitting and memorization which can impact the effectiveness of inference-time attacks that rely on those factors. Here, the framework includes the factors underlying overfitting and memorization, which can be manipulated by the train-time adversary through poisoning. For inference-time collusion, knowledge inferred from the first attack is treated as an outcome that satisfies the prerequisites for the second attack. Here, factors related to adversary's knowledge (e.g., of training data, target model) are included as part of the framework. Using the common factors among attacks, we present a guideline to predict the potential for collusion. We validate the guideline empirically on five unexplored case of collusion, and analyze the effect of the adversary's characteristics on the collusion. This work was published in the USENIX Security Symposium (USENIX Sec), 2026.

Acknowledgements: This work was supported in part by Intel (as part of the Private AI Collaborative Research Institute, Wallenberg Visiting Professor Program, the Natural Sciences and Engineering Research Council of Canada (grant number RGPIN-2026-04826), and the Government of Ontario (RE011-038).

Making Software Both Memory-Safe and Leak-Proof: Introducing BLACKOUT

Two major challenges stand in the way of protecting critical software like cryptographic libraries that secure websites, messaging apps, and smartphone communications: memory-safety and side-channel leakage.

The memory-safety hardening challenge

Memory-safety means making sure a software application only reads from and writes to memory it’s supposed to. Without it, applications can inadvertently contain memory-safety vulnerabilities, flaws in software that can allow attackers to hijack programs, steal secrets, or crash systems. Unfortunately, most of today’s foundational software is written in C and C++, languages that are not memory-safe.

This has been a problem for decades. Regulators and security agencies such as the US CISA and the UK NCSC are now urging the move to memory-safe languages like Rust, or to new hardware that enforces memory safety, like CHERI (Capability Hardware Enhanced RISC Instructions).

Rust is promising, but rewriting the billions of lines of C and C++ code in existence is not realistic. That’s why memory-safe hardware like CHERI, which can make old code safer without rewriting it, is so attractive.

The side-channel leakage problem

Even if memory is safe, secrets can still “leak” through side-channels.

A side-channel is an indirect way an attacker can spy on a program—for example, by measuring how long it takes to run, or watching how it uses the CPU cache. These tiny clues can reveal sensitive information like encryption keys which must remain confidential for encrypted data to remain private.

Developers try to prevent this by writing constant-time code, which takes exactly the same amount of time to run no matter what the secret data is. That way, an attacker can’t learn any new information merely by timing it.

But constant-time programming is hard. A single misplaced if statement, or an optimization by the compiler, can reintroduce leaks. Unlike memory bugs, which usually cause crashes and get noticed, side-channel flaws often fail silently. You might not know anything is wrong until after an attacker steals data.

Rust makes things trickier: its high-level abstractions are nice for developers, but they make it even harder to guarantee true constant-time behavior once the compiler generates machine code.

Why hardware matters

Side-channel resistance isn’t just about software. It’s about the whole system: hardware and software together.

CHERI hardware prevents memory-safety issues but doesn’t directly fix side-channel leaks. Some research has hardened CHERI against speculative-execution attacks (think Spectre or Meltdown), but those don’t cover all the ways secrets can leak.

Other approaches exist, like adding special “tag bits” to memory to mark which data is sensitive, but those come with heavy performance costs and extra memory usage.

Introducing BLACKOUT

Working with experts from Ericsson Research, we have designed BLACKOUT: a hardware-software co-design that extends CHERI to tackle both problems—memory-safety and side-channel leakage—at the same time. 

 

We designed BLACKOUT to build on CHERI’s memory-safety features, which replace all memory references in C and C++ programs with capabilities—secure objects that include information about the memory they can access. This prevents buffer overflows, stops capabilities from being mistaken for regular data, and—together with a CHERI-aware memory allocator—blocks access to freed and reused memory.

How it works:

BLACKOUT adds special blinded capabilities to CHERI. When data is marked secret, the CPU always addresses it using a blinded capability. Inside the CPU, a special blindedness bit is used to track secret data loaded via a blinded capability. Any result of calculation based on secret data is automatically marked as secret too, and the CPU ensures that if secret data it is handling is written to memory, it must happen also via a blinded capability.

Crucially, if an application tries to use secret data in a way that risks leaking it—say, by using it to control program flow or memory addresses—the CPU halts the operation. This cuts off entire classes of side-channel leaks at the root.

In short: if secret data is used incorrectly, the program fails fast instead of silently leaking.

Writing software for BLACKOUT

The BLACKOUT processor blocks any program that could leak secrets, but writing constant-time code that runs correctly on it remains challenging. To ease this burden, we built a BLACKOUT programming model for CHERI C with Clang/LLVM support. Developers can mark secret data with a “blinded” annotation, and the hardware enforces security by faulting if that data is ever misused.

Domain experts can annotate further uses of secret data, while the compiler propagates these annotations where possible. If propagation is incomplete, developers can rely on hardware faults to reveal missing annotations until the program runs cleanly. The compiler can also warn the developer of clear leaks already at compile time.

This makes writing constant-time code much more reliable—and far less error-prone—than current approaches.

Developers can also use other constant-timeness verification tools together with BLACKOUT. But achieving constant-time software via static analysis or formal methods alone each face their own challenges. Capturing microarchitectural subtleties of real-world hardware in formal models (and keeping the models up-to-date as hardware evolves) is difficult. Static analyses may not be sound and can lead to over- or under-tracking. BLACKOUT closes this gap because it ensures constant-time implementations.

BLACKOUT hardware

We built BLACKOUT on a CHERI-RISC-V processor running on an FPGA, integrated it with the CheriBSD operating system, and ran industry-standard benchmarks.

The results show that BLACKOUT imposes

• almost no overhead compared to baseline CHERI for constant-time code (just 1.5%) and 
• only a moderate overhead compared to completely unprotected processors (around 23%).

BLACKOUT is the first unified approach to achieve both memory-safety and side-channel confidentiality  with minimal performance, which can pave the way to practical deployment.

Try It Yourself

The source code for BLACKOUT—both hardware and software—is available here: blindedcapabilities.github.io

Takeaways

• Memory-safety stops attackers from breaking software through memory bugs.  
• Side-channel resistance stops them from observing secrets indirectly.  
• BLACKOUT combines both, in hardware and software, with minimal performance costs.

This moves us a big step closer to building truly safe cryptographic systems for the real world.

Learn more

• For more information regarding BLACKOUT, read our paper: Data-Oblivious Computation with Blinded Capabilities to appear in the ACM Conference on Computer and Communications Security (CCS ’25) (also available on arXiv).
• For more information on data-oblivious execution, take a look at our other papers.

Unintended Interactions among ML Defenses and Risks

A significant amount of work has been done in understanding various individual security/privacy risks in machine learning models. However, more work is needed in understanding how a given defense interacts with other, unrelated, risks. We have been exploring this problem in our recent work including a systematization of knowledge (SoK) paper in the 2024 IEEE Security and Privacy Symposium. We are also building a software tool to facilitate systematic empirical exploration of such interactions.

Unintended Interactions between ML defenses and risks 

Machine learning (ML) models are susceptible to a wide range of risks including security threats like evasion, poisoning and unauthorized model ownership, privacy breaches through inference attacks, and fairness issues such as discriminatory behavior. Various defenses have been proposed to protect against each of these risks separately focusing on their effectiveness in addressing specific risks. Yet, using a defense may inadvertently increase or decrease susceptibility to other risks, leading to unintended interactions. Despite their practical relevance, such interactions have not yet been systematically explored in the research literature.

We first started by looking at pairwise interactions between simultaneous defenses (AAAI 2023). We looked at how deploying a model ownership verification technique (such as watermarking or fingerprinting) interacts with simultaneously deploying a defense against a different risk – we specifically looked at differential privacy and adversarial training. Our research revealed that often these defenses resulted in conflicts impacting the effectiveness of either defense and/or model utility.  This prompted us to systematically study the potential unintended interactions between an ML defense (intended to address a specific risk) and other (unrelated) risks.

As part of our systematization (IEEE SP 2024), we carried out an exhaustive survey of existing work: we conjectured that overfitting and memorization are the likely causes that underlie such unintended interactions, and identified several factors that can influence overfitting or memorization, including factors such as characteristics of a model, its training data, or its objective function. We find that by exploring how these different factors contribute to making a defense or a risk more or less effective, we can anticipate unintended interactions between the defense and the risk.

Table: Overview of different defenses and risks considered in our SoK: For RD1 (Adversarial training), the interactions with all the risks other than R1 (Evasion) are unintended interactions.

            

We present a framework to summarize how the effectiveness of a defense correlates with different influencing factors and how a change in a factor correlates with the susceptibility to different risks. Using this framework, we propose a guideline for studying the common factors between a given pair of  defense and (unrelated) risk to conjecture the nature of their interactions (whether the risk increases/decreases when the defense is effective). We empirically evaluated two such pairs that were not studied in prior work, and were able to show that our guideline is effective at predicting unintended interactions.

We are grateful to the program committee of the 2024 IEEE Security and Privacy Symposium for recognizing this work with a Distinguished Paper Award. 

Visit our project page for the papers, talks, and source code from this work

Based on this work, we identified the need for a software tool for systematic empirical evaluation of defenses and risks in ML models. Such a tool can facilitate systematic analysis of previously unexplored unintended interactions between defenses and risks. It can also serve as the means for systematic comparative evaluation of new defenses or attacks.

Next steps

We have been working on Amulet, an extensible, open-source pytorch library for systematic empirical evaluation of ML defenses and attacks. It currently has six defenses and eight attacks already integrated. We designed Amulet to be extensible so that ML researchers and analysts could easily add support for new defenses, attacks, and datasets. This summer, we expect to make Amulet available as open source under Apache license. We welcome the ML security/privacy community to try Amulet and contribute to making it better, by providing us feedback to improve it, and by integrating more defenses and attacks into Amulet.

Acknowledgements: This work was supported in part by Intel (as part of the Private AI Collaborative Research Institute and the Government of Ontario.

ML Security at SSG: Past, Present and Future

 “Security is a process not a product” – Bruce Schneier

This year, we wrap up an eight-year-long, machine learning (ML) focused research effort at Secure Systems Group at Aalto University. In that period, we went from using ML for security, to exploring whether ML-based systems are vulnerable to novel attacks and how to defend against them.

We started off by using ML to provide security guarantees to various systems. We used ML models to e.g., detect phishing websites, detect adversaries in IoT systems, or discover financial fraud in a payment platform. However, in the process, we realised that in many domains, ML models are not just an isolated component or a tool but the core of the system. This in turn, led us to look into the security of the models themselves.

We quickly learnt that models are quite brittle – they can be fooled using evasion attacks (both in vision, and text domains), stolen by a malicious client, and are difficult to use in a privacy-preserving manner. To address these issues, we spent years looking into ways of protecting them, focusing on model extraction and ownership. We and others have shown that model extraction attacks are a realistic threat. We proposed the first model watermarking scheme designed to deter model extraction. Fingerprinting schemes have emerged as one of the most promising defences against model extraction. We have highlighted concerns with leading fingerprinting schemes. In particular, we have highlighted that robustness against malicious accusers is an understudied aspect in the literature – we have shown that all existing watermarking and fingerprinting schemes are vulnerable to malicious accusers.

Having many people with industry background, in our research, we’ve continuously focused on how our ideas can be integrated into systems, and given attention to real-world deployment considerations. In particular, we raised a concern that is typically overlooked in academic literature: practitioners have to deploy defences against multiple security concerns simultaneously; sometimes these interact negatively. Understanding the interaction between different defences remains an important open problem.

All in all, our research output has been a collective effort of many researchers across two universities, and thanks to many collaborations – academic and industrial. Secure Systems Group continues its work at the University of Waterloo, continuing on a broad range of ML security and privacy topics including the exploration of how defences against a particular concern influence other concerns, and what hardware security mechanisms can be used to secure ML models.

Poster PDF presented at the annual SSG Demo Day 2023.

SSG ML research page.

Better visualisation of consensus protocols

Almost every BFT paper includes an "arrow diagram", showing who sends how many messages to whom and when. Despite their universality, these diagrams are difficult to interpret, leaving out many important details and including unimportant ones. In this blog post, we describe a new way to visualise BFT protocols that is easier to draw, easier to read, and easier to interpret.

Arrow diagrams in BFT

Earlier I presented a new Byzantine fault tolerance (BFT) protocol, SACZyzzyva, at SRDS 2019. Afterwards, I received a number of positive comments about this talk, and in particular about the fact that it did not include a single "arrow diagram".

If you don't know what I mean by an arrow diagram (or even if you do), then this article by James Mickens in ;login: provides an excellent introduction to the BFT literature, having this to say on arrow diagrams:

In a paper about Byzantine fault tolerance, the related work section will frequently say, "Compare the protocol diagram of our system to that of the best prior work. Our protocol is clearly better." The paper will present two graphs that look like Figure 2. Trying to determine which one of these hateful diagrams is better is like gazing at two unfathomable seaweed bundles that washed up on the beach and trying to determine which one is marginally less alienating.

Here is one that we included in our paper on SACZyzzyva:

Each arrow represents a message from one machine to another, and the difference in colour (grey vs black) shows how SACZyzzyva transactions can complete with two fewer rounds of communication than Zyzzyva transactions. This isn't the clearest diagram—it is difficult for the brain to pick out which lines it should be counting, so a quick skim of this figure doesn't do much except show whether each phase has O(1), O(n), or O(n²) messages with respect to the number of replicas.

But this diagram also hides some important features of the two protocols that can make a big difference to performance. First, Zyzzyva doesn't always need the extra rounds of communication—when all replicas respond, then it is just as fast as SACZyzzyva. If these extra rounds are needed rarely enough, then using SACZyzzyva might not be worthwhile. Secondly, it doesn't say anything about the size of the messages; the size of each of Zyzzyva's prepare messages actually increases O(n) with the number of replicas, giving it a bit-complexity of O(n²) whenever there are actually faults to tolerate. This suggests that SACZyzzyva may have better scalability than Zyzzyva in normal operation, but you can't see this in the arrow diagram.

Ball diagrams

If not arrow diagrams, then what? The key observation is that when comparing these diagrams, the reader doesn't pay much attention to the sender or recipient of the message, but only the number of messages. This lets us simplify the diagram dramatically:

In this diagram, it is much easier to see how many messages are sent in each round. We also show that the second phase is triggered by a timeout—meaning that there is a long wait between the two bracketed parts—but that it shouldn't happen very often—indicating that the first phase might be more important from a performance perspective.

Ball diagrams are also much less busy, making it easier to fit other information into the diagram:

Figure 3: The SACZyzzyva normal-case protocol, annotated with the changes relative to Zyzzyva's.

Showing message size

However, these diagrams don't show the relative sizes of the messages: the diagram shows the message complexity, but not the bit complexity. We can visualise this by making the area of each ball proportional to the size of its message. In the case of Zyzzyva, the request and order-request messages may be quite large relative to the spec-response messages, since they need to contain request data, and the size of the commit messages will depend on the number of replicas, since they contains 2f+1 signatures from other replicas:

As the number of replicas increases, both the number of messages and their relative sizes change, showing that as the number of replicas increases, scalability will ultimately be limited by the number and size of the order-request messages may be quite large relative to the spec-response messages, since they need to contain relevant data, and the size of the commit messages:

This fact is invisible on a regular arrow diagram.

Conclusion

This post introduced ball diagrams, new type of diagram for visualisation of BFT protocols. These diagrams are easier to read than the usual arrow diagrams that appear in most BFT papers and presentations, so if you are less interested in the source/destination information than the raw number of bits or messages, then consider giving these a try in your next presentation.

Historical insight into the development of Mobile TEEs

Today, Trusted Execution Environments (TEEs) are ubiquitous in mobile devices from all major smartphone vendors. They first appeared in Nokia smartphones fifteen years ago. Around the turn of the century, Nokia engineers in Finland played a crucial role in developing mobile TEE technology and paving the way for its widespread deployment. But this important story is not widely known as there is limited public documentation. To address this gap of “missing history”, we carried out an oral history project during the spring of 2019. In this post, we summarize our findings.

Historical insight into TEEs

Trust in mobile devices is a prerequisite for the modern mobile industry ranging from e-commerce to social media. Customers, companies, and regulators need to trust the mobile phone with sensitive information such as credit card numbers and fingerprints. A trusted execution environment (TEE) — a secure area that runs a computing system in parallel with, but isolated from, the main operating system — is central to modern mobile security. Through creating technical conditions for different stakeholders to rely on the mobile system, it has been an essential component in the development of the modern mobile business in general. 

During the spring of 2019, the Secure Systems Group at Aalto University hosted an oral history project on the development of  mobile TEEs. The project focused on the role played by Nokia experts in the emergence and establishment of mobile TEEs. We conducted a series of interviews with fifteen key actors: senior directors and managers, researchers, and security professionals. The aim was to increase the understanding of the mobile platform security systems today and to recognize the human actors behind their development and widespread deployment.

The starting point in any historical inquiry is that technological development is never self-evident nor pre-determined, neither does it takes place in isolation. Instead, technological systems are developed by individuals at a certain place and time, restricted by different economic, technical, regulatory, and other factors. Understanding the development of the mobile secure execution system helps us to master it today and to improve it tomorrow.

Emergence of mobile security

Communication is imperative for modern societies. The history of telecommunication goes back centuries from visual signalling, e.g., the semaphore system to the electronic telegraph and radio transmission. A common theme in the history of telecommunication is that security follows the communication technology with a delay. In Finland, the first public mobile network, the Autoradiopuhelin (ARP) and the first generation Nordisk MobilTelefon (NMT) system transmitted analog voice signal originally without any cryptographic protection. Yet, the security of the device itself was hardly recognized as a critical problem before the 1990s. The primary reason for that was the strictly regulated operation: both ARP (launched in 1971) and NMT (1981) were operated by governmental organisations. Mobile phones themselves were closed systems with few extra capabilities compared with traditional landline phones: The bulky and heavy equipment were best protected by doors and locks.  

 

The 1990s revolutionized mobile telecommunication. First, the GSM standard addressed the communication security problem by encrypting the signal and protecting the air-interface. Second, the deregulation of telecommunication opened market for private companies which skyrocketed the number of operators within a short time. The physical dimensions of mobile handsets shrank, making them attractive targets for thieves. Finally, after the mid-1990s, support for third-party applications written in the Java programming language that could be downloaded from the Internet transformed phones rapidly from closed devices to open systems that increasingly started to resemble small, general-purpose computers. Resulting from more users, less governmental control over the industry, and more sources of potential vulnerabilities, device security emerged as a new problem in the design of mobile phones. 

The security of the phone under these new conditions referred merely to the integrity of the device: Regulators and mobile network operators came up with novel needs to protect certain pieces of information inside the phone from unauthorized changes after the phone left the assembly line. In particular, the regulators wanted a secure storage for the device identity (International Mobile Equipment Identity, IMEI) and for certain parameters such as those for radio frequency transmission, which could affect the safety of the phone and functionality of the mobile network. Mobile operators, which were the principal customers for Nokia, needed a strong subsidy lock mechanism (colloquially known as “SIM locks”) that would tie the phone to a certain operator for a predetermined time. 

Security of Nokia’s Digital Core Technology (DCT) generation phones was enforced with mainly software solutions and protected by secrecy within the organization: even security professionals had  little more than educated guesses about the structure and requirements of the DCT security architecture. The essential weakness of this kind of “security through obscurity” design is that after the secret designs are revealed, the protection is lost. The high market share of Nokia made it an attractive target for hackers. DCT4 generation brought in hardware component in the form of one-time-programmable memory but particularly in the case of the SIM locks, the economic motives to break the security system outstripped the technological capabilities to protect it. The profit losses of important customers increased pressures to design a better security architecture.  

Towards mobile platform security

The interest towards a coherent, hardware-enforced platform security stemmed from a team of engineers working with mobile payments and security. The initial idea was to introduce a separate security chip to implement physical isolation of security-critical processing. Yet, an additional hardware chip was deemed too expensive in the strictly cost-sensitive organization. At the turn of the millennium, a newly graduated engineer came up with an idea to implement a logically isolated security mode using just one chip. A new status bit was introduced in the processor, which determined the status of data stored in memory and whether or not the processor was in a secure mode. This “secure processor environment” design was adopted as the fundamental cornerstone of the next baseband generation: Baseband 5 (BB5). 

In the history of technology no technological innovations really emerge out of nowhere but  are always influenced by preceding ideas and inventions. Certain features of the secure environment, were decisively invented around already existing patents.  What was novel with Nokia’s solution was the combination of the software and hardware features to implement an architecture for mobile platform security which were deployed on a large-scale.

The launch of the first BB5 phone in 2004 was a major landmark in the development of Nokia’s mobile phones as it effectively ushered in the era of 3G phones. Less visible to the public but equally important was the changes introduced to the platform security model as a whole.     

First, it marked a switch from “security through obscurity” towards “security through transparency”. Open communication throughout the process, considerable level of transparency of the design, together with a public key infrastructure was required to develop a strong, usable, and cost-efficient security architecture. 

Second, security transformed from add-on feature into an integral part of the platform design. At the time when the early smartphones started to increasingly resemble general purpose computers in terms of their capabilities and function, the security engineers opted against following the PC world in the security design: instead of reactive firewalls and anti-virus tools, mobile security was better addressed proactively in the platform design.

Alliance of hardware and software

At the time when the secure processor environment was initiated, Nokia produced its own Radio Application Processor (RAP) chipsets. It simplified the efforts of making the hardware components to accommodate security requirements. Yet, the first BB5 phones were already equipped with Texas Instruments OMAP processors. Nokia and Texas Instruments had a close partnership which involved intense cooperation in chipset design. Later, Texas Instruments branded the technology as M-Shield. M-Shield stemmed from the same origin as  Nokia’s secure processor environment but was subsequently developed in a different direction. 

Around 2003, ARM proposed to develop a system-wide hardware isolation for secure execution for Nokia. Cooperation between ARM and Texas Instruments had its independent business goals separate from Nokia’s needs but it was also in Nokia’s interests. It provided Nokia with a possibility to implement a secure environment on any chip implementing ARM’s security architecture, which would later become known as ARM TrustZone. (A 2004 article is possibly the first public technical paper describing ARM TrustZone. At present, there is no official website hosting this article, but it appears to be stashed at this unofficial site).

Security as an enabler

A deep paradox in the development of security technology is that security is important to have but difficult to sell. The importance of security becomes apparent only when it does not work and its benefits for the business are rarely manifested by increased sales. In corporate management, security remained overshadowed by competition for customers’ satisfaction and optimization of global supply chains. Instead of a strategic R&D project, the secure processor environment proceeded as a technology-driven skunkworks of a handful of engineers  and researchers. The development of security technology outside the strategic spotlights was facilitated by Nokia’s organizational culture that granted technological experts with considerable room to maneuver. Also critical was that the security engineers successfully translated security from a problem into an enabler. E-commerce still remained a marginal use case at the beginning of the 2000s, SIM lock, IMEI protection, and later digital rights management (DRM) became the main business cases that justified the adjustment of the hardware and software architectures. 

Once the platform security architecture was accepted for product programs, hardware suppliers had adopted the secure processor environment in their designs, and complementary adjustments in manufacturing process and key distribution services were implemented, the security technology constituted an infrastructure for other applications and functions to take advantage of. These novel uses of the security infrastructure ranged from the rather trivial case of the protection of audio compression attributes of Nokia headphones to the widely influential use of security certificates for distinguishing among model variants during the manufacturing process.

Standardised trust

After the adoption of the hardware-enforced secure execution environment as a de facto internal standard, Nokia turned the attention onto the state-of-the art of mobile security standards in formal standards development organizations. Two prime rationales motivated the representatives of the company to take an active role in international standardization forums. First, an open standard that was revised in an international cooperation community, required no maintenance from Nokia, and was available for potential suppliers would facilitate competitive bidding in chipset production. Second, as the mobile standards were anyway going to take form in the future, Nokia wanted to make sure they would be compatible with the solution it had already adopted. 

At first, Nokia’s representatives chaired a mobile working group within Trusted Computing Group (TCG). Although being founded by PC companies, TCG was the only industrial forum working with hardware security standards for global use in the early 2000s. In 2007, TCG announced the first hardware security standard for mobile, Mobile Trusted Platform Module (mobile TPM, MTM), which became an ISO standard. It was different from Nokia’s secure processor environment but more importantly, it was compatible with it.

The concept of TEEs was first described publicly by Open Mobile TerminalPlatform (OMTP) in its specification for Advanced Trusted Environment in 2009. A while later, the center of TEE standardization became another industry forum, GlobalPlatform. With two industrial forums striving to standardize hardware-enforced mobile security, there was a risk that they would end up with mutually incompatible specifications. It was in Nokia’s interest to turn the forums from competitors into cooperators. In 2010, GlobalPlatform published its first TEE standard, TEE client API 1.0 that defines the communication between trusted applications which are executed in TEE, and applications executed by the main operating system. In 2012, GlobalPlatform and TCG announced the founding of a joint working group focusing on security topics.

After 2010, Nokia had less resources for extensive mobile device R&D projects or participation in international forums. Development of TEE technology continued even as Nokia's role in it diminished over time.  Today, TEE technology is widely deployed on mobile devices and is extensively used on both iOS and Android smartphones.

Some concluding remarks

History does not repeat itself. Lessons of past failures and success are not readily applicable in the future. Yet, historical insight into the development of mobile TEEs helps us to comprehend  comparable systems of today. In particular, despite the convergence between the PC and mobile worlds, the different approaches to platform security architecture still manifest the legacy of the past: The technological paths once taken create dependencies over time and continue shaping the framework in which the security engineers operate today. 

In addition, the constitutive role of only a few dedicated security professionals in just one company in the development and establishment of an international standard demonstrates the malleability of technological systems when they are still under construction. 

Finally, the concept of “security” resonates with the complex needs of authorities, customers, and users but translates into very different meanings for different stakeholders. Mobile security technology may protect the privacy of the user from corporations, or the business from its customers; it may also ensure the safety of the device or enable the law-enforcement to  access personal data in a lawful manner. Security technology is bound to the  interaction with its political, cultural, and economic environment and is always shaped by them.

The TEE-history project team:

Saara Matala, project researcher

Thomas Nyman, doctoral candidate

N. Asokan, professor

We interviewed fifteen former or current Nokia employees for this project: two senior executives, three managers, four researchers, and  six engineers. We thank them for being generous with their time and insights. Among them are:

• Timo Ali-Vehmas, Nokia
• Jan-Erik Ekberg, Huawei
• Janne Hirvimies, Darkmatter
• Antti Jauhiainen, ZoomIN Oy
• Antti Kiiveri, Boogie Software Oy
• Markku Kylänpää, VTT
• Meri Löfman, Brighthouse Intelligence Oy
• Janne Mäntylä, Huawei
• Yrjö Neuvo, Aalto University
• Valtteri Niemi, University of Helsinki
• Lauri Paatero, F-Secure
• Jukka Parkkinen, OP Financial Group
• Janne Takala, Profit Software Oy.
• Janne Uusilehto, Google

Protecting against run-time attacks with Pointer Authentication

Since the Morris Worm of 1988, buffer overflows and similar have been the source of many remote-code-execution vulnerabilities. They can allow attackers to overwrite pointers in memory and make a vulnerable program jump to unexpected locations.  The ARMv8.3-A architecture includes Pointer Authentication, a set of instructions that can be used to cryptographically authenticate pointers and data before they are used.  We show several ways that Pointer Authentication can be used to improve security, and prevent attackers from turning programmer errors into remote-code-execution vulnerabilities.

Pointer Authentication: the What, the How and the Why

The fundamental purpose of Pointer Authentication (PA) is to allow software to verify that values read from memory—whether data or pointers—were generated by the same process in the right context. It does this by allowing software to generate a pointer authentication code (PAC), a tweakable MAC that can be squeezed into the unused high-order bits of a pointer, and whose key is stored in a register accessible only by software running at a higher privilege level, such as the operating system kernel.

PA provides three main types of instructions:

Type	Examples	Purpose
Generate an authenticated pointer	pacia, pacibsp	Generate a short PAC over a pointer and store the PAC into the high-order bits of the pointer.
Verify an authenticated pointer	autia, autibsp	If the pointer contains a valid PAC for its address, turn it back into a "normal" pointer; otherwise, make the pointer invalid so that the program will crash if the pointer is used.
Generate a "generic" PAC	pacga	Generate a 32-bit PAC over the contents of a whole register.

These instructions combine three values:

1. The value to be authenticated. For all the instructions except pacga, the PAC is computed over the low-order bits that contain the actual pointer data (the high-order bits being reserved for the PAC, a "sign" bit used to determine whether the reserved bits of a verified pointer are set to all zeros or all ones, and an optional address tag).
2. A modifier value. This is used to determine the "context" of a pointer so that an authenticated pointer can't be taken from one place and reused in another (more on this later). There are some special-case instructions that are hard-coded to use e.g. the stack pointer or zero as the modifier. The modifier is used as the "tweak" for the tweakable MAC computation.
3. A key. There are five of these, and which one is used depends on the choice of instruction. This is stored in a register that (on Linux) cannot be accessed from user-space and is set to a random value when the process is started so that authenticated pointers aren't interchangeable between processes.

We are primarily interested in the first two types of instructions, which store and verify PACs in the high-order bits of a pointer, as illustrated below. The actual number of bits depends on the configured virtual address size (39 bits on Linux by default) and whether an address tag is present. By default, these PACs are 16 bits long on Linux.

A PAC instruction is used to generate a PAC over an address and store it in high-order bits of the pointer. The instruction takes an address and modifier as operands, with the PAC key being stored in a separate system register that on Linux is configured to be accessible only by the kernel. There are several families of PAC instructions, each of which uses a different key: PACIA and PACIB for code pointers, and PACDA and PACDB for data pointers.

By using these instructions to verify the authenticity of pointers before they are used, we can prevent an attacker from e.g. overwriting return addresses on the stack using a buffer overflow, or overwriting other program values as part of a data-oriented programming attack. When the program returns, it verifies the PAC bits of the return address, causing the program to crash if the return address has been changed. This has been implemented in both GCC and LLVM as the -msign-return-address option.

Much of the difference in security of PA-based protection schemes comes from the choice of modifier. A modifier should be outside the attacker's control, as well as quick and easy to compute from the available data, but if modifiers coincide too often, then this gives an attacker too many opportunities to reuse pointers outside the context that they are meant to be used in.

PARTS: Protecting data pointers with PA

Modern operating systems have many protections against buffer-overflow-type attacks—e.g. W^X and ASLR. W^X prevents an attacker from injecting their own code, and is defeated by return-oriented programming, in which return addresses are overwritten to make the program return to a series of "gadgets", small pieces of code already present in the program that can be assembled into the attacker's desired functionality. This has encouraged the use of control-flow integrity mechanisms such as shadow stacks, but even perfect control flow integrity is not enough. Data-oriented programming attacks piggy-back on the program's correct control flow, performing arbitrary computation by manipulating the program's data.

We have introduced a PA-based scheme, PARTS (Usenix SEC 2019), which protects against data-oriented programming attacks that depend on pointer manipulation, as well as many control-flow attacks. PARTS prevents a pointer that ostensibly points to one type from being dereferenced into an object of a different type. Since the compiler knows all of the types at compile time, it can select modifiers statically that will be used to generate and verify PACs when an address of an object is put and taken from memory, respectively. This can be used to protect against the misuse of both data pointers as well as function pointers. Verifying the PAC of a function pointer before the pointer is used in an indirect call provides protection for the program's forward control-flow as well.
To protect the backward control-flow (i.e. to prevent the program from jumping to return addresses overwritten by the attacker), the return addresses are also authenticated; we discuss this in the following sections.

PA-based return address protection

PA is not only useful for this kind of "static" protection, where modifiers can be chosen at compile time. Dynamically-selected modifiers can be particularly powerful.

One of the first uses of Pointer Authentication was to protect return addresses on the stack, since overwriting a return address makes the program jump to a memory address of the attacker's choice. Including a PAC in the return address will make the jump fail, unless the authenticated return address was previously generated by the program. But this has a problem: if an attacker can use a memory vulnerability to read from the program's memory, then they can obtain authenticated return addresses that will validate correctly, and overwrite the return address with one of these. This is where the modifier comes into play: if the modifier depends on the path that the program has taken through its call-graph, then the authenticated return pointers from different paths cannot be swapped.
One way (and the first proposed use of ARM PA) to make this modifier path-dependent is to use the stack pointer as the modifier. Each time a function is called, the stack pointer is reduced in order to make space for stack variables, saved registers, and the return address. Since the value subtracted from the stack pointer depends on the function that has been called, this results in a modifier that depends on the stack layout of a particular path through the program. This approach is illustrated below.

PA-based return-address protection. At the beginning of each function, a PAC is generated for the current return address, which can then be saved on the stack. At the end of the function, the PAC is verified to ensure that the address has not been tampered with.

However, the resulting modifier can be predicted from static analysis of the program, so an attacker can find distinct paths through the program that lead to identical stack pointers, allowing their corresponding return addresses to be exchanged.

PACStack: an authenticated call stack

To overcome this problem, we have developed an alternative technique called PACStack (poster at DAC 2019, full report on arXiv), which uses chained PACs to bind the current return address to the entire path taken through the call graph.

Apart from PA, the key feature of the ARM architecture that makes PACStack possible is that after a call and before a return, the return address of the current function is stored in a register, called the Link Register (LR). By ensuring that the current return address is always stored in a register, we prevent an attacker exploiting a buffer overflow from ever overwriting the current (authenticated) return address, but only the next one, which will be loaded into LR when the function returns. By verifying the PAC of the return address kept in a register (and therefore known to be good) using the previous authenticated return address as a modifier implicitly verifies the authenticated return address being loaded from the stack.

Since the new value of LR also contains a PAC, authenticating the head of this chain of PACs recursively authenticates the entire call stack, providing perfect backward control-flow integrity.

The chain of PACs produced by PACStack. Each PAC is generated using the previous authenticated return address as modifier.

For the attacker, this cryptographic protection means that returning to a different address is now far more difficult than just overwriting a return address, as seen below.

Anatomy of a control-flow violation with PACStack in use. In the correct control flow (left), after a call from A to C, C returns back to A. The goal of the attacker is to return from C back to some other function B. To do this, they must replace C's return value by overwriting it on the stack when C calls some other function ("loader" in the diagram). This new value must pass two PAC-checks before the program will return to the new pointer.

A major issue with this type of scheme is that because the PACs are short—16 bits in this case—and the attacker can use their ability to overwrite variables in memory to influence the path through the call graph, the attacker can take advantage of the birthday paradox, guiding the program along many different paths through the call-graph and obtaining colliding PACs after around 320 attempts on average. This allows the attacker to call down through the program's call graph and return up a different path, as illustrated in the figure above.

Not content with this, we have developed a technique that we refer to as PAC masking, which prevents the attacker from exploiting PAC collisions. PACStack makes a second use of the PA instructions as a pseudo-random generator, which is used to generate a modifier-dependent masking value that is XOR-ed with the PAC. This prevents the attacker from recognizing when two PACs generated with different return values collide, forcing them to risk a guess. The result is that no matter how many authenticated return pointers the attacker is able to obtain, they cannot successfully change a function's return address with probability better than one in 65536, making return-to-libc, return-oriented programming, and similar types of attacks extremely unlikely to succeed.

Next steps

The specification for Pointer Authentication leaves it to the implementer to decide how the PAC is actually implemented. One possibility here is for an implementer to use something like our PAC masking primitive for all of the non-generic PACs, but it is not yet clear whether further security requirements will become apparent in the future.

Together, these examples show the great flexibility of Pointer Authentication in ARMv8.3-A. The cost of this flexibility is the need for thorough cryptographic analysis. However, our experience with PACStack shows that this is viable for practical systems. This flexibility is what makes Pointer Authentication especially exciting as a run-time security feature, enabling compiler-writers to integrate highly secure run-time protection mechanisms without waiting for hardware to catch up. As more of these enabling features—e.g. memory tagging and branch target indicators—are deployed in the coming years, new defenses will become possible, and run-time protection schemes will continue to become faster and more secure.

How to evade hate speech filters with "love"

It has often been suggested that text classification and machine learning techniques can be used to detect hate speech. Such tools could then be used for automatic content filtering, and perhaps even law enforcement. However, we show that existing datasets are too domain-specific, and the resulting models are easy to circumvent by applying simple automatic changes to the text.

 

The problem of hate speech and its detection

Hate speech is a major problem online, and a number of machine learning solutions have been suggested to combat it.

The vast majority of online material consists of natural language text. Along with its undeniable benefits, this is not without negative side-effects. Hate speech is rampant in discussion forums and blogs, and can have real world consequences. While individual websites can have specific filters to suit their own needs, the general task of hate speech detection involves many unanswered questions. For example, it is problematic where we should draw the line between hateful and merely offensive material.

Despite this, multiple studies have claimed success at detecting hate speech using state-of-the-art natural language processing (NLP) and machine learning (ML) methods. All these studies involve supervised learning, where a ML model is first trained with data labeled by humans, and then tested on new data which it has not seen during training. The more accurately the trained model manages to classify unseen data, the better it is considered.

While a number of labeled hate speech datasets exist, most of them are relatively small, containing just some thousands of hate speech examples. So far, the largest dataset has been drawn from Wikipedia edit comments, and contains around 13 000 hateful sentences. In the world of ML, these are still small numbers. Other datasets are typically taken from Twitter. They may also differ in the type of hate speech they focus on, such as racism, sexism, or personal attacks.

Google has also developed its own “toxic speech” detection system, called Perspective. While the training data and model architecture are unavailable to the public, Google provides black-box access via an online UI.

We wanted to compare existing solutions with one another, and test how resistant they are against possible attacks. Both issues are crucial for determining how well suggested models could actually fare in the real world.

Applying proposed classifiers to multiple datasets

Datasets are too small and specific for models to scale beyond their own training domain.

Most prior studies have not compared their approach with alternatives. In contrast, we wanted to test how all state-of-the-art models perform on all state-of-the-art datasets. We gathered up five datasets and five model architectures, seven combinations of which had been presented in resent academic research.

The model architectures differed in input features and the ML-algorithm. They either looked at characters and character sequences, or alternatively at entire words and/or word sequences. Some models used simple probabilistic ML algorithms (such as logistic regression or a multilayered perceptron network), while others used state-of-the-art deep neural networks (DNNs). More details can be found in our paper.

To begin with, we were interested in understanding how ML-algorithms differ in performance when trained and tested on the same type of data. This would give us some clue as to what kinds of properties might be most relevant for classifying something as hate speech. For example, if character frequencies were sufficient, simple character-based models would suffice. In contrast, if complex word-relations are needed, recurrent neural network DNN-models (like LSTMs or CNNs) would fare better.

We took all four two-class model architectures, and trained them on all four two-class datasets, yielding eight models in total. Next, we took the test sets from each dataset, and applied the models to those. The test sets were always distinct from the training set, but derived from the same dataset. We show the results in the two figures below (datasets used in the original studies are written in bold.)

Performance of ML-algorithms on different datasets (F1-score)

Performance of models with different test sets (F1-score)

Our results were surprising on two fronts. First, all models trained on the same dataset performed similarily. In particular, there was no major difference between using a simple probabilistic classifier (logistic regression: LR) that looked at character sequences, or using a complex deep neural network (DNN) that looked at long word sequences.

Second, no model performed well outside of its training domain: models performed well only on the test set that was taken from the same dataset type that they were trained on.

Our results indicate that training data was more important in determining performance than model architecture. We take the main relevance of this finding to be that focus should be on collecting and labeling better datasets, not only on refining the details of the learning algorithms. Without proper training data, even the most sophisticated algorithms can do very little.

Attacking the classifiers

Hate  speech classifiers can be fooled by simple automatic text transformation methods.

Hate speech can be considered an adversarial setting, with the speakers attacking the people their text targets. However, if automatic measures are used for filtering, these classifiers might also be attack targets. In particular, these could be circumvented by text transformation. We wanted to find out whether this is feasible in practice.

Text transformation involves changing words and/or characters in the text with the intent of altering some property while retaining everything else. For evading hate speech detection, this property is the classification the detector makes.

We experimented with three transformation types, and two variants of each. The types were:

1. word-internal changes:
   

   typos (I htae you) 

   leetspeak (1 h4te y0u)
2. word boundary changes:
   

   adding whitespace (I ha te you)

   deleting whitespace (Ihateyou)
3. word appending:
   

   random words (I hate you dog cat...)

   "non-hateful" words (I hate you good nice...)

Details of all our experiments are in our paper. Here, we summarize three main results.

• Character-based models were significantly more robust against word-internal and word boundary transformations.
• Deleting whitespace completely broke all word-based models, regardless of their complexity.
• Word appending systematically hindered the performance of all models. Further, while random words did not fare as well as “non-hateful” words from the training set, the difference was relatively minor. This indicates that the word appending attack is feasible even in a black-box setting.

The first two results are caused by a fundamental problem with all word-based NLP methods: the model must recognize the words based on which it does all further computation. If word recognition fails, so does everything else. Deleting all whitespaces makes the entire sentence look like a single unknown word, and the model can do nothing useful with that. Character-models, in contrast, retain the majority of original features, and thus take less damage.

Word appending attacks take advantage of the fact that all suggested approaches treat the detection task as classification. The classifier makes a probabilistic decision of whether the text is more dominantly hateful or non-hateful, and simply including irrelevant non-hateful material will bias this decision to the latter side. This is obviously not what we want from a hate speech classifier, as the status of hate speech should not be affected by such additions. Crucially, this problem will not go away simply by using more complex model architectures; it is built into the very nature of probabilistic classification. Avoiding it requires re-thinking the problem from a novel perspective, perhaps by re-conceptualizing hate speech detection as anomaly detection rather than classification.

The "love" attack 

Deleting whitespaces and adding "love" broke all word-based classifiers.

Based on our experimental results, we devised an attack that uses two of our most effective transformation techniques: whitespace deletion and word appending. Here, instead of adding multiple innocuous words, we only add one: “love”. This attack completely broke all word-based models we tested, as well as severely hindered character-based models. However, character-models were much more resistant against it, for reasons we discussed above. (Take a look at our paper for more details on models and datasets.)

The "love" attack applied to seven hate speech classifiers

We also applied the “love” attack to Google Perspective, with comparable results on all example sentences. This indicates that the model is word-based. To improve readability, the attacker can use alternative methods of indicating word boundaries, such as CamelFont.

Example of the "love" attack applied to Google Perspective

 

Conclusions: can the deficiencies be remedied?

Our study demonstrates that proposed solutions for hate speech detection are unsatisfactory in two ways. Here, we consider some possibilities for alleviating the situation with respect to both.

First, classifiers are too specific for particular text domains and hence do not scale across different datasets. The main reason behind this problem is the lack of sufficient training data. In contrast, model architecture had little to no effect on classification success. Alleviate this problem requires focusing further resources on the hard work of manually collecting and labeling more data.

Second, existing classifiers are vulnerable to text transformation attacks that can easily be applied automatically in a black box setting. Some of these attacks can be partially mitigated by pre-processing measures, such as automatic spell-checking. Others, however, are more difficult to detect. This is particularly true of word appending, as it is very hard to evaluate whether some word is "relevant" or simply added to the text for deceptive purposes. Like we mentioned above, avoiding the word appending attack ultimately requires re-thinking the detection process as something other than probabilistic classification.

Character-models have a far superior resistance to word boundary changes than word-models. This is as expected, since most character sequences are still retained even if word identities are destroyed. As character-models performed equally well to word-models in our comparative tests, we recommend using them to make hate speech detection more resistant to attacks.