Industrial Practitioners' Mental Models of Adversarial Machine Learning: A Qualitative Study

1. Introduction & Overview

Adversarial Machine Learning (AML) is a critical subfield focused on the security and reliability of learning-based systems under adversarial conditions. While academic research has produced sophisticated attacks (e.g., evasion, poisoning, backdooring) and defenses, there is a significant gap in understanding how these threats are perceived and managed by practitioners who deploy ML in real-world, industrial settings. This study, presented at USENIX SOUPS 2022, pioneers an exploration into the mental models of these practitioners. Mental models are internal representations of how a system works; in security, accurate models are crucial for effective risk assessment and mitigation. The research reveals a fundamental disconnect: practitioners often conflate ML-specific security issues with general cybersecurity concerns and view security through the lens of entire integrated workflows, not just isolated models—a perspective largely absent from mainstream AML literature.

2. Methodology & Study Design

The study employed a qualitative, interview-based methodology to gain deep, contextual insights that quantitative surveys might miss.

2.1. Participant Selection & Demographics

The researchers conducted 15 semi-structured interviews with ML practitioners from European startups. Participants held roles such as ML engineers, data scientists, and developers, ensuring a sample with hands-on experience in building and deploying ML systems. The focus on startups is strategic, as they often represent the cutting edge of applied ML but may lack mature security protocols.

2.2. Data Collection & Analysis

Each interview included a drawing task, where participants were asked to sketch their perception of the ML pipeline and indicate where vulnerabilities might exist. This visual methodology helps externalize internal mental models. Interview transcripts and drawings were then analyzed using qualitative coding techniques to identify recurring themes, patterns, and conceptual gaps.

3. Core Findings: Two Facets of Mental Models

The analysis crystallized two primary facets that characterize practitioners' understanding of ML security.

3.1. Facet 1: Blurred Lines Between AML and Non-AML Security

Practitioners frequently did not distinguish between attacks targeting the statistical properties of an ML model (core AML) and general system security threats. For example, a discussion about adversarial evasion attacks might segue into concerns about API authentication or cryptographic key management. This conflation suggests that for practitioners, "ML system security" is a monolithic challenge, not a layered one with distinct attack surfaces. This blurring can lead to misallocated defense resources, where classic IT security measures are over-prioritized for AML problems, and vice-versa.

3.2. Facet 2: Holistic Pipeline View vs. Isolated Model Focus

Academic AML research often focuses on attacking or defending a single, trained model (e.g., crafting adversarial examples for an image classifier). In stark contrast, practitioners described security in the context of entire ML pipelines—from data collection and labeling, through multiple training and validation stages, to deployment, monitoring, and feedback loops. Their mental models included multiple interconnected components (databases, preprocessing code, serving infrastructure), each seen as a potential vulnerability point. This holistic view is more realistic but also more complex, making it harder to apply focused academic defenses.

4. Key Insights & Implications

• Communication Gap: There is a clear terminology and conceptual gap between AML researchers and practitioners. Research papers often fail to contextualize attacks within end-to-end workflows.
• Uncertainty & Risk: Practitioners reported significant uncertainty about how to prioritize and address ML security risks, partly due to the blurred mental models identified.
• Regulatory & Standardization Need: The findings underscore the need for security frameworks and standards (like those from NIST or MITRE's ATLAS) that address the entire ML pipeline, not just model robustness.
• Tooling Deficiency: A lack of practical, pipeline-integrated security tools exacerbates the problem. Most AML tools (e.g., CleverHans, Adversarial Robustness Toolbox) are designed for researchers, not DevOps pipelines.

5. Technical Framework & Attack Taxonomy

To ground the discussion, it's essential to understand the technical landscape of AML that practitioners are (often imperfectly) grappling with.

5.1. Mathematical Formulation of Threats

A canonical evasion attack can be formulated as an optimization problem. For a classifier $f(x)$ and original input $x$ with true label $y$, an adversary seeks a perturbation $\delta$ such that:

$\min_{\delta} \|\delta\|_p \quad \text{subject to} \quad f(x + \delta) \neq y$

where $\|\cdot\|_p$ is a $p$-norm (e.g., $L_2$, $L_\infty$) constraining perturbation perceptibility. This formal, model-centric view is typical in papers like Goodfellow et al.'s "Explaining and Harnessing Adversarial Examples" (ICLR 2015), but it abstracts away the surrounding pipeline.

5.2. The ML Pipeline Attack Surface

The paper references a taxonomy (visualized in a figure) mapping attacks to pipeline stages, which is more aligned with the practitioners' holistic view:

• Data/Design Phase: Poisoning attacks, Backdooring.
• Training Phase: Adversarial initialization, Weight perturbations.
• Model Phase: Model stealing, Reverse engineering, Membership inference.
• Deployment Phase: Evasion attacks, Adversarial reprogramming, Sponge attacks.

This framework explicitly shows that threats exist at every stage, validating the practitioners' broader concerns.

6. Analysis Framework & Case Study

Scenario: A fintech startup deploys a credit scoring model. Practitioners might worry about:
1. Data Poisoning (AML): An attacker subtly corrupts historical loan repayment data to bias the model.
2. API Security (Non-AML): An attacker exploits a vulnerability in the model-serving endpoint to gain unauthorized access.
3. Pipeline Integrity (Holistic View): A failure in the data validation step allows poisoned data into training, and a lack of model monitoring fails to detect the resulting drift in predictions.

Analysis: A practitioner with a blurred mental model might treat (1) and (2) with similar network security tools. A practitioner with a holistic view would implement controls across the pipeline: data provenance checks, adversarial training, robust serving APIs, and continuous output monitoring. The study suggests most practitioners are intuitively leaning toward the holistic view but lack the structured framework to implement it systematically.

7. Future Directions & Application Outlook

• Integrated Security Platforms: The future lies in DevSecOps for ML (MLSecOps). Tools need to integrate vulnerability scanning for data, model hardening, and runtime attack detection directly into CI/CD pipelines (e.g., leveraging ideas from continuous security validation).
• Education & Training: Curricula for data scientists and ML engineers must expand to include threat modeling for ML systems, distinguishing AML from traditional security. Resources like Google's "Machine Learning Security" course are a step in this direction.
• Standardized Benchmarks & Audits: The community needs benchmarks that evaluate the security of entire ML systems, not just model accuracy under attack. This will drive tool development and enable third-party security audits for critical ML applications.
• Regulatory Evolution: As seen with the EU AI Act, regulations will increasingly mandate risk management for "high-risk" AI systems. This study's findings highlight that such regulations must be based on a pipeline-centric, not model-centric, view of risk.

8. References

1. Biggio, B., & Roli, F. (2018). Wild patterns: Ten years after the rise of adversarial machine learning. Pattern Recognition.
2. Goodfellow, I. J., Shlens, J., & Szegedy, C. (2015). Explaining and harnessing adversarial examples. International Conference on Learning Representations (ICLR).
3. Papernot, N., McDaniel, P., Sinha, A., & Wellman, M. P. (2016). Towards the science of security and privacy in machine learning. arXiv preprint arXiv:1611.03814.
4. MITRE ATLAS (Adversarial Threat Landscape for Artificial-Intelligence Systems). https://atlas.mitre.org/.
5. NIST AI Risk Management Framework (AI RMF). https://www.nist.gov/itl/ai-risk-management-framework.
6. Carlini, N., & Wagner, D. (2017). Towards evaluating the robustness of neural networks. IEEE Symposium on Security and Privacy (S&P).

9. Original Analysis & Expert Commentary

Core Insight: This paper delivers a crucial, and frankly overdue, reality check to the AML research community. It exposes a dangerous "ivory tower" syndrome: while academics duel over marginal improvements in adversarial robustness on CIFAR-10, the practitioners actually building the systems that affect loans, healthcare, and autonomous navigation are operating with mental models that are both broader and fuzzier than the pristine attack definitions in our papers. The core tension isn't just about technical efficacy; it's about conceptual alignment. The study's revelation that practitioners see "ML security" as an undifferentiated mass—lumping together cryptographic key leakage with gradient-based evasion attacks—is a damning indictment of our failure to communicate and contextualize our work. This isn't merely a knowledge gap; it's a framing failure. As the NIST AI Risk Management Framework emphasizes, managing risk requires a systemic view, a principle clearly reflected in the practitioners' holistic pipeline perspective but often absent in narrow model-focused AML literature.

Logical Flow: The research logic is sound and revealing. By using qualitative interviews and drawing exercises—methods proven in seminal HCI-security work like those by Dourish and Anderson—the authors bypass superficial survey responses to tap into deep-seated cognitive structures. The flow from data collection (interviews) to analysis (coding) to synthesis (two key facets) cleanly supports the conclusion that a disconnect exists. The link to implications for tooling, regulation, and education is logical and compelling. However, the study's focus on European startups, while valuable, limits generalizability. A follow-up with large, regulated enterprises (e.g., in finance or healthcare) would likely reveal even more pronounced process-oriented mental models and regulatory concerns.

Strengths & Flaws: The paper's primary strength is its foundational nature. It is the first to systematically probe this space, providing a vocabulary and framework for future work. The methodological choice is a strength, yielding rich data. A significant flaw, acknowledged by the authors, is the sample size and scope (n=15, startups only). This isn't a representative survey; it's an exploratory deep dive. Furthermore, while it diagnoses the problem of blurred mental models, it offers less on why they are blurred. Is it due to a lack of education, the inherent complexity of integrated systems, or the marketing of "AI security" solutions that bundle disparate threats? The paper also doesn't fully grapple with a critical irony: the practitioners' holistic view is more correct from a systems security standpoint (aligning with frameworks like MITRE ATLAS), yet the academic community's focused, model-centric research has driven most of the algorithmic advances. Bridging this gap is the real challenge.

Actionable Insights: For researchers, the mandate is clear: stop publishing attacks in a vacuum. Frame every new threat within a real-world pipeline diagram. Collaborate with software engineering and security teams. Develop benchmarks for end-to-end system security, not just model robustness. For industry leaders and tool builders, invest in integrated MLSecOps platforms. Don't just sell an "adversarial training" module; sell a pipeline scanner that identifies vulnerabilities from data ingestion to prediction logging. For practitioners and educators, use this study to advocate for and develop training that separates the threat landscape: explain how a membership inference attack exploits model overfitting (a statistical flaw) versus how a backdoor is inserted (a supply-chain/data integrity flaw). This conceptual clarity is the first step toward effective defense. Ultimately, the field must mature from publishing clever hacks against isolated models to engineering secure machine learning systems. This paper is the stark wake-up call that we are not there yet.