How to protect your AI systems against adversarial machine learning
Applying ATT&CK to machine learning
The Adversarial ML Threat Matrix is presented in the style of ATT&CK, a tried-and-tested framework developed by MITRE to deal with cyber-threats in enterprise networks. ATT&CK provides a table that summarizes different adversarial tactics and the types of techniques that threat actors perform in each area.
Since its inception, ATT&CK has become a popular guide for cybersecurity experts and threat analysts to find weaknesses and speculate on possible attacks. The ATT&CK format of the Adversarial ML Threat Matrix makes it easier for security analysts to understand the threats of machine learning systems. It is also an accessible document for machine learning engineers who might not be deeply acquainted with cybersecurity operations.
“Many industries are undergoing digital transformation and will likely adopt machine learning technology as part of service/product offerings, including making high-stakes decisions,” Pin-Yu Chen, AI researcher at IBM, toldTechTalksin written comments. “The notion of ‘system’ has evolved and become more complicated with the adoption of machine learning and deep learning.”
For instance, Chen says, an automated financial loan application recommendation can change from a transparentrule-based systemto a black-boxneural network-oriented system, which could have considerable implications on how the system can be attacked and secured.
“The adversarial threat matrix analysis (i.e., the study) bridges the gap by offering a holistic view of security in emerging ML-based systems, as well as illustrating their causes from traditional means and new risks induce by ML,” Chen says.
The Adversarial ML Threat Matrix combines known and documented tactics and techniques used in attacking digital infrastructure with methods that are unique tomachine learningsystems. Like the original ATT&CK table, each column represents one tactic (or area of activity) such as reconnaissance or model evasion, and each cell represents a specific technique.
For instance, to attack a machine learning system, a malicious actor must first gather information about the underlying model (reconnaissance column). This can be done through the gathering of open-source information (arXiv papers, GitHub repositories, press releases, etc.) or through experimentation with the application programming interface that exposes the model.
The complexity of machine learning security
Each new type of technology comes with its unique security and privacy implications. For instance, the advent of web applications with database backends introduced the concept SQL injection. Browser scripting languages such as JavaScript ushered in cross-site scripting attacks. Theinternet of things(IoT) introduced new ways to createbotnetsand conduct distributed denial of service (DDoS) attacks. Smartphones and mobile apps create new attack vectors for malicious actors and spying agencies.
The security landscape has evolved and continues to develop to address each of these threats. We have anti-malware software, web application firewalls, intrusion detection and prevention systems, DDoS protection solutions, and many more tools to fend off these threats.
For instance, security tools can scan binary executables for the digital fingerprints of malicious payloads, and static analysis can find vulnerabilities in software code. Many platforms such as GitHub and Google App Store already have integrated many of these tools and do a good job at finding security holes in the software they house.
But in adversarial attacks, malicious behavior and vulnerabilities are deeply embedded in the thousands and millions of parameters of deep neural networks, which is both hard to find and beyond the capabilities of current security tools.
“Traditional software security usually does not involve the machine learning component because it’s a new piece in the growing system,” Chen says, adding that adopting machine learning into the security landscape gives new insights and risk assessment.
The Adversarial ML Threat Matrix comes with aset of case studiesof attacks that involve traditional security vulnerabilities, adversarial machine learning, and combinations of both. What’s important is that contrary to the popular belief that adversarial attacks are limited to lab environments, the case studies show that production machine learning system can and have been compromised with adversarial attacks.
For instance, in one case study, the security team at Microsoft Azure used open-source data to gather information about a target machine learning model. They then used a valid account in the server to obtain the machine learning model and its training data. They used this information to find adversarial vulnerabilities in the model and develop attacks against the API that exposed its functionality to the public.
Other case studies show how attackers can compromise various aspect of the machine learning pipeline and the software stack to conductdata poisoning attacks, bypass spam detectors, or force AI systems to reveal confidential information.
The matrix and these case studies can guide analysts in finding weak spots in their software and can guide security tool vendors in creating new tools to protect machine learning systems.
“Inspecting a single dimension (machine learning vs traditional software security) only provides an incomplete security analysis of the system as a whole,” Chen says. “Like the old saying goes: security is only as strong as its weakest link.”
Machine learning developers need to pay attention to adversarial threats
Unfortunately, developers and adopters of machine learning algorithms are not taking the necessary measures to make their models robust against adversarial attacks.
“The current development pipeline is merely ensuring a model trained on a training set can generalize well to a test set, while neglecting the fact that the model is often overconfident about the unseen (out-of-distribution) data or maliciously embbed Trojan patten in the training set, which offers unintended avenues to evasion attacks andbackdoor attacksthat an adversary can leverage to control or misguide the deployed model,” Chen says. “In my view, similar to car model development and manufacturing, a comprehensive ‘in-house collision test’ for different adversarial treats on an AI model should be the new norm to practice to better understand and mitigate potential security risks.”
In his work at IBM Research, Chen has helped developvariousmethodstodetect andpatch adversarial vulnerabilities in machine learning models. With the advent Adversarial ML Threat Matrix, the efforts of Chen and other AI and security researchers will put developers in a better position to create secure and robust machine learning systems.
“My hope is that with this study, the model developers and machine learning researchers can pay more attention to the security (robustness) aspect of the model and looking beyond a single performance metric such as accuracy,” Chen says.
This article was originally published by Ben Dickson onTechTalks, a publication that examines trends in technology, how they affect the way we live and do business, and the problems they solve. But we also discuss the evil side of technology, the darker implications of new tech and what we need to look out for. You can read the original articlehere.
Story byBen Dickson
Ben Dickson is the founder of TechTalks. He writes regularly about business, technology and politics. Follow him on Twitter and Facebook(show all)Ben Dickson is the founder ofTechTalks. He writes regularly about business, technology and politics. Follow him onTwitterandFacebook
Get the TNW newsletter
Get the most important tech news in your inbox each week.
