Resource Development

• Resource Development

  • The adversary is trying to establish resources they can use to support operations.
  • Resource Development consists of techniques that involve adversaries creating, purchasing, or compromising/stealing resources that can be used to support targeting. Such resources include machine learning artifacts, infrastructure, accounts, or capabilities. These resources can be leveraged by the adversary to aid in other phases of the adversary lifecycle, such as ML Attack Staging.
• Acquire Public ML Artifacts

  • Adversaries may search public sources, including cloud storage, public-facing services, and software or data repositories, to identify machine learning artifacts. These machine learning artifacts may include the software stack used to train and deploy models, training and testing data, model configurations and parameters. An adversary will be particularly interested in artifacts hosted by or associated with the victim organization as they may represent what that organization uses in a production environment.
• Acquire Public ML Artifacts

  • Adversaries may identify artifact repositories via other resources associated with the victim organization (e.g. Search Victim-Owned Websites or Search for Victim's Publicly Available Research Materials). These ML artifacts often provide adversaries with details of the ML task and approach.
  • Artifacts might be hosted on victim-controlled infrastructure, providing the victim with some information on who has accessed that data.
• Acquire Public ML Artifacts

  • ML artifacts can aid in an adversary's ability to Create Proxy ML Model. If these artifacts include pieces of the actual model in production, they can be used to directly Craft Adversarial Data. Acquiring some artifacts requires registration (providing user details such email/name), AWS keys, or written requests, and may require the adversary to Establish Accounts.
• Acquire Public ML Artifacts: Datasets

  • Adversaries may collect public datasets to use in their operations. Datasets used by the victim organization or datasets that are representative of the data used by the victim organization may be valuable to adversaries. Datasets can be stored in cloud storage, or on victim-owned websites. Some datasets require the adversary to Establish Accounts for access.
  • Acquired datasets help the adversary advance their operations, stage attacks, and tailor attacks to the victim organization.
• Acquire Public ML Artifacts: Models

  • Adversaries may acquire public models to use in their operations. Adversaries may seek models used by the victim organization or models that are representative of those used by the victim organization. Representative models may include model architectures, or pre-trained models which define the architecture as well as model parameters from training on a dataset.
• Acquire Public ML Artifacts: Models

  • The adversary may search public sources for common model architecture configuration file formats such as YAML or Python configuration files, and common model storage file formats such as ONNX (.onnx), HDF5 (.h5), Pickle (.pkl), PyTorch (.pth), or TensorFlow (.pb, .tflite).
  • Acquired models are useful in advancing the adversary's operations and are frequently used to tailor attacks to the victim model.
• Obtain Capabilities

  • Adversaries may search for and obtain software capabilities for use in their operations. Capabilities may be specific to ML-based attacks Adversarial ML Attack Implementations or generic software tools repurposed for malicious intent (Software Tools). In both instances, an adversary may modify or customize the capability to aid in targeting a particular ML system.
• Obtain Capabilities: Adversarial ML Attack Implementations

  • Adversaries may search for existing open source implementations of machine learning attacks. The research community often publishes their code for reproducibility and to further future research. Libraries intended for research purposes, such as CleverHans, the Adversarial Robustness Toolbox, and FoolBox, can be weaponized by an adversary. Adversaries may also obtain and use tools that were not originally designed for adversarial ML attacks as part of their attack.
• Obtain Capabilities: Software Tools

  • Adversaries may search for and obtain software tools to support their operations. Software designed for legitimate use may be repurposed by an adversary for malicious intent. An adversary may modify or customize software tools to achieve their purpose. Software tools used to support attacks on ML systems are not necessarily ML-based themselves.
• Develop Capabilities

  • Adversaries may develop their own capabilities to support operations. This process encompasses identifying requirements, building solutions, and deploying capabilities. Capabilities used to support attacks on ML systems are not necessarily ML-based themselves. Examples include setting up websites with adversarial information or creating Jupyter notebooks with obfuscated exfiltration code.
• Develop Capabilities: Adversarial ML Attacks

  • Adversaries may develop their own adversarial attacks. They may leverage existing libraries as a starting point (Adversarial ML Attack Implementations). They may implement ideas described in public research papers or develop custom made attacks for the victim model.
• Acquire Infrastructure

  • Adversaries may buy, lease, or rent infrastructure for use throughout their operation. A wide variety of infrastructure exists for hosting and orchestrating adversary operations. Infrastructure solutions include physical or cloud servers, domains, mobile devices, and third-party web services. Free resources may also be used, but they are typically limited.
• Acquire Infrastructure

  • Use of these infrastructure solutions allows an adversary to stage, launch, and execute an operation. Solutions may help adversary operations blend in with traffic that is seen as normal, such as contact to third-party web services. Depending on the implementation, adversaries may use infrastructure that makes it difficult to physically tie back to them as well as utilize infrastructure that can be rapidly provisioned, modified, and shut down.
• Acquire Infrastructure: ML Development Workspaces

  • Developing and staging machine learning attacks often requires expensive compute resources. Adversaries may need access to one or many GPUs in order to develop an attack. They may try to anonymously use free resources such as Google Colaboratory, or cloud resources such as AWS, Azure, or Google Cloud as an efficient way to stand up temporary resources to conduct operations. Multiple workspaces may be used to avoid detection.
• Acquire Infrastructure: Consumer Hardware

  • Adversaries may acquire consumer hardware to conduct their attacks. Owning the hardware provides the adversary with complete control of the environment. These devices can be hard to trace.
• Publish Poisoned Datasets

  • Adversaries may Poison Training Data and publish it to a public location. The poisoned dataset may be a novel dataset or a poisoned variant of an existing open source dataset. This data may be introduced to a victim system via ML Supply Chain Compromise.
• Poison Training Data

  • Adversaries may attempt to poison datasets used by a ML model by modifying the underlying data or its labels. This allows the adversary to embed vulnerabilities in ML models trained on the data that may not be easily detectable. Data poisoning attacks may or may not require modifying the labels. The embedded vulnerability is activated at a later time by data samples with an Insert Backdoor Trigger
  • Poisoned data can be introduced via ML Supply Chain Compromise or the data may be poisoned after the adversary gains Initial Access to the system.
• Establish Accounts

  • Adversaries may create accounts with various services for use in targeting, to gain access to resources needed in ML Attack Staging, or for victim impersonation.