Initial Access

• Initial Access

  • The adversary is trying to gain access to the machine learning system.
  • The target system could be a network, mobile device, or an edge device such as a sensor platform. The machine learning capabilities used by the system could be local with onboard or cloud-enabled ML capabilities.
  • Initial Access consists of techniques that use various entry vectors to gain their initial foothold within the system.
• ML Supply Chain Compromise

  • Adversaries may gain initial access to a system by compromising the unique portions of the ML supply chain. This could include GPU Hardware, Data and its annotations, parts of the ML ML Software stack, or the Model itself. In some instances the attacker will need secondary access to fully carry out an attack using compromised components of the supply chain.
• ML Supply Chain Compromise: GPU Hardware

  • Most machine learning systems require access to certain specialized hardware, typically GPUs. Adversaries can target machine learning systems by specifically targeting the GPU supply chain.
• ML Supply Chain Compromise: ML Software

  • Most machine learning systems rely on a limited set of machine learning frameworks. An adversary could get access to a large number of machine learning systems through a comprise of one of their supply chains. Many machine learning projects also rely on other open source implementations of various algorithms. These can also be compromised in a targeted way to get access to specific systems.
• ML Supply Chain Compromise: Data

  • Data is a key vector of supply chain compromise for adversaries. Every machine learning project will require some form of data. Many rely on large open source datasets that are publicly available. An adversary could rely on compromising these sources of data. The malicious data could be a result of Poison Training Data or include traditional malware.
• ML Supply Chain Compromise: Data

  • An adversary can also target private datasets in the labeling phase. The creation of private datasets will often require the hiring of outside labeling services. An adversary can poison a dataset by modifying the labels being generated by the labeling service.
• ML Supply Chain Compromise: Model

  • Machine learning systems often rely on open sourced models in various ways. Most commonly, the victim organization may be using these models for fine tuning. These models will be downloaded from an external source and then used as the base for the model as it is tuned on a smaller, private dataset. Loading models often requires executing some saved code in the form of a saved model file. These can be compromised with traditional malware, or through some adversarial machine learning techniques.
• Valid Accounts

  • Adversaries may obtain and abuse credentials of existing accounts as a means of gaining Initial Access. Credentials may take the form of usernames and passwords of individual user accounts or API keys that provide access to various ML resources and services.
  • Compromised credentials may provide access to additional ML artifacts and allow the adversary to perform Discover ML Artifacts. Compromised credentials may also grant and adversary increased privileges such as write access to ML artifacts used during development or production.
• Evade ML Model

  • Adversaries can Craft Adversarial Data that prevent a machine learning model from correctly identifying the contents of the data. This technique can be used to evade a downstream task where machine learning is utilized. The adversary may evade machine learning based virus/malware detection, or network scanning towards the goal of a traditional cyber attack.
• Exploit Public-Facing Application

  • Adversaries may attempt to take advantage of a weakness in an Internet-facing computer or program using software, data, or commands in order to cause unintended or unanticipated behavior. The weakness in the system can be a bug, a glitch, or a design vulnerability.
• Exploit Public-Facing Application

  • These applications are often websites, but can include databases (like SQL), standard services (like SMB or SSH), network device administration and management protocols (like SNMP and Smart Install), and any other applications with Internet accessible open sockets, such as web servers and related services.
• LLM Prompt Injection

  • An adversary may craft malicious prompts as inputs to an LLM that cause the LLM to act in unintended ways. These "prompt injections" are often designed to cause the model to ignore aspects of its original instructions and follow the adversary's instructions instead.
• LLM Prompt Injection

  • Prompt Injections can be an initial access vector to the LLM that provides the adversary with a foothold to carry out other steps in their operation. They may be designed to bypass defenses in the LLM, or allow the adversary to issue privileged commands. The effects of a prompt injection can persist throughout an interactive session with an LLM.
• LLM Prompt Injection

  • Malicious prompts may be injected directly by the adversary (Direct) either to leverage the LLM to generate harmful content or to gain a foothold on the system and lead to further effects. Prompts may also be injected indirectly when as part of its normal operation the LLM ingests the malicious prompt from another data source (Indirect). This type of injection can be used by the adversary to a foothold on the system or to target the user of the LLM.
• LLM Prompt Injection: Direct

  • An adversary may inject prompts directly as a user of the LLM. This type of injection may be used by the adversary to gain a foothold in the system or to misuse the LLM itself, as for example to generate harmful content.
• LLM Prompt Injection: Indirect

  • An adversary may inject prompts indirectly via separate data channel ingested by the LLM such as include text or multimedia pulled from databases or websites. These malicious prompts may be hidden or obfuscated from the user. This type of injection may be used by the adversary to gain a foothold in the system or to target an unwitting user of the system.
• Phishing

  • Adversaries may send phishing messages to gain access to victim systems. All forms of phishing are electronically delivered social engineering. Phishing can be targeted, known as spearphishing. In spearphishing, a specific individual, company, or industry will be targeted by the adversary. More generally, adversaries can conduct non-targeted phishing, such as in mass malware spam campaigns.
• Phishing

  • Generative AI, including LLMs that generate synthetic text, visual deepfakes of faces, and audio deepfakes of speech, is enabling adversaries to scale targeted phishing campaigns. LLMs can interact with users via text conversations and can be programmed with a meta prompt to phish for sensitive information. Deepfakes can be use in impersonation as an aid to phishing.
• Phishing: Spearphishing via Social Engineering LLM

  • Adversaries may turn LLMs into targeted social engineers. LLMs are capable of interacting with users via text conversations. They can be instructed by an adversary to seek sensitive information from a user and act as effective social engineers. They can be targeted towards particular personas defined by the adversary. This allows adversaries to scale spearphishing efforts and target individuals to reveal private information such as credentials to privileged systems.