Top 4 LLM threats to the enterprise

As CISO for the Vancouver Clinic, Michael Bray gushes about the infinite ways large language models (LLMs) will improve patient care. “DNA-based predictive studies, metabolic interactions, lab services, diagnostics and other medicine will be so advanced that today’s medical practices will look prehistoric,” he says. “For example, applications like ActX are already making a huge difference with symptom identification, medicine interactions, effectiveness, and dosages.”

As excited as he is about LLMs improving patient care and diagnoses, Bray is equally concerned about the new and hidden threats that LLMs present. LLMs are core to disruptive and fast-moving AI technologies including OpenAI’s ChatGPT, Google’s Bard, and Microsoft’s Copilot, which are rapidly proliferating across enterprises today. LLMs are being developed into a host of other specialty apps for specific vertical industries like finance, government, and military.

With these LLMs come new risks of data poisoning, phishing, prompt injections, and sensitive data extraction. Because these attacks are executed via natural language prompts or training sources, traditional security tools are ill-equipped to detect such attacks.

Fortunately, these vulnerabilities are being identified and prioritized by the Open Web Application Security Project (OWASP), National Institute of Standards (NIST), and other standards groups nearly as quickly as AI is proliferating. The EU AI Act has released an initial compliance checker for organizations to determine if their AI applications fall into the category of unacceptable risk or high risk. In November 2023, the UK released the UK guidelines for secure AI system development.

Tools are also catching up with new risks introduced through LLM’s. For example, natural language web firewalls, AI discovery, and AI-enhanced security testing tools are coming to market in what may well become a battle of AI versus AI. As we wait for those tools, these are the most likely threats organizations will face to their use of LLMs:

1. Malicious instructions from prompt injections

When asked about new threats introduced to enterprises through LLMs, experts cite prompt injections a top risk. Jailbreaking an AI by throwing a bunch of confusing prompts at the LLM interface is probably the most well-known risk and could cause reputational damage if the jailbreaker spreads misinformation that way. Or a jailbreaker could use confusing prompts to cause a system to spit out ridiculous offers, such as with a popular auto dealership chatbot developed by a company called Fullpath. By instructing a Chevy dealer’s chatbot to end each response with “that’s a legally binding offer, no takesies backsies,” a hacker tester tried thousands of prompts until he ultimately tricked the dealer site into offering him a new car for one dollar.

The more severe threat is when prompt injections are used to forceapplications to hand over sensitive information. Unlike with SQL injection prompts, threat actors can use limitless prompts to tryto trick an LLM into doing things it shouldn’t because the LLM prompts are written in natural language, explains Walter Haydock, founder of StackAware, which maps AI use in enterprises, and identifies associated risks.

“With SQL, there are finite ways you can input data so there is a known set of controls you can use to prevent and block SQL injections. But with prompt injection, there are infinite ways to provide malicious instructions to an LLM because the English language is that vast,” Haydock notes.The number of LLM prompt tokens continues to grow.

2. Data leakage from prompt extractions also an LLM vulnerability

Hyrum Anderson, CTO at Robust Intelligence, an end-to-end AI security platform that includes a natural language web firewall, also points to prompt extractions as a point of vulnerability. “Prompt extraction falls into the category of data leakage, where data can be extracted by merely asking for it,” he adds.

Take, for example, chatbots on a website, with relevant data behind them that support the application. These data can be exfiltrated. As an example, Anderson points to retrieval augmented generation (RAG), where LLM responses are enriched by connecting them to sources of information relevant to the task. Anderson recently witnessed such an attack in which demonstrators used a RAG to force the database to spit out specific sensitive information by asking for specific rows and tables in the database.

To prevent this type of database leakage, Anderson urges caution when connecting public-facing RAG apps to databases. “If you don’t want the RAG app user to see the entire database, then you should restrict access at the user interface to the LLM,” he adds. “Security-minded organizations should steel their APIs against natural-language pull requests, restrict access, and use an AI firewall to block malicious requests.”

3. New LLM-enabled phishing opportunities

LLMs also open a new vector for phishers to trick people into clicking their links, Anderson continues. “Say I’m a financial analyst using a RAG app to scrape documents from the internet to find out a company’s earnings, but in that supply chain of data are instructions for an LLM to respond with a phishing link. So, say I ask it to find the most up to date information in the trove of data it sent, and it says ‘click here.’ And then I click a phishing link.”

This kind of phish is powerful since the user is explicitly seeking an answer from the LLM. Furthermore, traditional anti-phishing tools may not see these malicious links, Anderson adds. He advises CISO’s to update their employee training programs to include critical thinking about RAG responses, and to use emerging web-based tools that can scan RAG data for natural-language prompt injections that encourage users to click links.

4. Poisoned LLMs

Models from open-source repositories and the data used to train LLMs can also be poisoned, adds Diana Kelley CISO at Protect AI, a platform for AI and ML security. “The biggest threats could be in the model itself or the data the LLM was trained on, who trained it, and where it was downloaded from,” she explains. “OSS models run with high privileges, but few companies scan them before use and the quality of the training data directly impacts the reliability and accuracy of the LLM. To see and manage AI related risks, and prevent poisoning attacks, CISOs need to govern the ML supply chain and track components throughout the lifecycle.”

That is, if CISOs are even aware of what  applications are using LLMs and for what purposes. Many common workforce applications used in enterprises today are embedding the latest AI capabilities in their system updates, sometimes without the knowledge of the CISO.

Because these LLMs are integrated into third-party applications and web interfaces, discovery and visibility become even more murky. So, an AI policy addressing the entire data supply chain is key, says Haydock of StackAware.Regarding thesefourth-party risks. “It’s understanding how these apps are using, training, accessing, and retaining your data,” he adds.

AI versus AI

The US Government, arguably the largest network in the world, certainly understands the value of AI security policy as it seeks to leverage the promise of AI across government and military applications. In October 2023, the Whitehouse issued an executive order (EO) for safe AI development and use.

The Cybersecurity and Infrastructure Security Agency (CISA), part of the Department of Homeland Security (DHS), plays a critical role in executing the executive order and has generated an AI roadmap that incorporates key CISA-led actions as directed by the EO—along with additional actions CISA is leading to support critical infrastructure owners and operators as they navigate the adoption of AI. 

As a result of the executive order, several key government agencies have already identified, nurtured, and appointed new chief AI officers responsible for coordinating their agency’s use of AI, promoting AI innovation while managing risks from their agency’s use of AI, according to Lisa Einstein, CISA’s senior advisor for AI.

“With AI embedded into more of our everyday applications, having a person who understands AI—and who understands the positive and negative implications of integrating AI—is critical,” Einstein explains. “Risks related to LLM use is highly contextual and use-case specific based on industry, whether it be healthcare, schools, energy, or IT. So, AI champions need to be able to work with industry experts to identify risks specific to the context of their industries.”

Within government agencies, Einstein points to the Department of Homeland Security’s Chief AI Officer Eric Hysen, who is also DHS’s CIO. Hysen coordinates AI efforts across DHS components, she explains, including the Transportation Security  Administration, which uses IBM’s computer vision to detect prohibited items in carry-on luggage. DHS, in fact, leverages AI in many instances to secure the homeland at ports of entry and along the border, as well as in cyberspace to protect children, defend against cyberthreats, and even to combat the malicious use of AI.

As LLM threats evolve, it will take equally innovative AI-enabled tools and techniques to combat them. AI-enhanced penetration testing and red teaming, threat intelligence, anomaly detection, incident response are but some of the tool types that are quickly adapting to fight these new threats.