Artificial intelligence can help businesses through automation or by improving existing tasks, but like any technology it comes with risks if not managed well. For those businesses that decided to build their own AI or buy software that has AI embedded in it, assessing its risks is an important step to ensuring compliance and data security.
The explosion of generative AI adoption has magnified those and new emerging risks. Generative AI adoption is the top-ranked issue for legal, compliance, and privacy leaders for the next two years, according to a Gartner survey from December.
To counter the risks, organizations can thoroughly evaluate their exposure to AI, assess the risks, and set guiderails and mitigation strategies in place to deal with the most business-critical issues. The assessment strategies differ based on the kind of AI that’s involved, which generally falls into three categories: internal AI projects, third-party AI, and AI used maliciously by attackers.
How to assess internal AI risks
Whether by using existing risk and quality management frameworks or setting up an internal framework for how AI models can be deployed, companies need to know how AI is being used internally.
Risk and quality management frameworks
One company that’s been assessing and quantifying the performance of their AI models for a while now is ABBYY, an intelligent software automation company. ABBYY has been using risk management and quality management frameworks for many years, including the ISO risk management framework. In addition, NIST has recently come out with the first iteration of its risk management framework specifically for AI. This framework is currently voluntary, but a bill was introduced in January in the US House of Representatives — a similar bill was introduced in the Senate in November — to make it mandatory for federal agencies and vendors. In addition, NIST is now in the process of establishing a US Artificial Intelligence Safety Institute.
These AI frameworks are still very rudimentary, ABBYY AI ethics evangelist Andrew Pery tells CSO. “It’s still too early,” which is why ABBYY primarily relies on older risk management frameworks, which the company is now applying to its generative AI projects. The company’s core product extracts data and content from documents and the company has specific accuracy targets for its traditional AI models. “There are nuances with respect to risk management and quality assurance for generative AI,” he says. That includes model performance, data governance, and bias.
For some use cases, generative AI may not be suitable yet. “For example, in the financial services, the ability to provide accurate information with respect to the disposition of loans is extremely important,” Pery says. “For them, the use of generative AI technology is speculative and risky. The same goes for insurance claims processing, health care, employment and HR applications.”
ABBYY isn’t alone in experimenting with generative AI — and in taking a careful approach to evaluating the associated risks. According to Wiz survey of 150,000 public cloud accounts, 70% of cloud environments are now using AI services such as Azure AI — which includes OpenAI — as well as Amazon SageMaker and Google Cloud’s Vertex AI. The most popular AI platform is OpenAI, which is in 53% of cloud environments. This is closely followed by Hugging Face and several other open-source projects.
Know how AI is being use across the organization
These AI models pose several risks to enterprises, starting with compliance risks — some industries, including health care and finance, have strict guidelines about how AI models can be used. Since AI models also require data, both for training and for real-time analysis and embeddings, there are also privacy and data loss risks. Finally, AI models might be inaccurate, biased, prone to hallucinations, or change in unpredictable ways over time.
To get a handle on all this conduct a comprehensive survey of how AI is being used. That requires both a top-down and a bottom-up approach, says Priya Iragavarapu, VP of digital technology services at AArete, a management consulting firm. “Business leaders need to review processes using AI and come to sessions with the AI governance team and review them,” she says. And when leaders don’t know what AI is being used, that’s when the bottom-up approach comes in — tracking all the endpoints in an organization to find the systems and users accessing AI applications. “Most are cloud-based applications,” she says. “And when people are using them, IT can track every query to ChatGPT.”
It’s not a perfect solution, Iragavarapu adds. “There are still things we’ll miss, that fall through the cracks. But that’s where thorough training, guidance and education comes in, so that users and employees can check themselves before doing something stupid.”
Set up your own AI deployment framework
Companies should also set up a framework for how AI models can be deployed, based on a company’s compliance environment and tolerance for risk. For example, some use cases might require a human review of all AI outputs. There are several dimensions of risks that are relevant for generative AI, Iragavarapu says. “How many people is it affecting? How big is the impact?” AArete works with companies to help them measure these risks, with some preliminary generative AI risk benchmarks available for some industries.
This is an area that a lot of consulting companies are now focusing on. Ernst & Young, for example, is developing an AI confidence index. “Our confidence index is founded on five criteria — privacy and security, bias and fairness, reliability, transparency and explainability, and the last is accountability,” says Kapish Vanvaria, EY Americas risk markets leader. That’s one axis of the index. The other includes regulations and ethics. “Then you can have a heat map of the different processes youâre looking at and the functions in which theyâre deployed,” he says. “And you can go through each one and apply a weighted scoring method to it.” In the legal and compliance arena, these calculations are relatively straightforward, Vanvaria says. “Most regulations are founded on scoring and are aligned to those things.â
In other areas, there’s more discretion. For example, a company could evaluate maturity based on such factors as having an acceptable AI use policy. “You can apply numerical scores to this to measure baselines, and as it becomes a moving target, apply changes to it,” Vanvaria says.
How to assess third-party AI risk
Companies have long faced problems securing their data when using third-party services. Any vendor — email provider, marketing vendor, data processor — might have access to sensitive company data, and even pass that access along to its own vendors. Enterprises have had a hard enough time dealing with third-party risk management before GenAI. Now, with generative AI, security teams have to step up their vigilance.
Tools that previously were not considered security risks — photo editing programs or grammar checkers — might now have generative AI functionality and send text or images off to an AI for analysis. “We take a very good look at the legal contract terms,” says Greg Notch, CISO at Expel, a security operation provider. “We draw the line on using our data to train your model — you can use our data to generate results, but then you have to throw it away.”
It’s a challenge to stay on top of it since the vendors can add new AI services any time, Notch says. That requires being obsessive about staying on top of all the contracts and changes in functionalities and the terms of service. But having a good third-party risk management team in place can help mitigate these risks. If an existing provider decides to add AI components to its platform by using services from OpenAI, for example, that adds another level of risk to an organization. “That’s no different from the fourth party risk I had before, where they were using some marketing company or some analytics company. So, I need to extend my third-party risk management program to adapt to it — or opt out of that until I understand the risk,” says Notch.
One of the positive aspects of Europe’s General Data Protection Regulation (GDPR) is that vendors are required to disclose when they use subprocessors. If a vendor develops new AI functionality in-house, one indication can be a change in their privacy policy. “You have to be on top of it. I’m fortunate to be working at a place that’s very security-forward and we have an excellent governance, risk and compliance team that does this kind of work,â Notch says.
Assessing external AI threats
Generative AI is already used to create phishing emails and business email compromise (BEC) attacks, and the level of sophistication of BEC has gone up significantly, according to Expel’s Notch. “If you’re defending against BEC — and everybody is — the cues that this is not a kosher email are becoming much harder to detect, both for humans and machines. You can have AI generate a pitch-perfect email forgery and website forgery.”
Putting a specific number to this risk is a challenge. “That’s the canonical question of cybersecurity — the risk quantification in dollars,” Notch says. “It’s about the size of the loss, how likely it is to happen and how often it’s going to happen.” But there’s another approach. “If I think about it in terms of prioritization and risk mitigation, I can give you answers with higher fidelity,” he says.
Pery says that ABBYY is working with cybersecurity providers who are focusing on GenAI-based threats. âThere are brand-new vectors of attack with genAI technology that we have to be cognizant about.â
These risks are also difficult to quantify, but there are new frameworks emerging that can help. For example, in 2023, cybersecurity expert Daniel Miessler released The AI Attack Surface Map. âSome great work is being done by a handful of thought-leaders and luminaries in AI,â says Sasa Zdjelar, chief trust officer at ReversingLabs, who adds that he expects organizations like CISA, NIST, the Cloud Security Alliance, ENISA, and others to form special task forces and groups to specifically tackle these new threats.
Meanwhile, what companies can do now is assess how well they do on the basics if they arenât doing this already. Including checking that all endpoints are protected, if users have multi-factor authentication enabled, how well can employees spot phishing email, how much of a backlog of patches is there, and how much of the environment is covered by zero trust. This kind of basic hygiene is easy to overlook when new threats are popping up, but many companies still fall short on the fundamentals. Closing these gaps will be more important than ever as attackers step up their activities.
There are a few things that companies can do to assess new and emerging threats, as well. According to Sean Loveland, COO of Resecurity, there are threat models that can be used to evaluate the new risks associated with AI, including offensive cyber threat intelligence and AI-specific threat monitoring. âThis will provide you with information on their new attack methods, detections, vulnerabilities, and how they are monetizing their activities,â Loveland says. For example, he says, there is a product called FraudGPT that is constantly updated and is being sold on the dark web and Telegram. To prepare for attackers using AI, Loveland suggests that enterprises review and adapt their security protocols and update their incident response plans.
Hackers use AI to predict defense mechanisms
Hackers have figured out how to use AI to observe and predict what defenders are doing, says Gregor Stewart, vice president of artificial intelligence at SentinelOne, and how to adjust on the fly. âAnd weâre seeing a proliferation of adaptive malware, polymorphic malware and autonomous malware propagation,â he adds.
Generative AI can also increase the volumes of attacks. According to a report released by threat intelligence firm SlashNext, there’s been a 1,265% increase in malicious phishing emails between the end of 2022 to the third quarter of 2023. “Some of the most common users of large language model chatbots are cybercriminals leveraging the tool to help write business email compromise attacks and systematically launch highly targeted phishing attacks,” the report said.
According to a PwC survey of over 4,700 CEOs released this January, 64% say that generative AI is likely to increase cybersecurity risk for their companies over the next 12 months. Plus, gen AI can be used to create fake news. In January, the World Economic Forum released its Global Risks Report 2024, and the top risk for the next two years? AI-powered misinformation and disinformation. Not just politicians and governments are vulnerable. A fake news report can easily affect stocks price — and generative AI can generate extremely convincing news reports at scale. In the PwC survey, 52% of CEOs said that GenAI misinformation will affect their companies in the next 12 months.
AI risk management has a long way to go
According to a survey of 300 risk and compliance professionals by Riskonnect, 93% of companies anticipate significant threats associated with generative AI, but only 17% of companies have trained or briefed the entire company on generative AI risks — and only 9% say that they’re prepared to manage these risks. A similar survey from ISACA of more than 2,300 professionals who work in audit, risk, security, data privacy and IT governance, showed that only 10% of companies had a comprehensive generative AI policy in place — and more than a quarter of respondents had no plans to develop one.
That’s a mistake. Companies need to focus on putting together a holistic plan to evaluate the state of generative AI in their companies, says Paul Silverglate, Deloitteâs US technology sector leader. They need to show that it matters to the company to do it right, to be prepared to react quickly and remediate if something happens. “The court of public opinion — the court of your customers — is very important,” he says. “And trust is the holy grail. When one loses trust, it’s very difficult to regain. You might wind up losing market share and customers thatâs very difficult to bring back.” Every element of every organization he’s worked with is being affected by generative AI, he adds. “And not just in some way, but in a significant way. It is pervasive. It is ubiquitous. And then some.”