|
GenAI offers strategic business gains by automating routine tasks, producing creative content, and unlocking new efficiencies across industries. However, as with any transformative technology, GenAI carries substantial risks. Gartner predicts that over 30% of GenAI projects will be abandoned after POC stages by 2025, in part due to inadequate risk management controls. Organizations must navigate an evolving landscape of GenAI risks that could undermine trust, compromise data integrity, and lead to far-reaching negative consequences. Among the most critical risks are:
Understanding these risks is vital for organizations seeking to harness the full potential of GenAI without jeopardizing their operations or reputation. As GenAI continues to reshape industries, a proactive approach to risk management will separate the successful adopters from those whose projects fail due to unforeseen challenges. In this blog, we’ll explore four complex GenAI risks that can derail the realization of potential value, and provide insights on how organizations can safely and ethically explore new business opportunities in the GenAI space. #1 GenAI System Prompt LeaksSystem prompt leaks pose a significant security risk in the deployment of GenAI systems. Typically, a GenAI system prompt involves a set of instructions or a framework that’s provided to guide AI behavior during interactions. Prompts define the tone, scope, and context for the Large Language Model (LLM) responses, making them a critical element of the AI system’s functionality. Implementing various model safeguards and constraints ensure the system behaves appropriately and securely, preventing the system from generating harmful content or performing unauthorized actions. However, system prompts can be potentially accessed and exploited by attackers, bypassing safeguards implemented by developers. Attackers can then manipulate the AI to perform malicious actions, like executing harmful code, or extracting sensitive data. Vulnerability research conducted by HiddenLayer on Google’s Gemini Pro (previously Google Bard) revealed susceptibility to system prompt leakage when re-phrasing different questions to the system asking it to disclose its system prompts. HiddenLayer also misled Gemini into divulging sensitive data, such as secret passkeys, via coincidental input. Organizations must deploy and continuously improve robust GenAI safeguards and explore all possible vulnerabilities and abuse techniques impacting LLMs and GenAI. The AI models should also be trained to protect against harmful behaviors and sophisticated attacks, including prompt injection or jailbreaking. #2 GenAI Proprietary Data ExposureOne in four organizations (27%) prohibit the use of GenAI tools and applications, per a recent global survey conducted of 2,600 privacy and security professionals. Apple, Spotify, JPMorgan Chase, and Verizon are among some of these leading organizations banning ChatGPT use due to security and privacy concerns. One of the primary risks is the potential for employees to enter confidential or proprietary company data. Cisco reports that 45% of individuals have entered employee information and 48% have entered non-public company data. Per this same report, intellectual property risks account for 69% of the perceived threats reported by businesses, with the risk of information disclosure to competitors or the public ranking a close second at 68%. During the pandemic, many organizations grappled with security vulnerabilities because of employee use of personal devices while working remotely. In the rapidly evolving GenAI era, organizations face similar risks from unauthorized employee use of publicly accessible GenAI tools. Many organizations are proactively combating these risks by implementing controls to minimize exposure, including bans on specific GenAI tools and restrictions on the type of data that can be entered. Others are building their own internal GenAI applications instead of adopting off-the-shelf versions. Organizations must carefully weigh the benefits and pitfalls of either investing in a third-party GenAI solution, like those offered by OpenAI, against building an internal version. This decision ultimately comes down to budget, viable use case, existing infrastructure and capabilities, available skilled talent, and stakeholder buy-in, among other considerations. Also, organizations should evaluate ROI potential against business value; GenAI models only drive approximately 15% of average project costs, according to McKinsey. #3 GenAI Implementation RisksThe integration of GenAI into existing enterprise solutions and applications is accelerating as technology vendors embed AI functionalities into their products. This trend introduces potentially unknown risks for organizations that are exposed to GenAI’s effects through vendor solutions. In 2023, Zoom incorporated GenAI capabilities like automated meeting summaries, granting itself permission to train its AI models using customer data. Following backlash from enterprises and individuals concerned about privacy, Zoom reversed this policy. This incident illustrates the unforeseen risks that organizations face when vendors introduce GenAI features without sufficient transparency or controls. Even without adopting an end-to-end GenAI solution, organizations remain vulnerable as GenAI features become increasingly incorporated into their existing systems. For example, Microsoft introduced Copilot into Windows 11, and ServiceNow integrated GenAI across all workflows on its Now Platform. These implementations can introduce new challenges, especially as legal and regulatory risks surrounding GenAI continue to evolve. Many AI firms, such as OpenAI, are currently involved in copyright infringement lawsuits that could restrict the types of data they can use for model training. This may include new laws or regulations limiting access to specific data sources, particularly those involving personal information or copyrighted content. These legal and regulatory risks could extend to the organizations that use these AI products and services. Organizations might face legal liability if they benefit from AI models trained on contested data. Additionally, they could experience disruptions if a vendor is forced to change its business model, shut down services, or raise prices to cover legal fees or settlements. With 66% of CIOs and 45% of CEOs expressing concern that technology vendors fail to fully understand AI risks, organizations must be strategic when selecting third-party partners. To mitigate these risks, organizations should conduct thorough due diligence when evaluating vendors’ data protection, privacy policies, and training data usage. Requesting detailed documentation on AI model training processes is essential. Additionally, understanding how GenAI integrations work and their operational conditions is critical. Legal protection and indemnification are also key considerations. To promote responsible GenAI practices, some vendors offer indemnity against potential copyright infringement liabilities. For instance, Microsoft provides IP indemnification for its commercial customers through its Copilot Copyright Commitment. Other AI firms, such as Adobe and Google, offer similar legal safeguards for their GenAI products. #4 GenAI Data Poisoning AttacksData poisoning involves a malicious actor intentionally introducing harmful or misleading data into a GenAI model’s training dataset. The compromised data can corrupt the model’s learning process, resulting in biased, inaccurate, discriminatory, or dangerous outputs once the model is deployed. In more severe scenarios, the data set could be corrupted by inserting a backdoor or exploiting AI system vulnerabilities. For example, consider an AI model designed to detect suspicious emails or anomalous activity in a corporate network. A data poisoning attack could allow ransomware or phishing attempts to slip past security measures, avoiding detection by email filters or spam protection systems. Data poisoning can also cause GenAI systems to generate misleading information or make decisions that compromise security. This attack typology is often challenging to detect because it can mimic normal variations in data. Interestingly, data poisoning attacks are being leveraged by technologists to aid artists in combating IP theft from GenAI tools that automatically scrape the internet for content to train algorithms. Nightshade is a technology that applies a “cloaking” technique to deceive GenAI training algorithms into interpreting images as something different than what it actually depicts. An image of a car, for instance, could be falsely interpreted by the GenAI system as being an apple. To prevent data poisoning risks, many organizations are exploring the feasibility of Small Language Models (SMLs), which have fewer parameters and computational demands compared to LLMs while remaining effective for numerous language-related tasks. SMLs reduce the need to gather massive datasets that can be more difficult to validate and secure. A smaller dataset can also provide organizations with greater control, transparency, and security, potentially helping them avoid some of the challenges posed by large-scale GenAI models, including data poisoning. However, it’s important to note that SMLs may not replace larger models in every use case. How to Balance First-Mover Benefits with GenAI Risk MitigationWhile there’s often first-mover benefits to early adoption, organizations must remember that GenAI is a nascent technology and its full implications are still unknown. It's critical to mitigate potential GenAI risks by striking a balance between innovation and security. Organizations implementing GenAI applications must:
Organizations must also be mindful that human oversight and guardrails can make the difference between realizing early GenAI value and losing out on new business opportunities. Embracing a people-centric GenAI approach can augment human capabilities, foster creative potential, and ensure GenAI solutions are inclusive, ethical, and aligned with business values.
|