Managing the Risks of Shadow AI

1. What is Shadow AI Shadow AI is the use of AI within an organization without the knowledge of or oversight from the IT or Compliance Department. Within Shadow AI, there are two categories of concern that can be particularly relevant:

2. Hidden Internally Built Models: Updating existing models for use cases such as automation internally or external interactions with customers in undetected ways (updating a credit risk model using an OpenAI model). • AI in 3rd party applications in models: Having AI within 3rd party applications or models upon installation or update without the knowledge of the firm using the application. • Unauthorized Internal Use of 3rd Party AI Tools: Using Generative AI tools such as Chat GPT to write code, get answers to questions, etc. without being careful about data & privacy leaks, sharing proprietary code, or lack of proper governance.

3. Why is Shadow AI so Dangerous? What makes these models especially dangerous is that firm’s don’t know what they don’t know, making addressing this risk particularly difficult without consistent and robustmonitoring systems. Here are some specific examples of when Shadow AI posed a significant risk to organizations: • Data Breaches: A notable e-commerce company suffered a significant data breach when an employee made use of an unauthorized AI product in order to optimize customer data analysis. This tool lacked the proper security measures and this resulted in the leak of customer information.

4. Biased Decision Making: An unapproved AI algorithm was found to be biased against certain demographic groups, which led to unfair lending practices. This resulted in a significant regulatory penalty to the organization. • Operational Failure: Major production delays and losses were incurred to a manufacturing company when an AI system without proper monitoring started making error-prone predictions for predictive equipment maintenance.

5. Regulatory Landscape In addition, these models specifically fall under the supervision of several current & likely upcoming regulations and therefore pose additional risk of regulatory penalties. • SS 1/23: This Supervisory Statement from the PRA goes into effect May 17th and sets the expectations for banks and financial firms that operate within the UK. SS1/23 Principle 2.6 Use of externally developed models, third-party vendor products. Firms should:(i) satisfy themselves that the vendor models have been validated to the same standards as their own internal MRM expectations.

6. The AI Risk Management Framework (U.S.): Released by NIST from the U.S. Department for Commerce on January 26, 2023, this framework guides organizations on how to govern, map, and measure risk to the organization, including 3rd party shadow AI risk. NIST GOVERN 6.1: Policies and procedures are in place that address AI risks associated with third-party entities, including risks of in- fringement of a third-party’s intellectual property or other rights. • The E.U. AI Act: This legislation passed by the E.U. more broadly regulates the use of AI within firms that may directly impact the safety and well being of the public and holds firms accountable for errors or poor practices that lead to public harm. • The Artificial Intelligence and Data Act (Canada): Sets the expectations for the use of AI within Canada in order to protect the interests of the public and require that appropriate measures be put in place to identify, assess, and mitigate risks of harm or biased output. 3rd party vendors that pose a risk to creating bias or harm within models are likely included within the risk mentioned within the regulation.

7. Hidden Use of AI or GenAI within an Organization Understanding the use of AI internally without the proper knowledge or oversight of the appropriate teams is a significant step in addressing the risk from Shadow AI. The following are strategies to mitigate this risk: • Consistent Monitoring for Undetected AI Models: Periodically scheduled scans that detect the probability of the use of AI within Models & EUCs can uncover risks before they result in errors and help meet regulatory requirements. • Comprehensive AI Testing Suite: Implementing a comprehensive AI testing suite is crucial for detecting and controlling Shadow AI. This suite should include tests for data drift, validity, reliability, fairness, interpretability, and code quality. Consistent documentation of test results in a standardized format helps maintain transparency and accountability for AI models that are detected.

8. Large Language Models (LLMs) Vulnerability Testing: Testing LLMs for vulnerabilities such as bias, fairness, harmful content generation, and revealing sensitive information helps stress test a model before it’s used by customers. • Explainable Large Language Models (LLMs): Content Attribution can help explain where within internal data sources the responses for prompts are coming from, helping to identify and mitigate causes of errors or the dissemination of incorrect information. • LLM Hallucination Testing: New research suggests that hallucination rates for LLMs may be higher than initially expected. As competitors race to adopt this technology and leverage it to enhance the customer experience, it can be critical to leverage the latest developments in RAG models and Challenger LLMs to monitor rates of LLMs giving customers incorrect information, or Hallucination Rates. • Implementing Controls and Accountability Measures: Controlling the use of Shadow AI involves managing access to End User Computing (EUC) models and tools. Implementing an Audit Trail to track model changes and Approval Workflows to ensure accountability can help mitigate risks associated with Shadow AI.

9. Identifying AI in 3rd Party Applications AI adoption in financial services has surged 2.5x from 2017 to 2022, per McKinsey, and continues to rise. As third-party vendors increasingly embrace AI, the risk of its use without firms' knowledge also grows. Key mitigation strategies include: • Identifying AI Models within 3rd Party Applications: Monitoring the behavior of 3rd party tools and executables and looking for patterns that may be indicative of the use of AI can be a necessary way to identify hidden risk of shadow AI. Consistent scheduled scans to identify and look for this risk can be a great way to mitigate this risk.

10. Interdependency Map: A model’s level of risk is highly dependent on the models and data sources that serve as inputs to that model. With an interdependency map, you can easily visualize these relationships and interdependencies. Paying special attention to 3rd Party Models that feed into high impact models can help prioritize where to look for shadow AI. • Security Vulnerabilities: Even if firms are aware of the use of AI within a 3rd party, it can be important to automate checks for security vulnerabilities within AI 3rd party libraries. • Monitor 3rd Party Model Performance: Many of these 3rd party models are black boxes and here the risk of shadow AI is highest as firms do not know what techniques a 3rd party vendor is using. Monitoring 3rd party models for sudden changes in performance can be an indicator for the use of shadow AI.

11. Monitoring Improper Use of 3rd Party Generative AI Unregulated reliance on tools such as Chat GPT or Microsoft Co-Pilot can lead to accidents such as how, at a major technology company, code was leaked to Open AI through the use of the company’s LLMS. Effective Risk Management of this unauthorized use could involve the following: • GenAI Detection Reporting: Scanning your landscape of EUCs and Models through cutting-edge AI detection algorithms can help get a better sense of the overall risk profile of your ecosystem in terms of inadvertent uploads into AI generators.

12. Securing Proprietary Code: Within code repositories, flagging the use of Generative AI can help uncover risks of leaking proprietary code to 3rd parties. • Flagging Hallucination: Running reports through AI detection can help identify documents that might suffer from errors due to hallucination from LLMs. • Demonstrating Governance and Compliance: Regulations such as The EU AI Act and SS1/23 are just the start among regulations requiring the documentation and enforcement of policy regarding the internal use of Generative AI within an organization.

13. Effective Shadow AI Risk Management According to The Economist, 77% of bankers report that AI will be the key differentiator between winning and losing banks so avoiding the use of AI is not impossible. Shadow AI can be a tough challenge for organizations to face, but with the right level of proactive monitoring, firms can unleash the massive benefits of AI, and especially GenAI, while limiting the risk. This involves effectively monitoring the risk from hidden AI models being used within the organization, the AI within 3rd party applications, and the submission of information into 3rd party AI Generators.

14. About Us • Established in 1988, CIMCON Software, LLC is a pioneer in end-user computing and model risk management, serving over 800 companies across industries. • Recognized by Gartner, Insurance ERM, and others as a top risk management vendor, CIMCON brings 25+ years of experience and industry best practices to support AI & GenAI readiness and governance. • With the largest global installed base, our feature-rich, extensively tested solutions offer unmatched depth, support, and reliability.

15. Contact Us Boston (Corporate Office) +1 (978) 692-9868 234 Littleton Road Westford, MA 01886, USA New York +1 (978) 496 7230 394 Broadway New York, NY 10013