🦾 Shadow AI - 9 November 2023

Demystifying AI - The Black Box Problem

When the business comes to me seeking to use an AI tool, one of the biggest challenges I face is the "black box problem." The opacity of the decision-making processes for many foundational AI models can make it challenging to fully evaluate the risk versus the benefit to the business.

The “black box problem” stems from the fact that many AI models are highly complex and consist of millions of parameters. These models can process vast amounts of data to make decisions, but the inner workings of their decision-making processes are often cryptic to humans. When you input data, the model produces an output, but the precise mechanisms in between to arrive at that conclusion remain hidden.

Security and IT practitioners can take several strategies with the business to help address the “black box problem”:

1. Explainable AI (XAI): XAI is a field focused on making AI more interpretable and understandable. Researchers are developing methods and tools to provide explanations for the decisions made by a LLM. These explanations are typically in human-readable forms, such as natural language descriptions, highlighting important features, or showing relevant data. Include XAI as a secure design requirement when building internal or customer-facing LLMs.

2. Data Governance: Ensuring the quality and fairness of training data is crucial. Data governance practices can mitigate the risk of biased or inaccurate outputs in AI models. Partner with your data leaders to help build out your organization’s data governance policy, data quality standards, data preprocessing capabilities, and data labeling capabilities.

3. Model Auditing and Red Teaming: Regularly auditing and red-teaming AI models can help uncover biases, errors, or unusual decision patterns. As we head into 2024, review your annual penetration testing budget and plan for the upcoming year and determine whether it needs to be bolstered to support the security needs of any AI model objectives the business has planned.

4. Refine your Third Party Risk Management: Look beyond a SOC2 report and standard penetration test and evaluate if you third party assessment process asks the right questions to help you assess the risk of external LLM usage:

   • Can the vendor provide details about the data sources used to train AI models?

   • Can they provide information about the AI model workings and logic?

   • What methods and technologies are used for data preprocessing, feature engineering, and model training?

   • How does the vendor ensure fairness and mitigate bias in their AI models, especially in sensitive applications like hiring, lending, or criminal justice?

   • Can the vendor provide historical performance data and insights into the model's success rate and false positives/negatives?

   • How long does the vendor retain company data, do they train their models with it, and what is their process for data deletion?

   • Can they ensure that data related to your organization is securely managed and deleted when no longer needed?

AI News to Know

• OpenAI’s Developer Conference: OpenAI held it’s first developer conference on Monday and, while I won’t recap the details, the announcement that OpenAI is launching a GPT Store where people can share their customized AI apps and earn money based on the number of users has huge implications for security practitioners. Daniel Miessler summarized it best in his Linkedin post where he highlights the risk Custom GPTs present by lowering the barrier to entry for attackers. “The possibilities of attack just became endless… We're entering a world where everything is about to be parsed by AIs that have code execution and action-taking capabilities, and implications are going to be massive.” In an earlier issue of Shadow AI, we examined theories for why we haven’t seen more AI-enabled attacks yet. With the release of Custom GPTs, AI-enabled attacks may quickly become the norm. On the bright side, I can direct all the tech support questions I get from family here:

  

  https://openai.com/blog/introducing-gpts?es_id=6a9eeae733

• AI Disclosure for Meta Ads: Starting in 2024, Meta will require advertisements on social issues, elections, or politics to disclose whether they are using AI in an attempt to reduce misinformation. This announcement follows one by Google that political ads played on YouTube will have to disclose the use of AI. It’s good to see companies start taking steps to combat election disinformation, but will be interesting to see if this goes far enough in educating platform users on what information in the messaging is actually real and what isn’t.

• AI Generated Sexually Explicit Photos of Minors: A troubling trend is emerging where AI is being used to make fake pornographic images of minors. The Internet Watch Foundation recently released a report on how AI is being abused to create child sexual abuse imagery that is “indistinguishable from real images.” At Westfield High School in my home state of New Jersey, the faces of at least 30 female students were pasted on top of pornographic images and posted on a website. Similar reports are surfacing over the past months in cities around the globe. There’s no federal law governing deepfake porn and President Biden’s executive order does not require companies to label AI-generated photos, videos and audio. At the local level, schools are going to need to train students on the dangers of AI, how to use it responsibly, and update their anti-bullying and harassment policies.

AI on the Market

• AI Copyright Battle: The U.S. Copyright Office and FTC have fractured viewpoints on generative AI copyright issues. The Copyright Office has argued the generative AI is “largely ineligible for copyright” because they open aren’t primarily human generated. However, the FTC is warning the AI development has enabled potential copyright infringement and consumer deception. This has prompted AI companies like Google, Microsoft, and OpenAI to pledge to protect customers if they get sued for copyright infringement.

• My LLM is Bigger than Yours: Amazon is reportedly training AI with 2 trillion parameters, twice as many as OpenAI’s GPT4. A model with more parameters is not necessarily better, and there are huge hardware, water, and energy requirements for such a large LLM. For water usage, it’s been estimated that ChatGPT uses up to a 16-ounce water bottle every time you ask it a series of between 5 to 50 prompts or questions.