🦾 Shadow AI - 6 June 2024

Demystifying AI - Safeguarding Model Weights

Model weights, “the learnable parameters that encode the core intelligence of an AI,” are a crown jewel Frontier AI companies need to protect. Despite the critical importance of securing model weights, there is surprisingly little written on the topic. RAND’s recent research paper on securing model weights provides some important thought leadership on the issue and I dove into the report so you don’t have to. Coincidentally, as I was writing this, OpenAI released an article on how they are securing their research infrastructure, which includes a high-level description of how they are protecting model weights.

What are model weights and why are they important?

Model weights are the learnable parameters within an AI model that are adjusted during the training process. These weights encode the knowledge and capabilities of the AI, making them the core asset of any AI system. They stem from large investments in data, algorithms, compute resources, and other efforts. Compromising the weights would give an attacker direct access to the core intelligence of the AI, enabling them to misuse or replicate the model’s capabilities.

What are the potential attack vectors?

The RAND report identifies 38 different attack vectors associated with model weights, which can be broadly categorized into:

1. Running Unauthorized Code: Exploiting vulnerabilities in software to run unauthorized code.

2. Compromising Existing Credentials: Using social engineering or brute force methods to gain access to existing credentials.

3. Undermining Access Control Systems: Exploiting vulnerabilities in encryption, authentication, or other access control mechanisms.

4. Bypassing Primary Security Systems: Finding alternative ways to access sensitive data, such as through misconfigurations or less secure copies of data.

5. AI-Specific Attack Vectors: Exploiting vulnerabilities in the machine learning stack, engaging in model extraction or distillation, or using prompt-triggered code execution.

6. Nontrivial Access to Data or Networks: Gaining digital access to air-gapped networks or using side-channel attacks.

7. Unauthorized Physical Access to Systems: Direct physical access or placement of malicious devices in sensitive areas.

8. Supply Chain Attacks: Compromising services, equipment, or vendors used by the organization.

9. Human Intelligence: Using bribes, extortion, or insider threats to gain access.

How can companies secure them?

RAND proposes a five-level security framework to secure AI model weights, each with specific benchmarks:

Security Level 1: Basic Security Measures

• Develop a comprehensive security plan focused on preventing unauthorized access and theft.

• Centralize weights to a limited number of access-controlled and monitored systems.

• Reduce the number of authorized personnel with access to the weights.

• Implement insider threat programs.

Security Level 2: Enhanced Security Measures

• Harden interfaces against weight exfiltration.

• Conduct regular security audits and vulnerability assessments.

• Increase monitoring and logging of access to the model weights.

Security Level 3: Advanced Security Measures

• Implement multi-factor authentication for all access points.

• Use encrypted communication channels for accessing weights.

• Regularly update and patch systems to prevent exploitation of known vulnerabilities.

Security Level 4: High Security Measures

• Engage in advanced third-party red-teaming exercises to simulate sophisticated threat actors.

• Employ confidential computing techniques to secure weights during use.

• Establish secure, isolated networks for training and research activities.

Security Level 5: State-of-the-Art Security Measures

• Implement physical bandwidth limitations to prevent unauthorized data transfer.

• Develop hardware security modules specifically for securing model weights.

• Utilize secure and isolated environments for all stages of model development and deployment.

Recommendations

To start, companies should prioritize the following actions:

• Develop a Comprehensive Security Plan: Focused on protecting model weights from unauthorized access and theft.

• Centralize and Control Access: Limit the number of systems and individuals with access to the weights.

• Harden Interfaces: Ensure all access points to the weights are secure against exfiltration attempts.

• Implement Defense-in-Depth: Use multiple layers of security controls to provide redundancy and mitigate risks.

• Engage Advanced Red-Teaming: Simulate attacks from relevant threat actors to test and improve security measures.

Confidential computing is another key area that, although it may not be production ready yet, is critical to longer term efforts to protecting weights.

Conclusion

A breach of AI model weights by one of the leading Frontier AI companies could have widespread ramifications. The boards of Frontier AI companies should be closely governing their strategies to protect model weights while enterprises adopting AI should develop response and resiliency strategies if model weights of their AI providers are ever breached.

AI News to Know

Generative AI and Privacy: Google released a working paper outlining recommendations for how GenAI can use personal information while protecting user privacy. The paper provide a solid explanation for the different data stages of model development - data collection, pre-training, and fine-tuning - and how “personal data often makes up a small, but critical, portion of these datasets to help ensure the accuracy of the models.” It also shares initial considerations for how to apply foundational privacy principles, such as accountability, transparency, user controls, data minimization and protecting the privacy of minors, to GenAI.

AI and Covert Influence Operations: OpenAI recently released a report detailing campaigns by threat actors that have used their products to further covert influence operations (IO) online. In reviewing the full report, three things stood out:

1) 5 IO campaigns were linked to operators in Russia (2), China (1), Iran (1), and a commercial company in Israel (1).

2) They leverage Brookings Institute’s breakout scale to assess the impact on a scale from 1-6 and none of the operations included in the report scored higher than a 2. It’s unclear, however, whether they have seen IO operations not covered in this report that have registered greater impact.

3) All of these operators used AI, but none used it exclusively in their online IO. AI was primarily used to improve the quality of content to post or to generate large volumes of short comments.

AI on the Market

The State of AI in Early 2024: In the latest McKinsey Global Survey on AI, 65 percent of respondents reported that their organizations are regularly using Generative AI, nearly double the percentage from their last survey ten months ago. The number of organizations using AI for multiple business cases is also increasing with the most common business cases being marketing and sales, product development, and IT:

In regards to risks with GenAI adoption, inaccuracy and intellectual property infringement risks top the survey with cybersecurity risk close behind. Interestingly, the number of risks enterprises are actively working to mitigate reveal that there’s a lot more work to do in AI risk management strategies and governance.