🦾 Shadow AI - 28 March 2024

Demystifying AI - Securing Open Source LLMs

One of the AI predictions I made for 2024 as 2023 wrapped up was that “open source AI models augmented by proprietary corporate data sets will continue to pick up steam as enterprises look to increase AI adoption on specific use cases.” Last week, a16z released the results of a survey with enterprise technology leaders, and their findings support this prediction:

The adoption of open source large language models (LLMs) is poised to accelerate further with the recent release of DBRX, an open source LLM by Databricks whose performance surpasses other popular open source models like Llama 2, Mistral, and Grok AI and comes close to GPT-4.

Open source LLMs are attractive to many enterprises because running them within internal IT infrastructure or private cloud environments allows greater control over proprietary data shared with the models, reducing third-party risk exposure, but increasing internal security demands.

So What For Security?

The growing adoption of open source LLMs by enterprises seeking to enhance their AI capabilities has significant security implications across several key domains:

1. Data Privacy and Information Leakage: Training LLMs on proprietary enterprise data increases the risk of sensitive information leaking into model outputs or parameters. Security teams must implement robust data governance, anonymization, and monitoring processes to prevent unintended disclosure of confidential data.

2. Model Integrity and Supply Chain Security: Open source LLM codebases and pre-trained models may inadvertently or maliciously contain vulnerabilities or backdoors. Comprehensive code reviews, secure deployment pipelines, and runtime monitoring are crucial to mitigate supply chain risks.

3. Adversarial Robustness and Model Hardening: LLMs can be susceptible to adversarial input perturbations that cause them to generate unsafe or undesirable outputs. Enterprises should invest in techniques like adversarial training, output filtering, and behavioral monitoring to enhance model robustness.

4. Responsible AI and Ethical AI Governance: The use of open source LLMs amplifies concerns around bias, factuality, and the sociotechnical implications of large language models. Security teams must collaborate with ethics boards and legal counsel to develop appropriate guard-rails and governance frameworks that uphold ethical AI principles.

5. Access Controls and Credential Management: Fine-tuning LLMs with proprietary data requires robust access controls, encryption, and credential management to prevent unauthorized model access or misuse by insiders or external threat actors.

6. Compliance and Regulatory Considerations: Depending on the industry and use case, enterprises may need to navigate relevant regulatory requirements (e.g., GDPR, HIPAA) when handling sensitive data used for LLM training or when deploying LLM-based applications.

As enterprises increasingly leverage the power of open source LLMs for internal and external use cases, security teams must proactively address these emerging risks. Close collaboration with data scientists, DevOps, ethical AI teams, and business stakeholders is essential to build a secure, trustworthy, and responsible enterprise AI ecosystem.

AI News to Know

ShadowRay: Oligo Security discovered an active attack campaign targeting a vulnerability in a widely used open-source AI framework called Ray. The vulnerability had been disputed which created a blindspot on security teams because it several popular vulnerability databases and scanning tools did not yet account for it. The “shadow vulnerability” allowed attackers to exploit expensive GPUs, mine cryptocurrency, and leak sensitive data.

Hack the Box: Google DeepMind investigates the potential risks posed by advanced AI systems through comprehensive evaluations of the Gemini 1.0 models in key areas such as persuasion and deception, cybersecurity, self-proliferation, and self-reasoning. Key findings reveal no immediate dangerous capabilities in these models, but forecasts that AI systems will be able solve 50% of diverse set of medium difficulty Capture the Flag challenges from the Hack the Box platform.

Leveraging LLMs for Threat Modeling: Marcin Niemiec assessed Claude 3 Opus vs GPT-4 against four areas: high-level security design review, threat modeling, security-related acceptance criteria, and review of architecture. While results can vary from execution to execution, he generally found that “Claude 3 Opus performed slightly better, demonstrating a greater ability to identify threats and adhere closely to instructions.” Interestingly, GPT-4 did not provide analysis for Denial of Service or Elevation of Privilege threats.

Managing AI Cybersecurity Risk in the Financial Sector: Large financial institutions are well positioned to leverage AI because they have built up strong governance and guardrails over the years as they leverage AI for common use cases like anti-fraud operations. A report by the U.S. Department of the Treasury highlights how financial institutions are integrating AI risk management into existing frameworks, developing tailored AI risk management strategies, and ensuring cross-functional collaboration for effective risk governance. It also emphasizes the evolving role of the Chief Data Officer in managing the data supply chain critical for AI systems. If you’re interested in learning more about the integration points between cybersecurity and the Chief Data Office check out a recent issue of Shadow AI where I shared some lessons.

AI on the Market

AWS <> Anthropic: AWS completed its $4B phased investment in Anthropic as part of their strategic collaborative agreement. Companies like ADP, Amdocs, Bridgewater Associates, Cloudera, and Delta Airlines as already using Amazon Bedrock and Anthropic’s family of models and others to deploy generative AI applications.

Microsoft Co-Pilot ROI: Lumida Wealth puts out an interesting non-consensus newsletter each week and a recent one had an insightful strop from an unnamed CTO at a top 3 bank. The bank is testing Microsoft Co-Pilot and is seeing a 15% increase in productivity compared to Microsoft’s marketed 50% in productivity. A 15% productivity increase can still be significant, but there are regulatory, data security, and intellectual property challenges to navigate.