🦾 Shadow AI - 27 June 2024

Demystifying AI - Q2 AI Regulation Update

I received good feedback on last week’s Q2 AI threat update so I thought it’d also be helpful to share an AI regulation update as we wrap up the quarter.

Although U.S. Federal AI legislation is moving at a snail’s pace (not surprisingly given it’s an election year), there have been two notable state-level developments in the United States.

Colorado’s Consumer Protections for Artificial Intelligence Act

In May, Colorado was the first state to pass AI legislation, which requires developers of high-risk AI systems to use reasonable care to protect consumers from risks of algorithmic discrimination in the system starting February 1, 2026. High-risk AI systems are defined as “any AI system that, when deployed, makes, or is a substantial factor in making, a consequential decision,” but the legislation does include some important exceptions. Colorado’s AI Act also requires the deployer of a high-risk system to use reasonable care to protect consumers from any known or reasonably foreseeable risks of algorithmic discrimination. Reasonable care including having a risk assessment policy and program, completing an impact assessment, and public disclosure of the high-risk AI system.

California’s AI Safety and Innovation Bill (SB 1047)

SB 1047 is a proposed bill in California that would make large frontier model providers like OpenAI, Anthropic, and Meta liable for any “critical harms” from their AI systems. It also effectively requires large frontier model providers to create a kill switch for their AI model in the event of an emergency. The bill presents real risk to open-source AI efforts at companies like Meta, Mistral, and open-source AI startups because they would be responsible for bad actors that use their AI models. The bill pass the California Senate in May and recently passed the Assembly Privacy Committee 8-0. It is fast approaching an August vote that, if passed, would send the bill to Governor Newsom’s desk for decision on whether to sign.

Globally, there are a number of key regulations at various stages of development that bear watching to inform your security compliance programs:

AI News to Know

AI and Zero-Day Vulnerabilities: Researchers from the University of Illinois Urbana-Champaign recently published a paper detailing how teams of LLM agents are capable of exploiting real-world zero day vulnerabilities. They tested a system of agents against 15 real-world web-based, open source vulnerabilities and found the system of agents can exploit real-world zero-day vulnerabilities with an overall success rate of 18%. With the average cost for a run at $4.39, the total cost would be $24.39 per successful exploit. The research has both positive and negative implications. On the positive side, penetration testers could potentially conduct more frequent and cost-effective testing. On the negative side, even though we have not seen broad use of attackers leveraging LLMs for vulnerability exploitation, it’s very possible LLM agents exploiting zero-day vulnerabilities is on the horizon.

AI to Bolster Cybersecurity: Google Cloud CISO Phil Venables lays out 3 promising use cases for cybersecurity in June’s edition of Google’s Cloud CISO Perspectives:

1) AI for malware analysis: Google recently tested the malware analysis effectiveness of Gemini 1.5 Pro and found it was able to analyze malware in 30 to 40 seconds and generate summarized reports. While Google says it was “notably accurate,” the exact accuracy level was not revealed.

2) AI to boost SecOps teams: Google highlights how customers like “Pfizer and Fiserv are using natural language queries with Gemini in Security Operations to help new team members onboard faster, enable analysts to find answers more quickly, and improve efficiency of their security operations.”

3) Identifying and fixing vulnerabilities: Google successfully instructed AI foundation models to write project-specific code that could improve fuzzing coverage to identify more vulnerabilities. They then built an automated pipeline to analyze vulnerabilities, generate patches, and test the fixes before selecting the best ones for human review.

AI on the Market

OpenAI Withdrawing from China: OpenAI is planning to restrict access to its API platform in China starting July 9th. Chinese companies like Baidu and Alibaba are swarming to fill the void, but perhaps this is a carrot to gain more business from the United States Government? Appointing Paul Nakasone, a well respected retired leader of the NSA and CYBERCOM to their Board and then restricting access to China are two big moves by OpenAI in the past two weeks.

State of the Cloud in 2024: Bessemer Venture Partners released its annual State of the Cloud report which includes 5 key AI trends. For IT security professionals reports like these are valuable to see where the market may be headed, including vertical AI companies eventually taking market share from legacy SaaS products.