🦾 Shadow AI - 15 February 2024

Demystifying AI - Nation State Use of AI and Countermeasures

Microsoft and OpenAI released briefings yesterday about nation state use of AI. When you look beyond the headlines, there weren’t a lot of newsworthy takeaways, but it did reinforce three viewpoints we’ve already been discussing in Shadow AI:

1) All types of cyber threat actors, including nation state actors, are using AI to varying degrees.

2) The main areas of current uplift are around enhancing existing tactics, techniques, and procedures in reconnaissance and social engineering.

3) AI has not led to significant attacks yet, but has enabled moves to enhance productivity and attack techniques.

Specifically, OpenAI and Microsoft note the activity of:

• Charcoal Typhoon, a Chinese threat actor, used their services to research various companies and cybersecurity tools, debug code and generate scripts, and create content likely for use in phishing campaigns.

• Salmon Typhoon, another China-backed actor, used their services to translate technical papers, retrieve publicly available information on multiple intelligence agencies and regional threat actors, assist with coding, and research common ways processes could be hidden on a system.

• Crimson Sandstorm, an Iranian threat actor, used their services for scripting support related to app and web development, generating content likely for spear-phishing campaigns, and researching common ways malware could evade detection.

• Emerald Sleet, a North Korean threat actor, used their services to identify experts and organizations focused on defense issues in the Asia-Pacific region, understand publicly available vulnerabilities, help with basic scripting tasks, and draft content that could be used in phishing campaigns.

• Forest Blizzard, an advanced Russian military intelligence actor, used their services primarily for open-source research into satellite communication protocols and radar imaging technology, as well as for support with scripting tasks.

As security practitioners threat model LLMs, identifying countermeasures for each tactic, technique, and procedure (TTP) being seen is critical. Potential countermeasures can blend of technological solutions, procedural adjustments, and educational initiatives to reduce the risk of exploitation.

AI News to Know

• AI in the Indonesian Election: As AI companies pledge to combat electoral disinformation, Indonesia provides a good case study into how AI companies will be challenged in meeting their commitments. Indonesia, the world’s third largest democracy, had it’s national election yesterday. Prabowo Subianto, a seventy two year-old Defense Minister and alleged human rights abuser who has claimed victory in the election, used AI image generator Midjourney to create an “cuddly grandpa” avatar in the lead up to the election. In a Council on Foreign Relations blog, Kat Duffy does a phenomenal job breaking down the challenges AI companies face establishing, and consistently and equitably enforcing, an election policy:

  “Is it problematic to make a cartoon version of a political candidate using an inexpensive American technology platform, as Prabowo’s campaign did? Many would argue it’s not. Would it seem more problematic if the AI avatar wasn’t a cartoon, but looked and sounded exactly like a political candidate, such as Imran Khan’s campaign in Pakistan? Might your decision depend on whether the AI use was disclosed? Or who was creating and disseminating the avatar? Or why they were doing it?”

• AI Security Overview: OWASP published a very helpful guide on how to address AI security. It distinguishes three types of threats: during development-time (when data is obtained and prepared, and the model is trained/obtained), through using the model (providing input and reading the output), and by attacking the system during runtime (in production) and offers controls to help mitigate the risks.

  

AI on the Market

• Slack Adds AI: Slack has launched some generative AI features as a paid add-on for Enterprise plans to help users search answers, provide channel recaps, and get caught up on threads. An internal analysis by Slack during the pilot found that customers, such as SpotOn, Uber and Anthropic could save an average of 97 minutes per user each week using Slack AI to find answers, distill knowledge and spark ideas.

• LLMs as an Operating System: NVIDIA launched “Chat with RTX,” an AI chatbot that can run locally on your Windows PC. This is NVIDIA's first step towards the vision of "LLM as Operating System" - a locally running, heavily optimized AI assistant that deeply integrates with your file systems, preserves privacy, and leverages your desktop gaming GPUs to the full. The localized approach ensures you do not have to share your data with cloud hosted LLMs like OpenAI or Anthropic and you can safely train it on your own locally stored data.