When AI meets Cybersecurity: Strategic Management for a Safer Tomorrow

BY: Venus Liong

As AI systems become deeply embedded in critical infrastructure, enterprise workflows, and consumer applications, the cybersecurity stakes have never been higher. Recently, I had the opportunity to attend the Women in Cybersecurity (WiCyS) AI Learning Series — a focused 6-session program for mid- to senior-level professionals navigating the intersection of AI and security. 100 completed participants were also admitted to a condensed course of AIS247: AI Security Essentials for Business Leaders from SANS Institute led by Dan deBeaubien. The sessions delivered actionable insights into the emerging challenges of AI cybersecurity management, from adversarial risks and model misuse to governance gaps and regulatory pressure.

In the sections below, I have summarized the key concepts from the series and listed some public resources that can help organizations build resilient, transparent, and accountable AI systems. Whether you’re refining your threat modeling strategy or developing AI-specific security policies, this guide is designed to help you move from awareness to execution.

AI secure software development
AI threat modeling
AI-BOM: Third-party AI/ML product management
AI Policy and Regulations
AI Red Teaming
Where to start: Turning principles into practice

AI secure software development

NIST has published SP800–218 Secure Software Development Framework (SSDF): Recommendations for mitigating the risk of software vulnerabilities. In addition, SP800–218A is published for Generative AI and Dual Use Foundation Models Secure Software Development Practices is published to address the new AI use cases. The two guidelines provide a checklist of practices to enable secure software development. In fact, AI secure development is an augmentation instead of replacement of existing secure software development processes.

Some of the rules of thumbs when designing AI features (from AIS247):

Treat AI as a user with principle of least privilege. AI should not be an admin or privileged user with blind trust.
Don’t ask AI to keep data as secret. Whatever AI can see will be leaked.
Don’t ask AI to verify itself. It will always said it’s correct. Instead, always cross-verify results with subject-matter experts.

AI threat modeling

There are many guidelines for AI threat modeling. In high level, it is split into the data, the model and the usage. To make the analysis more thorough, many guidelines have disserted the threat modeling problems into different views of diagrams. One good start is NIST AI 100–2 E2023: Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations. It classifies the threat models for AI into Predictive AI and Generative AI. The threat models are extracted in Figure 1 and Figure 2 below. I like this model because it extends from traditional Confidentiality, Integrity and Availability (CIA) triads. Privacy can be considered as confidentiality. Then, the only security goal that is added is Abuse in Generative AI.

If you don’t like this representation, there are other threat models posted from:

OWASP AI Threat Model, Security Matrix and Threat Model with controls: https://owaspai.org/docs/ai_security_overview/#threat-model

Microsoft AI Threat Model Ontology: https://ashy-coast-00aeb501e.6.azurestaticapps.net/MS_AIRT_Lessons_eBook.pdf

Figure 4: Microsoft AI Threat Model Ontology

TTPs: The Tactics, Techniques, and Procedures leveraged by the Microsoft AI Red Teaming (AIRT).

A typical attack consists of multiple Tactics and Techniques, which is mapped to MITRE ATT&CK® and MITRE ATLAS Matrix whenever possible.
Tactic: High-level stages of an attack (e.g., reconnaissance, ML model access).
Technique: Methods used to complete an objective (e.g., active scanning, jailbreak).
Procedure: The steps required to reproduce an attack using the Tactics and Techniques.

Are there tools that can generate threat models for AI products? Some has pointed to STRIDE-GPT. If you are concerned with IP leak, you can host your own version from its git repo: mrwadams/stride-gpt: An AI-powered threat modeling tool that leverages OpenAI’s GPT models to generate threat models for a given application based on the STRIDE methodology.

AI-BOM: Third-party AI/ML product management

Similar with Software Bill of Materials (SBOM) which was created to manage risks for software supply chain, AI Bill of Materials (AI-BOM) was created to track dependency and validate artifact integrity. One article that explores this very well was Google’s Securing the AI Software Supply Chain.

The supply chain risks associated with creating a software artifact can be found in Figure 5 below. It is modeled very similarly with the supply chain risks associated with training a model in Figure 6.

Figure 5: Supply chain risks in traditional software

Figure 6: Supply chain risks associated with training a model

The differences specific to AI development are illustrated in Table 1 below. The central concept of the AI-powered application is the model, which is the pairing of code and weights created as part of the training process. It extracts statistical patterns from data and use these to make predictions (or inferences) on new data.

Table 1: Differences for supply chain management between traditional software and AI development with model training

Since training from scratch is expensive, many companies prefer to build a pre-trained model and construct a new model on top, which is referred as Transfer Learning. Transfer learning teaches a model trained for a specific task to perform a different task. Fine tuning is a type of transfer learning that freezes most of the model weights and updates only the last few computations in the model architecture. An example is to adapt LLM to X-ray imaging for medical purposes.

Figure 7: An overview of transfer learning process. For finetuning, the first stages of the model weights are frozen and the model is trained on a task-specific dataset.

How to manage the complexity and difference of the supply chain of AI development? The answer is model and data provenance! Model provenance refers to the traceable history of how a ML model was created, including its training code, config and dependencies, ensuring reproducibility and accountability. It could be signed in the form of Supply-chain Levels for Software Artifacts (SLSA). On the other hand, data provenance captures the origin, transformation, and lineage of the datasets used, allowing auditors to verify the data used for training or evaluation is authentic, untampered and policy-compliant.

Google has provided open source SigStore for developer to sign and verify software or AI models by use of ephemeral (or temporary) keys and therefore abstracting away the key management complexity. It can also be used to sign and verify AI agent identities and capabilities. The model-validation-operator can also be used to verify signed AI models before they are deployed in Kubernetes, ensuring only trusted models run in production.

An AI-BOM is a structured inventory that documents every component involved in building, training, deploying and operating an AI system. It is an umbrella structure that encapsulates both model and data provenance. However, much of its work is still ongoing and captured in OWASP AI-BOM project.

With the fascinating materials presented, we will end the section with an end-to-end supply chain risks in Figure 8, which present you a more macro view of supply chain risks than the model training process in Figure 6.

Figure 8: Supply chain risks in foundation model training

AI Policy and Regulations

It’s almost impossible to talk about cybersecurity management without referencing Governance and Regulations.

NIST has launched AI Risk Management Framework (AI RMF) in 2023. It aims to guide organizations on the responsible development, deployment and use of AI systems. On the other hand, there is no comprehensive US Federal Law dedicated solely to AI. In 2025, The Trump administration has taken steps to deregulate the development and use of AI systems at the federal level. In response, more states may fill the void by adopting their own legislation and regulations such as California, NYC, Colorado and Illinois regulating AI systems’ use in employment.

The European Union (EU) AI Act passed in Dec 2023 established a legal framework for the development, deployment and use of AI systems in the EU, with most terms becoming effective in August 2026. US companies will be obligated to comply if the output produced by AI systems is intended to be used in the EU. It explains the EU’s approach to regulating AI based on risk levels — banning systems with unacceptable risk (like social scoring), tightly regulating high-risk applications (e.g. biometric surveillance, hiring tools), and leaving low-risk systems mostly unregulated.

Other reference also included General Data Protection Regulation GDPR (Article 22: Automated individual decision making, including profiling) which grants individuals the right to information about automated decisions.

AI Red Teaming

Microsoft AI Red Team has published its lessons from red teaming 100 generative AI products. The results and lessons are very insightful! Part of the work is also presented in BlueHat India 2025 Day 2 Keynote by Ram Shankar Siva Kumar.

Understand what the system can do and where it is applied, including capability constraints and downstream applications.
You don’t have to compute gradients to break an AI system. In other words, you don’t need to know the internals or training mechanics to break an AI system. Real hackers can simply use prompt engineering!
AI red teaming is not safety benchmarking.
Automation can help cover more of the risk landscape. PyRIT, an open-source framework for AI red teaming and security professionals, provides power components including prompt datasets, automated attack strategies, scoring the outputs of the models and automatically jailbreak a target model. It can help testing at scale.
The human element of AI red teaming is crucial
Responsible AI harms are pervasive but difficult to measure
Large Language models (LLMs) amplify existing security risks and introduce new ones
The work of securing AI systems will never be complete

Where to start: Turning principles into practice

Every effective AI cybersecurity program begins with clarity: define your security objectives, identify critical assets, and document assumptions about system attack surface and threats. From there, anchor your efforts in robust guidance — NIST SP 800–218A, tailored for Generative AI and Dual-Use Foundation Models, extends the Secure Software Development Framework (SSDF) to meet the unique demands of modern AI systems. This is not just a compliance checkbox — it’s the foundation of a resilient, scalable AI management strategy.

Success hinges on cross-functional collaboration. Product teams, IT, governance, cybersecurity, and the Board must share ownership of AI risk. Clear roles, shared language, and aligned incentives foster transparency and accountability across the organization.

Your policies should go beyond theory. Embed human-in-the-loop safeguards, mandate AI-BOMs, and operationalize threat modeling, risk assessments, and AI usage policies. Don’t forget the human layer: invest in training programs that equip teams to recognize, respond to, and mitigate AI-specific threats.

AI cybersecurity isn’t a silo — it’s a strategic imperative. Start small, but build with scale in mind. The threats are evolving. Your defenses should too.