Illustrated by Jeff Prymowicz
Most security teams aren't starting from zero with AI - they're starting from chaos: a handful of engineers using ChatGPT, personal Claude Code licenses, and no policy governing any of it. How do you get from “we’re experimenting with ChatGPT” to “AI is integrated with our security program”?
AI usage exists along a broad continuum. From greenfield organizations that vibe code 100% of their environments, to legacy ones that are slowly adopting chatbots, every company needs an AI security program and policy.
What does it take to build a great AI security program? In this blog post, we will discuss how to assess AI risk, how to train your team to use AI securely, how to measure whether tools are working properly, and a maturity model to assess how well your AI security program is working.
Why AI Security Needs Its Own Policies
Traditional security programs typically have policies for data, procedures, and compliance with those and regulatory bodies. However, AI adoption can make these policies obsolete, as they may not account for the velocity, operating procedures, and compliance implications of an AI agent. AI agents can make decisions at superhuman speed with less accountability, making policies more important yet more difficult to construct.
For example, a common security policy may be not to send sensitive corporate data to unknown endpoints. Humans can easily understand and follow this. However, Claude Code or other agents may not obey this policy due to context loss or prompt injection, making it harder to ensure DLP is properly followed.
On the defensive side, AI security tooling may be more difficult to evaluate than traditional deterministic tooling. For example, deterministic security tools like Semgrep can provide strong guarantees about how often they detect certain classes of vulnerabilities. AI tools may be more powerful, but due to non-determinism, they are not as measurable even with strong eval frameworks.
In order of importance, AI security policies need to:
- Address the risk introduced by AI tools
- Set up a framework on how to properly train your developers to use these tools; and
- Create metrics for measuring success criteria
Assessing Risk for AI Adoption
Using AI carries significant inherent risk around data exfiltration and excessive autonomy. To properly adopt AI into your organization, you should conduct a risk assessment first.
When running a risk assessment for AI adoption, the three main questions to ask are:
- What information will you be giving the AI access to?
The risk of AI adoption varies significantly depending on the data you are willing to give agents access to. Customer documentation, code, data, and logs may have different sensitivities, and pushing data may be more sensitive than pulling. Try to understand how and what an agent would be accessing, and the downstream implications of granting this access.
- What contractual obligations do you have?
Your customers and auditors may have certain expectations of how you use AI. For example, change management controls required by audits like SOC 2 may specify that a human must review all pull requests. Your customers may have certain contractual expectations about how their data is protected, which could also restrict how you can use AI.
- What technical safeguards can you apply?
Depending on your use case, you may be able to evaluate certain technical safeguards. For example, if you need to use a cloud AI agent, you may consider deploying it behind an agentic sandbox and firewall to reduce exfiltration risk. This may not be feasible for other use cases like local AI agents. You may also consider using a private LLM deployment vs hosted APIs, token filtering, or other AI security tools.
Secure AI Training
After assessing your organization’s risk tolerance, you should onboard and train your developers to securely use these tools. However, it is important to strike a balance between secure training and an overly long onboarding ramp. The best way to get results is through tactically focused workshops and online training.
Focus on the most impactful security implications of using AI: reviewing code and configuring these agents. The main questions to train your developers on are:
- How to review AI-generated code
AI is writing most code at greenfield companies these days. This makes code review the most important software skill today, as humans are the last line of defense before AI-generated pull requests make it to production. Secure code review training is incredibly important, so teach your developers how to review code for common security vulnerabilities like Insecure Direct Object Reference, authorization issues, and others from the OWASP Top 10.
- How to enable local agents securely
Given that most developers use local agents to interact with AI, your training should include tips on securely using these agents. These tips should include things like how to securely redact data before passing it to the AI, how to enable sandboxing for the current agent, how to prevent the agent from getting access to other files on the filesystem, etc.
Measuring Success
Now that your risk is understood and your organization is trained to use AI securely, the last step is to measure your success. Measuring success is important for determining which tools are most useful and for continuously improving your program.
Selecting the right metrics to measure can be difficult. Success means that your developers are using AI securely and that the AI properly detects issues. Some key metrics that should always be tracked are:
- Developer adoption
Whenever a new tool is rolled out, the share of developers who actively use it is a great indicator of the tool’s usefulness. Low developer adoption may indicate that the tool is not useful and should be improved or replaced.
- Detection rates
Continuously testing your tools to ensure that they properly detect issues (such as vulnerabilities) is the only way to get confidence in your tooling. For vulnerability detection, compare AI detection rates with traditional static application security testing (SAST) tools like Semgrep to gain confidence and deploy across your organization.
- False positive ratio
The false positive ratio helps determine the efficacy of the tooling. High false positives indicate that the tooling needs to be fine-tuned further.
For all metrics, make sure to collect a baseline before AI is enabled across the organization. This helps track how successful your rollout was.
AI Security Maturity Model
Based on all the steps above, you can create a maturity model for your organization to reach secure AI readiness.
Level 0: Ad Hoc
Level 0 is how most companies start. There are likely no AI security standards set, even if employees are using agentic software on corporate data.
Level 1: Experimenting
At level 1, the risk of AI is understood, and there are some metrics being tracked. You should be aware of the data that these AI tools can access, and a written policy should be in place, even if it is not fully comprehensive.
To get to this level, you should:
- Run a risk assessment
- Create data classifications for all data that is accessible to an AI agent
- Create a draft security policy, with information about acceptable data to use with the agent
Level 2: Defined
At level 2, AI is a standard part of the development and management process. Your policy should be comprehensive, and your processes should be consistent. To get here, you should:
- Create a comprehensive security policy that covers risk, data handling, and developer responsibility
- Have a plan for what metrics to collect, including:
- Detection ratios
- False positive ratios
- Developer adoption
- Have baselines for all metrics
- Collect metrics consistently across all teams
- Include a secure AI training within your organization’s onboarding
Level 3: Mature
At level 3, you should be using your metrics to continuously improve your security program. Developer adoption should be high, false positive rates should be tuned to low levels, and you should have high confidence in your agents. To achieve this stage, you should:
- Have a process to evaluate false positives and detection ratios regularly
- Have a process to increase true positives
- Evaluate new tool capabilities before they are adopted internally
- Build strong technical safeguards to prevent agents from going off their guardrails, such as agentic sandboxes
Conclusion
Building an AI security program is all about being intentional with how AI is enabled and driving progress with metrics as much as possible. Vibing may be great when coding, but it is not a good idea when building a security program. Targeting your secure AI training, building a robust risk register and generating the right metrics will give your organization the tools it needs to adopt AI responsibly, reduce risk, and continuously improve.
Reaching maturity for your AI security program can be a significant task, but it is crucial for ensuring safety and velocity. If you need help evaluating and building out your AI security program, please reach out to us at Cloud Security Partners.
Stay in the loop.
Subscribe for the latest in AI, Security, Cloud, and more—straight to your inbox.
