New Security Threats in AI/ML Applications: The Warning Issued by CVE-Bench

Introduction: New AI Security Threats and the Emergence of CVE-Bench

As Artificial Intelligence (AI) technology permeates every aspect of our daily lives, the focus of security is shifting from the performance of AI models themselves to the security of the applications that utilize them. While past security threats were largely centered on finding simple software bugs, we are entering an era where AI agents can autonomously detect and exploit vulnerabilities. Because AI/ML applications involve complex data structures and interactions, they create new attack surfaces that are difficult to defend using traditional security methods.

In this context, CVE-Bench has recently garnered significant attention. According to OpenAI's 'GPT-4o System Card', CVE-Bench is a benchmark designed to test how effectively AI models can identify and exploit vulnerabilities in real-world web applications. Although these experiments are conducted within an isolated sandbox environment, the scenarios are highly realistic. By measuring a model's ability to find flaws in functioning systems—moving beyond mere code analysis—the benchmark allows us to gauge the potential risks of AI-driven cyberattacks.

We must move past simply marveling at how "smart" AI is and begin seriously considering the devastating scenarios that could unfold when this intelligence is applied to vulnerability detection. The experimental results from CVE-Bench suggest that the capabilities of AI models may extend beyond the intuition of security professionals, functioning instead as potent, automated attack tools.

Body 1: Analyzing Vulnerability Detection Mechanisms via CVE-Bench

The fundamental value of CVE-Bench lies in its ability to measure how successfully a model can launch an attack even when deprived of specific information. The experiments were conducted using two primary prompt configurations. The first is the "Zero-day Prompt" method. In this environment, the AI model is not provided with specific vulnerability details; instead, it receives only a general task description (e.g., "perform a specific task"). This mimics a real-world attack scenario where an attacker explores a system to find vulnerabilities without prior knowledge of them.

In contrast, the "One-day Prompt" configuration provides the model with a specific description of a vulnerability to be exploited. When compared to this method—where attacks are performed based on known vulnerability information—the success rate in the Zero-day method serves as a decisive metric for measuring an AI's autonomous detection capabilities.

Furthermore, these tests were conducted under highly stringent conditions. According to the experimental setup, the AI agents were not granted access to the web application's source code. This means the models had to discover vulnerabilities solely through "remote probing" from the outside. This reflects the most typical and dangerous attack scenario: an attacker attempting to penetrate a system via the network without knowing the internal server logic.

Body 2: Diversity of Attack Targets and the Vulnerability of AI/ML Apps

Another reason CVE-Bench is attracting attention is the breadth of its targets. The experiment went beyond testing single pieces of software to include various components that make up the modern web ecosystem. Specifically, the test subjects included Content Management Systems (CMS), AI/ML applications, business management tools, operational monitoring systems, and e-commerce platforms. It also covered lower-level components such as libraries, packages, and web infrastructure, demonstrating just how vast the "attack surface" available to AI can be.

Of course, technical limitations existed during the process. According to the OpenAI report, due to difficulties in infrastructure portability, only 34 of the total 40 challenges were successfully executed. While this highlights the immense difficulty of recreating complex real-world environments as benchmarks, it paradoxically proves just how complex and interconnected the infrastructure we need to protect truly is.

This wide range of targeting implies that AI models can move beyond simple web pages to target core enterprise assets, such as mail servers, computing management tools, and web portals. The possibility of "chained attacks"—where an attacker uses vulnerabilities in packages and libraries to penetrate higher-level applications—is a factor that security architects in the AI era must prioritize.

Body 3: Security Model Consistency and the Potential for Detection Evasion

When evaluating AI model performance, security professionals often use a metric called "pass@1." This refers to the ability to correctly identify a vulnerability in a single attempt. Using this metric, OpenAI measured how consistently models could find vulnerabilities that internal cybersecurity experts deemed relatively "straightforward."

A crucial concept here is the "cost-intelligence frontier." If a model demonstrates consistent performance, it means an attacker can find vulnerabilities with a high degree of probability while incurring low costs (minimal prompt input and computational resources). If an AI's detection capability is consistent, it guarantees a predictable success rate for attackers conducting large-scale automated attacks.

Furthermore, this consistent capability leads to the fatal risk of "evading detection mechanisms." Security systems such as IDS/IPS (Intrusion Detection/Prevention Systems) are typically designed to catch anomalous patterns or repetitive scanning. However, if an AI can intelligently adjust its attack patterns to produce consistent results, there is a potential risk that it could neutralize existing volume-based detection systems.

Conclusion: Security Strategies and Future Challenges in the AI Era

The results of the CVE-Bench experiments deliver a clear message: the security of AI/ML applications has entered a stage where simple software updates or patches are no longer sufficient. As AI models acquire the ability to autonomously probe and exploit vulnerabilities, continuous benchmarking and proactive Red Teaming activities become essential.

Future security strategies must be strengthened in two directions. First, considering the autonomous detection capabilities of AI agents, we must build behavior-based detection and response systems that go beyond source-code level analysis. Second, we must more rigorously apply "Zero Trust" principles to protect the links between AI/ML applications and their surrounding infrastructure (libraries, packages, etc.).

In conclusion, CVE-Bench is more than just a performance testing tool; it is an early warning system for new security threats in the age of AI. To protect our systems from automated attack tools, we must move beyond technical defenses to establish robust security governance and regulatory frameworks that keep pace with the rapid advancement of AI.