The Execution Problem at the Heart of Closed-Loop Vulnerability Fixing

The Gap Between Detection and Proof

Static analysis tools speak in likelihoods, not certainties. When Semgrep fires a SQL injection rule, it is reporting that a dangerous pattern exists in your code. When a CodeQL query returns results, it is telling you that user-controlled data reaches a SQL execution sink without passing through a recognized sanitizer in the data flow graph. Neither of those tools can tell you whether the endpoint is publicly reachable, whether authentication middleware runs before the vulnerable code path, or whether the application’s actual runtime behavior differs from what the static model predicts.

That is the gap between “might be exploitable” and “is exploitable.” And closing it is the core technical challenge behind Codex Security, OpenAI’s AI application security agent now in research preview.

The three-step framing, detect then validate then patch, sounds like a natural extension of existing static analysis. In practice, the validate step requires crossing a hard technical boundary: from reading code to executing it.

What the State of the Art in Static Analysis Actually Does

CodeQL is the most sophisticated static analysis tool available for general use. It is what GitHub Advanced Security runs under the hood, and it is what teams at Google and Microsoft use for large-scale codebase audits. Its approach is fundamentally different from pattern-matching tools: CodeQL converts the codebase into a queryable relational database representing the entire code structure, including call graphs, data flow, control flow, and type information. Queries are written in QL, a purpose-built declarative language, and the query engine traverses those relationships to find code paths satisfying vulnerability patterns.

A CodeQL query for SQL injection expresses something like: find all flows of data originating from an HTTP request parameter that reach a database query execution function, where no sanitization function intervenes in the data flow path. The query engine answers that across the full codebase. It handles interprocedural analysis, understands library APIs, and can track taint through complex call chains across file and module boundaries.

This is sophisticated work, and it still produces false positives. The reason is structural: static analysis reasons about the code, not about the running system. It cannot know that a particular route always requires an admin session because that check happens at the framework routing layer in a configuration file the analyzer does not fully model. It cannot know that the database account used in production only has SELECT permissions, making the injection a data leak rather than an arbitrary write. Those facts are runtime properties, and static analysis is not in the business of runtime.

What Validation Actually Requires

Dynamic analysis tools like OWASP ZAP or Burp Suite take the opposite approach. They run against a live application, send crafted payloads to every parameter they can discover, and look for evidence that the payload had effect: a database error message, a time delay from a sleep injection, a reflection of injected content in the response. When ZAP reports a confirmed SQL injection, it has actually injected SQL and observed a consequence. That finding is proved, not inferred.

The infrastructure cost is significant. To run a dynamic analysis, you need a fully running instance of the application. For a web service with standard dependencies, that means a container image, a database, a cache if the application uses one, and mock versions of any external services the code calls. You need an entrypoint to drive HTTP traffic. You need a valid application state, including any seed data required to reach the code paths you want to test.

A closed-loop security agent that genuinely validates vulnerability findings has to do something in this space. The most plausible interpretation of “validate” is that the agent runs the code with a crafted payload targeting the specific vulnerability it identified statically, then observes whether exploitation succeeds. If it can do that, it has crossed from probabilistic detection into demonstrable confirmation. The noise problem goes away because every surfaced finding has been witnessed, not inferred.

The infrastructure required for that confirmation step is not trivial. It involves:

• Resolving and installing the project’s dependencies
• Building the application, including any compilation or bundling steps
• Provisioning the services the application depends on, or plausibly simulating them
• Generating a targeted exploit payload based on the specific code path identified
• Running the application, sending the payload, and observing the response

For a web application with a relational database and no unusual external dependencies, this is achievable in a containerized sandbox. For applications with deeper external integrations, or applications that require significant setup state before the vulnerable code path is even reachable, the infrastructure complexity scales quickly.

Patch Validation Follows the Same Logic

Once a fix is generated, validating it requires running the same confirmation in reverse. Apply the patch, attempt the original exploit again, and confirm exploitation fails. Then run the existing test suite and confirm nothing regressed. If both conditions hold, there is reasonable evidence the fix is correct.

The quality of this validation is bounded by two things: the quality of the exploit generator and the quality of the test suite. A patch that closes the specific injection point the agent tested may leave a variant intact. A patch that passes the tests being run may break a test that does not exist yet. These are not hypothetical concerns; they are the ordinary failure mode of automated testing in any domain, applied to a context where the consequences of a near-miss are harder to observe.

For vulnerability classes where the exploit is deterministic and the fix is local, this pipeline works well. SQL injection is an ideal case: the exploit is a crafted string, the fix is parameterization, and the behavioral change is testable. Hardcoded secrets are even simpler: extraction to an environment variable changes nothing about application behavior and is trivially confirmed.

The cases that resist this pattern are the ones that also resist human remediation: race conditions, which require concurrent execution to confirm reliably; deserialization vulnerabilities, which depend on specific gadget chains in the runtime environment; authentication and authorization flaws, which require reasoning about session state and permission models across the full application; and cryptographic weaknesses, which may require mathematical rather than observational confirmation. These vulnerability classes are also among the most consequential when exploited.

The Research Preview as Infrastructure Signal

Research previews exist because the hard cases are not fully worked out. For a tool like Codex Security, the hard cases are probably not primarily about model capability. Language models can reason about vulnerability patterns, understand data flow, generate targeted payloads, and write plausible patches. The harder problem is building an execution environment that is sandboxed enough to be safe, flexible enough to handle real-world application complexity, and reliable enough to produce consistent validation results.

It is worth noting that OpenAI recently acquired Promptfoo, a red-teaming and adversarial testing platform for AI-powered applications. The pattern is consistent: OpenAI is building a security layer that spans traditional application code vulnerabilities through Codex Security, and AI-specific attack surfaces through Promptfoo’s adversarial probing infrastructure. A closed-loop system for code vulnerabilities pairs naturally with tooling for testing model behavior under adversarial conditions.

What to Watch

As the research preview produces results, the most informative signal will be which vulnerability classes are demonstrated first. Injection flaws and secrets are the easiest showcase, and demonstrating them first indicates the infrastructure is still being built out for the harder dynamic cases. A demonstrated race condition, confirmed by concurrently executing requests and observing inconsistent state, would indicate that the execution environment is meaningfully more capable.

The secondary question is integration story. Dynamic validation requires a running application, which means the tool needs to understand how to build and start your specific application. The depth of that understanding determines the coverage. A system that works well on standard web application templates and struggles with unusual build systems or complex dependency graphs is useful, but not transformative.

The direction is correct. Security tooling that treats findings as things to be proved rather than inferred, and that closes the loop through automated remediation, is where this category needs to go. Whether the execution infrastructure catches up to the model capability quickly enough to deliver on the research preview’s implicit promises is the question the preview phase will answer.