Signed, Trusted, Exploited: How Kernel Anti-Cheat Became a BYOVD Vector

The debate around kernel-mode anti-cheat tends to focus on whether users should trust game companies with ring 0 access. That framing, while not wrong, skips past the more interesting technical story: why the escalation to kernel mode was architecturally inevitable, what specific capabilities it enables, and what the security consequences of deploying ring-0 drivers at consumer scale have turned out to be.

This deep dive into how kernel anti-cheats work covers the privilege model well. The short version: x86 processors implement privilege rings, of which Windows uses two. Ring 3 (user mode) is where applications run; ring 0 (kernel mode) is where the OS kernel, hardware drivers, and the hardware abstraction layer live. Code in ring 3 cannot read another process’s memory without an OS-granted handle, cannot intercept system calls, and cannot hide itself from other user-mode observers. Code in ring 0 has unrestricted access to all physical memory, all CPU state, and every running process’s address space.

Traditional anti-cheat ran entirely in ring 3, using ReadProcessMemory and OpenProcess to inspect the game process. This worked until cheats started operating from ring 0. A user-mode scanner cannot detect or stop a kernel-mode cheat; ring 0 dominates ring 3 completely. The kernel-mode anti-cheat was the direct, logical response.

What Ring 0 Access Actually Enables

The detection capabilities available to a kernel driver are substantially broader than anything possible from user mode. Windows exposes a set of kernel callbacks that drivers can register to receive system-wide notifications. PsSetLoadImageNotifyRoutine fires every time any executable or DLL is mapped into memory anywhere on the system, not just within the game process. This catches DLL injection techniques that target other processes. PsSetCreateProcessNotifyRoutineEx notifies the driver of every process creation and termination system-wide.

More significant is ObRegisterCallbacks, which lets a driver intercept and modify object access operations. Anti-cheat drivers use this to strip PROCESS_VM_READ rights from any handle another process tries to open against the game. The practical effect is that an external cheat calling ReadProcessMemory on the game process will receive an access-denied error even with administrator privileges, because the handle was downgraded by the anti-cheat callback before the read reached game memory.

DKOM (Direct Kernel Object Manipulation) detection is another kernel-only capability. The Windows kernel maintains a doubly-linked list of active EPROCESS structures, accessible via PsActiveProcessHead. A cheat driver can unlink itself from this list, making it invisible to tools that enumerate processes by walking the list. Anti-cheat drivers detect hidden processes by cross-referencing multiple enumeration paths: the EPROCESS linked list, the handle table, and CPU idle thread links. A process visible in one path but absent from others is a strong DKOM indicator.

Hypervisor detection also requires kernel context for reliable timing measurements. Sophisticated cheats have used Intel VT-x or AMD-V to run the game inside a virtual machine they control, operating cheat code from VMX root mode, a privilege level above the kernel. The anti-cheat driver, running inside the guest VM, has no visibility into the hypervisor above it. The defense relies on timing anomalies: VM exits add measurable latency to privileged instructions like RDTSC and CPUID. Kernel drivers run tight timing loops looking for the variance signature that indicates virtualization, and check CPUID leaf 0x40000000 for hypervisor presence flags.

The BYOVD Problem

Windows Driver Signing Enforcement (DSE) requires kernel drivers to carry a valid digital signature from Microsoft or an approved cross-signer. On a standard Windows 10 or 11 installation with Secure Boot active, unsigned code cannot load into ring 0. Cheats circumvented this constraint with BYOVD: Bring Your Own Vulnerable Driver.

The premise is that if you cannot get an unsigned driver into the kernel, you find a signed, legitimate driver with an exploitable vulnerability and use it to execute your code in ring 0. The LOLDRIVERS project catalogs signed drivers with known exploitable vulnerabilities; the list runs to hundreds of entries. Many of the most powerful entries are anti-cheat and hardware monitoring drivers, because these routinely expose IOCTLs that allow ring-0 operations from user mode. A user-mode process calls DeviceIoControl on the vulnerable driver, which executes the requested operation in kernel context, effectively acting as a ring-0 proxy for unsigned code.

The Capcom driver became the canonical example. Shipped with Resident Evil 7 and Devil May Cry 5, it deliberately included an IOCTL to execute arbitrary code in kernel mode, intended to patch game memory at runtime for DRM purposes. It was WHQL-signed by Microsoft. Within months of public disclosure, it was one of the most widely used kernel exploit primitives in the cheat community, because it offered a no-vulnerability-required path to ring-0 execution from any user-mode process.

BattlEye’s own driver has appeared in BYOVD chains. Since it is a widely distributed, properly signed driver, attackers have exploited its vulnerabilities to reach ring 0 without needing their own signed code, using the anti-cheat itself as a trust anchor. The irony is structural: the more widely distributed a signed driver is, the more valuable it becomes as a BYOVD vehicle if a vulnerability is found, because defenders cannot simply revoke or block it without breaking the product it ships with.

The most consequential case is mhyprot2. Genshin Impact ships a kernel anti-cheat driver, mhyprot2.sys, that remained installed on systems even after the game was uninstalled. Security researchers found it exposed IOCTLs allowing any unprivileged user-mode process to read and write arbitrary kernel memory. In 2022, Trend Micro documented ransomware operators from the BlackCat/ALPHV group using this driver to terminate endpoint detection and response processes before deploying their payload. The anti-cheat driver, left behind on a system that no longer ran the game, had become trusted infrastructure for disabling the security software that would have detected the ransomware.

This outcome is a structural consequence of the design, not a freak incident. A kernel driver with a memory-access IOCTL is a powerful primitive. Distribute it to tens of millions of systems, leave it installed after uninstall, and the probability that someone builds an exploit on top of it approaches certainty over time. Any kernel anti-cheat driver with a broad IOCTL interface and poor uninstall hygiene carries the same exposure profile.

What HVCI Closes and What It Does Not

Riot’s Vanguard, which loads its driver at system boot rather than game launch to catch cheats that establish themselves during early boot, now requires Windows 11 with Secure Boot and TPM 2.0. More significantly, it requires Hypervisor-Protected Code Integrity (HVCI) on many configurations.

HVCI uses the hardware hypervisor to enforce code integrity above ring 0. Even if code running in ring 0 attempts to execute unsigned code, the hypervisor’s Extended Page Table (EPT) mappings block it. This closes the standard BYOVD code execution path: a BYOVD attack can still read and write memory through a vulnerable IOCTL, but it cannot execute arbitrary unsigned ring-0 code on a system with HVCI active. That is a meaningful reduction in the exploitable surface, representing real progress on one of the primary bypass techniques.

HVCI does not address DMA attacks, which operate below the OS entirely.

The DMA Hardware Frontier

The ceiling above kernel anti-cheat is not ring 0. Commercial FPGA boards and dedicated DMA cards, such as the Screamer series of PCIe devices, can read a host computer’s physical memory over the PCIe bus from a second machine or external device. The host CPU never executes any cheat code. There is no process for the anti-cheat to detect, no driver to sign, no memory region to scan from the game PC’s operating system, because the memory read happens entirely in PCIe hardware, below the OS visibility boundary.

The architectural defense against DMA attacks is the IOMMU: Intel VT-d and AMD-Vi. An IOMMU allows the OS or hypervisor to restrict which physical memory regions specific PCIe devices can access via DMA. A correctly configured IOMMU can mark game process pages as inaccessible to unauthorized external DMA, preventing a PCIe-attached device from reading them. No shipping anti-cheat currently implements full IOMMU-based memory protection for this purpose.

The engineering obstacle is that IOMMU domain configuration interacts with GPU drivers, NVMe controllers, and other legitimate DMA-capable hardware throughout the system. Isolating game process memory within a dedicated IOMMU domain without disrupting GPU frame rendering or storage I/O is a non-trivial systems problem. The technical path is clear; the implementation has not shipped, and the DMA approach remains largely unaddressed by current production anti-cheat systems.

The Shape of the Trade-off

Each generation of anti-cheat escalates to match the privilege level of the cheats it needs to detect: user-mode inspection, kernel callbacks, boot-time driver loading, hypervisor enforcement, with hardware DMA as the current unaddressed frontier. The logic at each step is coherent.

The security cost is that each escalation ships more powerful kernel drivers to more consumer machines, often without clear uninstall guarantees. The mhyprot2 incident makes the systemic risk concrete: a driver designed to protect game integrity became ransomware infrastructure on systems that no longer ran the game. HVCI meaningfully narrows the exploitable surface, but the underlying condition, that kernel anti-cheat requires trusting closed-source ring-0 code at consumer scale without standardized auditing or uninstall guarantees, does not resolve cleanly. The architecture makes sense for the problem it solves; the security externalities are distributed to every user who ever installed the game.