The Arms Race Below the OS: Kernel Anti-Cheats, DMA Hardware, and Why Software Cannot Win

The standard framing of anti-cheat software treats it as a cat-and-mouse game between cheat developers and game companies. That framing is incomplete. The more precise description is a layered privilege escalation contest, where each side tries to operate at a lower privilege level than the other. s4dbrd’s walkthrough of how kernel anti-cheats work covers the mechanics well; what it leaves for another conversation is why the hardware-level end of the attack spectrum represents a structural defeat for software-only detection, regardless of how deep into the kernel the defender reaches.

The Protection Ring Foundation

Modern x86 CPUs implement four protection rings, numbered 0 through 3, though in practice Windows uses only Ring 0 for the kernel and Ring 3 for user processes. The distinction is enforced in silicon. Ring 3 code cannot execute privileged instructions directly: it cannot map arbitrary physical memory, cannot install interrupt handlers, cannot read hardware-reserved registers. When it tries, the CPU raises a general protection fault before the instruction completes. The process has no say in the matter.

Ring 0 has no such restrictions. Code running in Ring 0 can read and write any physical memory address, can install hooks into the CPU’s interrupt descriptor table, can enumerate every live process and module in the system. This is why kernel drivers can do things that user processes cannot, and it is why the anti-cheat arms race converged on Ring 0 as the target privilege level.

Early anti-cheats ran entirely in Ring 3, alongside the game client. User-mode integrity checks, process enumeration via the Win32 CreateToolhelp32Snapshot API, module list scans. A cheat running in Ring 3 with equivalent privileges could patch those checks in memory, inject DLLs to intercept API calls, or run in a separate process that the anti-cheat never inspected. The asymmetry was structural: the defender had no privileged vantage point from which to see the attacker’s code.

Kernel Drivers: What Moving to Ring 0 Buys

Windows kernel drivers are .sys files that load into Ring 0 through the Windows driver model. Since Windows Vista 64-bit, Kernel Mode Code Signing (KMCS) has required that drivers be signed by a trusted certificate authority before the kernel will load them. In practice this means anti-cheat vendors obtain Extended Validation code signing certificates; cheats either require test-signing mode (which most anti-cheats detect) or exploit signed-but-vulnerable drivers via BYOVD attacks, covered below.

Once a driver is loaded, the detection surface available to it is substantially larger than anything accessible from Ring 3. The Windows kernel exposes callback registration interfaces that user-mode code cannot touch:

// Notified when any process is created or destroyed
PsSetCreateProcessNotifyRoutineEx(callback, FALSE);

// Notified when any thread is created or destroyed
PsSetCreateThreadNotifyRoutine(callback);

// Notified when an image (DLL or EXE) is mapped into any process
PsSetLoadImageNotifyRoutine(callback);

// Register callbacks on object handle operations (process/thread handles)
ObRegisterCallbacks(&registration, &handle);

The ObRegisterCallbacks interface is particularly useful: it fires whenever any process opens a handle to the game process, and can strip access rights from that handle before the caller receives it. A cheat trying to call ReadProcessMemory on the game process gets a handle with PROCESS_VM_READ removed. The anti-cheat’s kernel driver intercepts the open and silently degrades it.

Memory scanning from Ring 0 uses MmCopyMemory to read the game process’s virtual address space directly, bypassing any hooks the cheat might have placed on Ring 3 memory APIs. Driver enumeration walks PsLoadedModuleList, the kernel’s own linked list of loaded kernel modules, to detect unsigned drivers or injected code. File system mini-filter drivers attach to the storage stack and receive notification of every file open, read, and write, which allows detection of cheat files even when those files are present only briefly or renamed.

Hardware fingerprinting is another capability that benefits from Ring 0 access. SMBIOS tables, disk volume serials, network adapter MAC addresses, and CPU identifiers are accessible from the kernel. Anti-cheats combine these into hardware ban identifiers that persist through game reinstallation. Circumventing them requires spoofing identifiers at the driver level, which in turn requires the cheat to load its own kernel driver.

PatchGuard and the Death of SSDT Hooking

Before Windows Vista 64-bit, the dominant technique for kernel-level interception was SSDT hooking: overwriting entries in the System Service Descriptor Table to redirect system calls through attacker-controlled code. Anti-cheats used it, rootkits used it, security software used it. It worked because the kernel had no integrity checking on its own data structures.

Microsoft introduced Kernel Patch Protection (PatchGuard) with 64-bit Windows to make that class of modification impossible. PatchGuard periodically checksums critical kernel data structures, including the SSDT, interrupt descriptor table, and global descriptor table, from an unpredictable execution context. If it detects modification, it immediately triggers a bug check. The check interval is randomized to defeat timing-based attacks. The result is that SSDT hooking on 64-bit Windows is not merely unsupported; it causes system crashes reliably enough that it cannot be used in practice.

This forced a migration. Anti-cheats that previously used SSDT hooks moved to the Microsoft-documented callback interfaces described above. Those interfaces are less flexible but PatchGuard-safe, because they operate through official kernel registration mechanisms rather than by patching kernel memory.

The IOCTL Channel

Kernel drivers do not have a UI. The visible anti-cheat client that runs alongside the game is a Ring 3 process that communicates with the Ring 0 driver through the I/O control interface. A user-mode process opens a handle to the driver’s device object and issues DeviceIoControl calls with operation codes (IOCTLs) and associated buffers. The driver’s IRP_MJ_DEVICE_CONTROL dispatch routine handles these, running in Ring 0 with full kernel privileges on behalf of the user-mode caller.

This interface is a significant attack surface. In 2016, Capcom shipped a driver for their PC game ports that included an IOCTL handler which executed arbitrary caller-supplied shellcode in Ring 0. The intent was to work around KMCS requirements for their game’s internal code, but the effect was a signed, Microsoft-approved driver that any process on the system could use to escalate to Ring 0 with no further authentication. Security researchers documented the vulnerability in detail, and it became a well-known example of why IOCTL handlers require careful input validation.

That incident also illustrates the BYOVD (Bring Your Own Vulnerable Driver) technique. Because KMCS only verifies that a driver is signed, not that it is safe to execute, an attacker who can load a legitimately signed but vulnerable driver gains the kernel access they need. The Microsoft Vulnerable Driver Blocklist exists to address this, but blocklist coverage lags the set of known vulnerable drivers in circulation.

Hypervisor-Level Detection

Kernel drivers run in Ring 0. The CPU also supports a privilege level below Ring 0, used by hypervisors. Intel VT-x and AMD-V extensions allow a hypervisor to run in VMX root mode, managing guest virtual machines where the guest OS runs in VMX non-root mode with Ring 0 semantics but under hypervisor control.

From VMX root mode, a hypervisor can monitor guest memory accesses using Extended Page Tables (EPT). EPT entries can be marked execute-only, allowing the hypervisor to hide its own code from the guest. EPT violation handlers fire on any guest access to a monitored physical page, giving the hypervisor visibility into memory operations the guest performs.

Riot’s Vanguard anti-cheat loads at system startup before most other drivers, giving it early visibility into what loads afterward and making it harder for cheats to establish themselves before Vanguard initializes. The general direction of hypervisor-based anti-cheat is to operate in a context the guest kernel cannot inspect or modify: a cheat running in Ring 0 within the guest is still observable to a hypervisor monitoring EPT accesses. This shifts the trust boundary one level lower.

Where Hardware Attacks Break the Model

All of the above operates on a shared assumption: the attacker’s code runs on the target CPU, inside the trust model that the CPU’s protection ring architecture governs. DMA cheats invalidate that assumption.

Modern PCIe is a peer-to-peer interconnect. A PCIe device, whether a GPU, NVMe drive, or FPGA expansion card, can be programmed to issue DMA transactions that read or write system DRAM without involving the CPU at all. The CPU does not execute instructions for these transfers; it does not schedule them; it does not observe them in any register visible to kernel code. The memory controller handles them directly.

Attackers use FPGA boards such as the Screamer PCIe to implement DMA cheat hardware. The FPGA enumerates as an innocuous PCIe device and continuously reads the game process’s physical memory pages over PCIe DMA. The cheat logic runs on the FPGA itself or on a second machine connected to it, with a separate display and input injection hardware for the overlay. From the perspective of the target machine’s CPU, nothing anomalous has happened. No unusual process exists. No kernel driver has been loaded. No system call has been issued. The memory was simply read from outside.

Kernel anti-cheats are blind to this. PsLoadedModuleList will not show an FPGA. ObRegisterCallbacks will not fire because no process opened a handle to the game. Memory scanning will not find cheat code because the cheat code does not run on this machine. The entire detection surface that Ring 0 access enables is predicated on the attacker operating within the same CPU execution context, and DMA cheats do not.

The IOMMU (Intel VT-d, AMD-Vi) can restrict which physical memory regions a PCIe device is permitted to access via DMA, and is used by Windows for certain security features. But IOMMU enforcement for anti-cheat purposes would require locking the game’s memory pages to specific IOMMU domains that the anti-cheat controls, which is not how current anti-cheat systems are designed. It would also require the IOMMU to be enabled and properly configured across consumer hardware, which is not guaranteed.

The Trust Boundary Is Physical, Not Logical

The progression from user-mode to kernel to hypervisor is a progression of software trust boundaries. Each level can, in principle, verify the integrity of the level above it. The hypervisor can inspect the kernel; the kernel can inspect user processes. But the physical machine sits below all of them, and PCIe DMA operates at the physical machine level.

There is no software component that can audit PCIe DMA transactions in real time with zero overhead, because the DMA controller does not interrupt the CPU to ask permission for each transfer. The closest approaches involve IOMMU auditing with non-trivial overhead on every DMA operation, affecting legitimate hardware too. Deployment on consumer gaming machines is not realistic.

The practical conclusion is that hardware attestation combined with bus topology measurement is the only path toward detecting this class of attack, and neither exists as a deployed anti-cheat mechanism. This is not a gap that better Ring 0 code will close. The attack surface is outside the address space. Any software-only detection system, regardless of privilege level, can only observe what the CPU executes. When the attacker’s code does not run on the CPU, the software detection model does not apply.

There is a separate dimension worth naming. A Ring 0 driver with the access required for effective cheat detection also has the access required to read every credential cached in memory, log every keystroke, and exfiltrate any file on the system. The anti-cheat vendor’s driver is fully trusted by the OS and runs before the game launches, often before the user has done anything at all. The question of whether that trust is warranted is one most users do not examine. Riot, Easy Anti-Cheat, BattlEye, and others have documented what their drivers do and do not do, but the verification burden rests entirely on the user’s willingness to trust a vendor’s self-reporting. That is a real tradeoff, and it compounds with every new game that ships its own kernel driver.