The Layout Oracle Pattern: How River Compositor Recovered X11's Best Accident

When X11 was designed, the separation between the X server and window manager was not intentional modularity. It was just how things worked: the X server handled rendering and input, and anything wanting to manage windows had to race to set SubstructureRedirectMask on the root window. Whoever got there first became the window manager. This accidental architecture produced decades of window manager diversity — i3, dwm, bspwm, xmonad, awesome, herbstluftwm — each a standalone process with its own personality.

Wayland collapsed that separation. In Wayland, the compositor owns everything: rendering, input, and window management. This was deliberate. X11’s model was leaky, with window managers doing things like patching server memory, relying on undocumented behaviors, and fighting over shared global state. Wayland compositors are sovereign over their windows, which makes the security model cleaner and eliminates an entire class of race conditions and protocol ambiguity.

But it also meant that to get a different tiling layout, you no longer swap out your window manager. You fork the compositor, or you use whatever layouts your compositor ships with, or you wait for the upstream project to accept your PR. The Wayland ecosystem started converging on a handful of compositors — sway, KWin, Mutter, labwc — each bundling its own layout opinions.

River, a Wayland compositor written in Zig by Isaac Freund, pushes back against this consolidation through a different architectural decision described in his recent blog post: it delegates window layout to external processes through a well-defined Wayland protocol extension. The compositor stays sovereign over where windows actually end up, but it asks an external oracle to decide where they should go.

The river-layout-v3 Protocol

River doesn’t compute tile positions internally. Instead, it publishes a Wayland global called river_layout_manager_v3. Any client process can bind to this global and register as a layout generator for a given output. When River needs to know where to place windows, it fires a layout_demand event at the layout generator. The layout generator does the math, responds with positions, and River applies them.

The protocol is request-response, keyed on serials to handle concurrent updates without races. A layout demand carries the window count, the usable output area, and a serial:

layout_demand(view_count=3, usable_width=1920, usable_height=1080, serial=42)

The layout generator responds by pushing dimensions for each view in order, then committing with the matching serial:

push_view_dimensions(x=0, y=0, width=960, height=1080, serial=42)
push_view_dimensions(x=960, y=0, width=960, height=540, serial=42)
push_view_dimensions(x=960, y=540, width=960, height=540, serial=42)
commit(serial=42)

River applies those positions. The layout generator can maintain whatever internal state it wants between demands: ratios, main window counts, padding values, or even a full tree of split nodes. River doesn’t care. It just needs positions back before the next frame.

The built-in layout generator, rivertile, ships as a separate binary — not as an internal module. It implements a classic master-stack layout, configurable via command-line flags:

rivertile -view-padding 6 -outer-padding 6 -main-ratio 0.6 -main-location left

You run rivertile as a background process in your init script. River discovers it over the Wayland socket, and from that point on, rivertile decides where your windows go. To change layout behavior, you replace rivertile with something else, or write your own generator in any language that can speak Wayland.

What This Enables

The ecosystem that has emerged around river-layout-v3 looks a lot like the X11 window manager ecosystem, but built on a sound protocol foundation. river-luatile lets you script layouts in Lua. stacktile implements a flexible stacking layout with extensive configuration options. Several paper-WM-style generators exist, treating the screen as a horizontal strip of scrollable windows rather than a fixed tiled grid.

Writing a custom layout generator is within reach for any programmer comfortable with a Wayland client library. In C with libwayland-client, the structure is straightforward: bind to the river_layout_manager_v3 global, create a river_layout_v3 object per output, and implement the layout_demand listener:

static void layout_demand(void *data,
    struct river_layout_v3 *layout,
    uint32_t view_count,
    uint32_t usable_width,
    uint32_t usable_height,
    uint32_t serial)
{
    /* compute positions */
    for (uint32_t i = 0; i < view_count; i++) {
        river_layout_v3_push_view_dimensions(layout,
            x[i], y[i], w[i], h[i], serial);
    }
    river_layout_v3_commit(layout, "my-layout", serial);
}

The generator is just a process that speaks a documented protocol. The compositor and the layout engine are fully decoupled at the process boundary, which means crashes in the layout generator don’t crash the compositor, and you can hot-reload layout logic by restarting the generator process without disturbing the running session.

Comparing the Alternatives

Sway takes the opposite approach. As a Wayland reimplementation of i3, it ships a complete tree-based layout engine internally. You get horizontal splits, vertical splits, stacking, and tabbed layouts. Extending the layout system means modifying sway’s source code. The control surface is deliberately minimal: you can write sway IPC scripts to move windows or trigger layout changes, but the actual tiling math lives inside the compositor binary and is not exposed as an external API.

KWin, KDE’s compositor, provides a JavaScript scripting API that can respond to window events and manipulate window geometry. This is more flexible than sway’s model, but the scripts run inside the compositor process, not outside it. A buggy KWin script can degrade compositor performance. The API is also compositor-internal: KWin can change it between releases without breaking anything external, and scripts targeting KWin don’t run anywhere else.

Mutter, used by GNOME, has a plugin architecture primarily used by GNOME Shell itself. Third-party window management customization largely happens through GNOME Shell extensions, which again run inside the compositor process and require coordination with GNOME Shell’s own state.

River’s model is the most principled separation: the protocol is documented at the Wayland XML level, the generator runs in a separate process with its own lifecycle, and any compositor that implements river-layout-v3 can share the same ecosystem of layout generators. The protocol boundary also makes it easy to reason about what state belongs where. River owns the authoritative window positions; the generator owns the layout algorithm and its parameters.

The Tags System and Layout Interaction

River also inherits dwm’s tag-based window organization, which interacts meaningfully with the layout protocol. Each window has a bitmask of tags, and the compositor displays all windows whose tags intersect with the currently active tag bitmask. You can view tag 1 alone, view tags 1 and 3 simultaneously, or assign a window to tags 2 and 5 so it appears in both contexts without duplication.

The key interaction with layout: the view_count passed to the layout generator reflects only the windows currently visible under the active tag filter. When you switch from a five-window view to a two-window view by changing your tag selection, the layout generator automatically receives a new demand with the correct count. The generator doesn’t need to know about tags at all; it just handles whatever count River sends. This keeps the layout protocol simple and composable with the windowing model.

The Broader Implication

What this architecture demonstrates is that the Wayland security model and the X11 ecosystem model are not actually in conflict. The compositor can remain sovereign over window positions, with no client able to override placement, while still accepting input from an external layout oracle through a controlled, typed protocol.

This is the same pattern that makes plugin architectures work in any software system: you keep the core invariants inside a trusted boundary, but you expose a well-defined interface through which external code can influence behavior. The difference between Wayland’s compositing model and X11’s window manager model is not that external influence is impossible in Wayland; it is that in Wayland, that influence flows through a deliberate protocol rather than through shared global server state and a race to claim a root window property.

The river-layout-v3 protocol specification is compact enough to read in an afternoon. Building a layout generator is a reasonable weekend project. The resulting generator can be shared across any compositor that chooses to implement the protocol, creating exactly the kind of portable, composable ecosystem that X11 cultivated by accident and that most Wayland compositors have not tried to replicate.

River did not recover X11’s architecture. It recovered X11’s best outcome through a considerably cleaner design.