When Hugging Face’s TRL team shipped delta weight sync, they collapsed per-step weight transfer from gigabytes to megabytes by exploiting a counterintuitive fact: in reinforcement learning, most model weights don’t actually change between optimizer steps. Not approximately unchanged, but bit-for-bit identical in their BFloat16 representation. The compression is not clever encoding or quantization. It falls directly out of how BFloat16 arithmetic works at the learning rates RL uses.
The practical implication is that distributed RL training no longer requires colocated infrastructure. Trainer and inference can live in different regions, connected only through object storage, because there are very few bytes to move. But the interesting part is why this works at all.
The Visibility Threshold
BFloat16 has 7 mantissa bits. Between any two consecutive powers of two, there are exactly 128 representable values. The spacing between adjacent BFloat16 numbers near some weight value w is roughly |w| * 2^-7, or about |w| / 128. An update gets absorbed by the float cast whenever it sits below half that spacing: when |Δw| < |w| / 256.
This is what the PULSE paper calls the BFloat16 visibility threshold. Updates smaller than this threshold do not survive the round trip through BFloat16 casting. The byte representation stays identical, even though the optimizer computed a nonzero gradient.
Now consider what Adam does at a typical RL learning rate of 3e-6. The update to a single weight is:
Δw = -η * (m̂ / (√v̂ + ε))The normalized gradient term m̂ / (√v̂ + ε) is order one, so the actual update magnitude is approximately equal to the learning rate: |Δw| ≈ η ≈ 3e-6.
Most weights in a language model sit between 0.01 and 0.1. At that scale, the visibility threshold |w| / 256 lands around 4e-5 to 4e-4. The update is an order of magnitude smaller than the threshold. The optimizer whispers; BFloat16 cannot hear.
Multiply this across hundreds of millions of parameters and you get the central result: at RL learning rates, roughly 99% of weights are bit-identical between consecutive steps. The PULSE authors measured this empirically across Qwen2.5 (0.5B, 1.5B, 7B), Llama-3.2-3B, and Gemma-3-4B, consistently finding mean per-step sparsity around 99% with standard deviation under 0.4%. The worst-case step stayed above 98%.
This is not a statistical claim that needs verification on your workload. It is what the arithmetic guarantees for any optimizer operating in this learning rate regime.
Why Prediction Failed
The obvious next question is whether you can predict which weights will change, using the optimizer’s internal state. If Adam tracks first and second moment estimates m and v, and you know the visibility threshold for each weight, you should be able to compute a boolean mask before the step happens.
The TRL team tried this. Recall was 30%. The analytical threshold is not tight enough; Adam’s normalization introduces enough nonlinearity that the simple formula misses two thirds of actual changes.
So they fell back to ground truth: snapshot the weights in BFloat16 before the optimizer step, snapshot again after, compare bytes. This costs one CPU copy of the model, which is acceptable overhead for training. The mask is exact because it reflects what actually happened, not what the formula predicts.
The detector is conceptually simple:
class BF16ChangeDetector:
def _pre_step_hook(self, optimizer, args, kwargs):
for param in self._params:
self._pre_step_bf16[name(param)] = param.detach().to(torch.bfloat16).cpu().clone()
def _post_step_hook(self, optimizer, args, kwargs):
for param in self._params:
self._validated_masks[name(param)] = (
param.detach().to(torch.bfloat16).cpu() != self._pre_step_bf16[name(param)]
)This hooks into the optimizer’s step boundary, snapshots before and after, and produces a per-parameter boolean tensor. The rest of the system encodes those masks as (indices, values) pairs in safetensors format and uploads them to a Hub bucket.
The Wire Format
Safetensors is the canonical checkpoint format on Hugging Face. It is a simple header with arbitrary metadata followed by zero-copy tensor data. The delta sync protocol extends this with two file types.
Anchors are full checkpoints, written every N steps:
anchors/step_000010.safetensors
model.layers.0.self_attn.q_proj.weight (bf16, full shape)
model.layers.0.self_attn.k_proj.weight (bf16, full shape)
...
metadata:
sparse: false
model_version: 10
sparsity: 0.0Deltas encode only changed elements:
deltas/step_000011.safetensors
model.layers.0.self_attn.q_proj.weight.indices (int32, [num_changed])
model.layers.0.self_attn.q_proj.weight.values (bf16, [num_changed])
...
metadata:
sparse: true
model_version: 11
sparsity: 0.9938
changed_params: ["model.layers.0.self_attn.q_proj.weight", ...]Each changed parameter gets two tensors: a flat index array and a value array at those indices. The receiver downloads the file, reads the sparse flag from metadata, and branches. For anchors, it loads every tensor and snapshots for future deltas. For deltas, it applies (indices, values) to the snapshot and hands reconstructed full tensors to the inference engine.
This is where the architecture becomes interesting. The trainer and inference engine never talk to each other about weights. They exchange a single HTTP POST containing {"repo_id": "...", "filename": "..."}, and that is the entire control plane. Actual bytes move between each side and the bucket, in parallel.
The Hub’s bucket storage is backed by Xet, a content-defined chunking layer that deduplicates at the chunk level. Even if you upload full anchors, Xet only transfers changed chunks. Stack sparse encoding on top and you pay for what moved, once.
What This Unlocks at Scale
The numbers in the original post are for Qwen3-0.6B: per-step payload drops from 1.2 GB to 20-35 MB. The interesting question is what happens at frontier scale.
Take Llama-3.1-405B. In BFloat16 that is 810 GB on disk. At 99% sparsity, the delta is 1% of parameters. PULSE’s measured encoding on a 7B model achieved roughly 130x reduction. Scaled linearly to 405B, the delta lands around 6 GB per step.
Assume a generous 100 GB/s NCCL broadcast bandwidth inside a cluster. A full weight sync takes 810 GB / 100 GB/s ≈ 8 seconds of paused inference, every step. With delta sync, the trainer streams 6 GB to the bucket in the background while generation continues. The rollout server’s actual paused window is just the apply step, around 2 seconds.
Even inside the cluster, delta sync cuts visible pause by 4x and bytes on the wire by 130x. Outside the cluster, NCCL does not work at all. Once your rollout fleet spans regions or clouds, bucket-based weight transfer stops being an optimization and becomes the only viable architecture.
Fireworks put a similar number on this in their post on frontier RL costs: for a 1TB checkpoint at fp8, conventional wisdom says you ship 1024 GiB per step. Their measured delta averaged 20.3 GiB, about 2% of the full model. Cursor’s Composer 2 report describes the same architecture: training and inference in different regions, stitched together with a shared S3 bucket, uploading compressed weight diffs every training step.
Both reached the same conclusion: most weights do not change, sending only deltas collapses bandwidth by two orders of magnitude, and routing those deltas through object storage removes the requirement for shared infrastructure.
The Three Box Architecture
The disaggregated setup has three components and one shared substrate:
- Trainer. Owns model weights, runs optimizer, emits sparse deltas. Can be anywhere: one GPU, eight GPUs, a laptop.
- HF Bucket. A single repo with two prefixes:
anchors/for occasional full snapshots,deltas/for sparse patches. This is the only thing both sides reference. - vLLM rollout server. Pulls from bucket, applies delta, serves rollouts. Does not need to be colocated with the trainer.
The vLLM side is a 30-line extension implementing WeightTransferEngine. The core logic:
def receive_weights(self, update_info, load_weights):
download_bucket_files(update_info.repo_id, files=[(update_info.filename, local_path)])
with safe_open(local_path, framework="pt", device="cpu") as f:
meta = PatchMetadata.from_metadata_dict(f.metadata())
if not meta.sparse:
for name in f.keys():
tensor = f.get_tensor(name)
self._bf16_snapshot[name] = tensor.clone()
load_weights([(name, tensor)])
else:
for name in json.loads(meta.changed_params):
indices = f.get_tensor(f"{name}.indices").long()
values = f.get_tensor(f"{name}.values")
snap = self._bf16_snapshot[name].flatten()
snap[indices] = values
self._bf16_snapshot[name] = snap.reshape(self._bf16_snapshot[name].shape)
load_weights([(name, self._bf16_snapshot[name])])This registers via vLLM’s --worker-extension-cls flag. No fork required; install TRL alongside vLLM and point the CLI at the extension class.
The full flow has four phases:
- Upload while inference runs. Trainer encodes delta and pushes to bucket. vLLM still serves old policy.
- Pause vLLM. Short HTTP call, hundreds of milliseconds.
- Signal
/update_weights. Send bucket coordinates. vLLM downloads, applies, returns. - Resume.
Log lines from a real run:
Delta: 1234567/200000000 elements changed (sparsity=99.38%)
[delta_engine] uploaded user/wordle-deltas/deltas/step_000042.safetensors (27.4 MB)
Weight sync: done. Total 9.4s (inference paused 1.1s)Inference was paused for 1.1 seconds. The remaining 9.4 seconds were upload time, which happened in parallel with token generation. With NCCL, the full sync time would be pause time.
Running It on Spaces
The TRL team demonstrated this with fully disaggregated infrastructure:
- One box with a GPU running the trainer
- A Hugging Face Space (Docker SDK, L4 GPU) running vLLM with the extension
- A second Space (CPU) running the Wordle environment server
- A Hub bucket connecting them
None of these share a network. The trainer never opens a port. The Space never sees the trainer’s IP. They all talk to the Hub.
The vLLM Space’s Dockerfile:
FROM vllm/vllm-openai:latest
RUN pip install "trl @ git+https://github.com/huggingface/trl.git@delta-weight-sync"
ENV VLLM_SERVER_DEV_MODE=1
EXPOSE 7860
ENTRYPOINT ["vllm", "serve", "Qwen/Qwen3-1.7B", \
"--host", "0.0.0.0", "--port", "7860", \
"--worker-extension-cls", "trl.experimental.async_grpo.delta_engine.DeltaWorkerExtension", \
"--weight-transfer-config", "{\"backend\":\"nccl\"}", \
"--max-model-len", "32768", \
"--gpu-memory-utilization", "0.8"]Training kicks off from anywhere:
python examples/scripts/openenv/async_wordle.py \
--vllm-server-url https://$USER-vllm-wordle-inference.hf.space \
--env-url https://openenv-wordle.hf.space \
--delta-sync-repo-id $USER/wordle-deltas \
--model Qwen/Qwen3-1.7BThis is async RL training without a cluster. One GPU, a Hugging Face account, and object storage.
What Remains
The implementation has known gaps:
- Two CPU BFloat16 snapshots, one on the trainer (for change detection) and one on the rollout server (to reconstruct full tensors). The trainer copy is unavoidable until someone cracks analytical mask prediction. The inference copy goes away when vLLM lands a sparse
load_weightsAPI; there is an open PR adding exactly this. - Fixed anchor cadence. Currently a full snapshot every N steps. An adaptive policy based on cumulative drift would be more efficient.
- Multi-node FSDP2 trainers. The change detector hooks per-process optimizers. It should generalize to FSDP2, but scaling to multi-node has not been measured.
The broader implication is that BFloat16’s arithmetic properties make weight synchronization a fundamentally cheaper operation than the naive byte count suggests. At RL learning rates, the vast majority of weights are invariant between steps, not because of compression or approximation, but because the updates are below the format’s resolution. Sparse encoding is just bookkeeping for what the float format was already doing.
This shifts the bottleneck. Weight transfer is no longer the thing that forces colocated infrastructure. You can run the trainer on one continent, inference on another, and connect them with a bucket.
The code is in TRL PR #5417. The full Wordle example and Space configurations are in the repo. If you have been avoiding distributed RL because of infrastructure complexity, the math just changed.