Your Idle Mac as a Private Inference Node: What Darkbloom Is Actually Building

The pitch for Darkbloom is compact: run LLM inference privately, distributed across Macs that are sitting idle. That sentence covers a lot of ground, and each word in it is doing real work. The interesting part is not just that it runs on Macs, but that Apple Silicon’s memory architecture makes Macs unusually well-suited to this specific problem in ways that NVIDIA consumer hardware is not.

Why Apple Silicon Changes the Equation

The dominant constraint in local LLM inference is memory bandwidth, not raw compute. A 70B parameter model in 4-bit quantization occupies roughly 35-40GB of memory. On a standard workstation GPU, a GeForce RTX 4090 tops out at 24GB of VRAM. You simply cannot fit the model. Inference requires loading weights into the computation unit, and if the model doesn’t fit, you’re either splitting across GPUs (expensive), using slower CPU offloading, or dropping to a smaller model.

Apple Silicon sidesteps this entirely through unified memory. The M2 Ultra in a Mac Studio ships with up to 192GB of memory shared between the CPU, GPU, and Neural Engine. A 70B model fits with room to spare, and the memory bandwidth on the M2 Ultra is around 800 GB/s. That’s not as high as a datacenter A100 (2 TB/s), but it’s dramatically better than what consumer NVIDIA hardware offers relative to its memory capacity. An M4 Max with 128GB of unified memory and 546 GB/s of bandwidth can run a 70B model with respectable token throughput.

The Apple Neural Engine (ANE) adds another dimension. On the M4, it delivers approximately 38 TOPS (tera-operations per second), specifically designed for the matrix multiplication patterns that dominate transformer inference. Frameworks like Apple’s MLX target this hardware directly, and benchmarks consistently show that Apple Silicon inference performance per dollar compares favorably to cloud GPU rentals for many use cases, particularly for models in the 7B to 70B range.

So the hardware premise is sound. There are millions of M1, M2, M3, and M4 Macs sitting mostly idle, attached to power and network, with enough memory and bandwidth to run serious inference workloads.

Distributed Inference: Splitting the Model

Where it gets architecturally interesting is the distributed side. Running a 7B model on a single Mac is already handled by Ollama and llama.cpp. Darkbloom’s approach, targeting idle machines and private inference as a network, implies splitting larger models across multiple devices.

There are two main strategies for distributed inference: tensor parallelism and pipeline parallelism.

In tensor parallelism, each transformer layer is sharded across devices. Every device participates in computing each layer simultaneously, which requires constant inter-device communication. This works well when devices are connected by high-bandwidth interconnects (NVLink, for example), but performs poorly over standard Ethernet because the all-reduce synchronization steps between layers introduce latency at every transformer block.

Pipeline parallelism assigns different layers to different devices. Device A holds layers 0-10, device B holds layers 11-20, and so on. A request travels through the pipeline sequentially. The communication cost is lower, because you only pass activations between adjacent pipeline stages rather than broadcasting across all devices at every layer. The tradeoff is that pipeline parallelism introduces bubbles (idle time while waiting for the previous stage to finish) and increases per-request latency compared to running the full model on a single device.

For a network of consumer Macs connected over LAN or even the internet, pipeline parallelism is the only viable approach. Projects like Petals, from the BigScience workshop, pioneered this pattern for running models like BLOOM-176B across volunteer GPU machines. The Petals architecture uses HTTP/2 streaming to pass hidden states between nodes, with the client holding the embedding and final projection layers locally and farming out transformer blocks to remote peers.

A 70B Llama model has 80 transformer layers. Distributing across four Macs means roughly 20 layers per machine. Each layer in LLaMA-3 70B has a hidden size of 8192 dimensions. The activations passed between pipeline stages are tensors of shape [batch_size, sequence_length, 8192], typically in bfloat16. For a batch size of 1 and a 512-token sequence, that’s 1 * 512 * 8192 * 2 bytes = ~8MB per inter-stage transfer. At gigabit Ethernet speeds, that’s under a hundred milliseconds of transfer time per stage, which adds up to noticeable latency over a full inference pass but stays within the range of usability for non-streaming applications.

What “Private” Actually Means Here

Privacy in this context is primarily about data sovereignty. When you send a prompt to OpenAI or Anthropic, that text leaves your machine, passes through their infrastructure, and is subject to their data retention and logging policies. For many users and organizations, that’s acceptable. For others, it’s a hard constraint, whether due to compliance requirements (HIPAA, GDPR, attorney-client privilege), sensitivity of the work (proprietary codebases, unreleased financial data), or simple preference.

Local inference solves this completely. The prompt never leaves the machine. Distributed inference across machines you control, on your own network, preserves this guarantee for the network as a whole, even though individual nodes only see partial activations rather than the full context.

That last point is worth examining. In pipeline parallelism, intermediate nodes receive the hidden state tensor, not the original prompt text. The hidden states are high-dimensional continuous vectors that have been transformed through multiple layers of attention and feedforward operations. While it’s theoretically possible to reconstruct some information from hidden states, it requires significant effort and access to the model weights. For a network of trusted machines (your own devices, or devices within a trusted organization), this is a reasonable privacy model. For a public volunteer network, it requires more careful analysis.

There’s active research on this: confidential computing approaches using trusted execution environments (TEEs) can provide stronger guarantees, and homomorphic encryption for ML inference is an ongoing research area, though the performance overhead currently makes it impractical for transformer-scale models.

The Idle Scheduling Problem

The “idle Macs” framing connects this to a long lineage of volunteer computing projects. BOINC, Folding@home, and SETI@home all established the pattern: detect when a machine is not in use and contribute spare cycles to a shared computation. macOS provides the mechanisms for this through IOPMLib power management APIs, which expose idle timers, display sleep state, and battery status. A well-behaved background process can check these conditions and throttle or suspend inference work when the user returns.

The challenge is that LLM inference has different scheduling characteristics than scientific simulations. A BOINC work unit can be checkpointed and resumed with minimal overhead. A pipeline inference request cannot be paused mid-request without disrupting the entire chain of nodes processing that request. This means the idle detection logic needs to work at request granularity rather than at arbitrary preemption points, which constrains how responsive the system can be to the user reclaiming their machine.

System-level background task support on macOS, through BGProcessingTask in newer macOS versions, handles some of this automatically, but it’s designed for tasks that can tolerate being deferred, not for tasks that need to be reliably present in a real-time inference pipeline.

The Competitive Landscape

Darkbloom sits at the intersection of several existing approaches without being exactly like any of them.

Ollama handles local inference on a single Mac extremely well. It wraps llama.cpp with a clean API, supports model management, and has solid Metal acceleration. It doesn’t distribute across machines.

Petals distributes inference but targets NVIDIA hardware and a public volunteer network model. It is not Mac-native and has no specific privacy guarantees beyond what the distributed architecture provides.

LM Studio provides a local inference GUI with good Apple Silicon support, again single-machine.

ExoLabs’ exo project is perhaps the closest analog in the open-source space: it explicitly targets running large models across clusters of Apple devices, using a ring topology for distributed inference. Exo supports iPhone and iPad as inference nodes alongside Macs, which is a notable architectural choice given that iPhones with A17/A18 chips also have Neural Engines capable of running smaller models.

Darkbloom’s differentiation, based on the privacy framing and the focus on idle time utilization, suggests it is building toward a deployment model somewhere between personal device clusters and small organizational networks, rather than a public volunteer compute grid.

The Practical Appeal

For a developer or small team with several Apple Silicon Macs, this kind of setup is genuinely useful. A Mac Studio serving as a primary inference node, supplemented by MacBook Pros that are docked and idle overnight, can provide meaningful collective capacity. A four-node cluster of M2 Pro MacBook Pros each with 32GB unified memory gives you 128GB of distributed addressable memory, enough to run a full 70B model with pipeline parallelism while keeping everything within a local network.

The cost model is also attractive. The inference happens on hardware you already own, on power you’re already paying for, without per-token API costs. For high-volume use cases in a team environment, that economics argument compounds quickly.

The engineering challenge, making pipeline parallelism across heterogeneous Mac hardware reliable, low-latency, and transparent to the application layer, is nontrivial. But the hardware substrate is genuinely excellent for the task, and the privacy angle addresses a real constraint that cloud inference cannot solve by definition.