Perplexity Open Sources Apple Chip Inference Engine Lily

Perplexity has open-sourced the local inference engine Lily, specifically designed for the Qwen3.6-35B-A3B on Apple chips, to facilitate hybrid computing for its Computer products, preventing edge computing power from bottlenecking tasks.

Lily operates as a single-process runtime: Rust loads weights and drives the generation loop, with a handwritten Metal kernel executing the model, and the execution path does not include PyTorch or MLX. It provides a minimal OpenAI-compatible chat completion interface, performing only greedy decoding. The code is available in the pplx-garden repository, licensed under Apache-2.0.

The target weights are MLX affine 4bit, with a group size of 64 for Qwen3.6-35B-A3B, approximately 19.4 billion bytes; during loading, it verifies the precise architecture and quantization layout. Dense versions, smaller Qwen, BF16, GGUF, AWQ, GPTQ, int8, and fp8 are not supported. The official benchmark locks the fixed revision number of the mlx-community checkpoint. Hardware requirements include Apple GPU family 10 and above (M5 and newer), with Metal tensor operations requiring a newer macOS, and the Rust version is pinned by the toolchain file.

The company measured an average pre-fill of about 1.23 times and decoding of about 1.35 times relative to MLX-LM on an M5 Max with 40 cores and 128GB of unified memory. With a 4K prompt and 4K decoding context, the pre-fill is about 5749.9 tokens/second and decoding about 186.6 tokens/second, compared to MLX-LM's approximately 4737.5 and 140.9. The maximum pre-fill gain comes from GPU-resident expert routing at about 89% and dequantization integrated into grouped GEMM at about 77.4%; speculative decoding, however, slows down by about 18%. In quality comparisons, perplexity is only 0.04% higher, with the first token consistency rate at about 96.35%.

Qwen3.6-35B-A3B is a sparse mixture of experts: total parameters are about 35 billion, with approximately 3 billion activated per token, and the router selects 8 out of 256 experts at each step, plus shared experts; the structure mixes full attention layers with Gated DeltaNet. Hybrid Compute offloads heavy inference to the cloud, while the local model handles private files and applications. The product side of hybrid computing lists macOS 15+, with a minimum memory of 24GB and 32GB for more stability.

Mechanically, the buyers are Computer users and developers running private tasks on Mac, while the seller is Perplexity, which open-sources the edge engine while keeping cloud inference within its own products. The event is driven by edge-cloud division of labor: weights come from the Alibaba-affiliated Qwen open-source line, and the runtime is specialized by a U.S. company using Metal; the beneficiaries are those needing to reduce local ignition latency on M5-level high-memory machines, while the burden falls on the general framework that handles multiple models, yielding throughput at this checkpoint.

Public information does not provide a complete comparison for non-M5 models, nor does it commit to supporting a second model in the future.

Source: Public Information

Perplexity has not turned Lily into a second MLX, but rather has extracted a production path: it only serves that one edge checkpoint within Computer. General frameworks rely on coverage to create ecosystems, while specialized engines strip away branches to gain bandwidth. The repository simultaneously contains an RDMA communication library and a tokenizer, indicating that the open-source strategy is "inference stack components," not merely renaming a Q&A product.

The capital path is to provide a free edge engine for Macs to enable private file capabilities, while still charging for Pro/Max subscriptions in the cloud. The motivation is to ensure that local computation no longer becomes a bottleneck in the Computer workflow, while leveraging Qwen's open-source weights to avoid training a large edge model. Resource mobilization involves Rust plus handwritten Metal, integrating expert routing and dequantization, rather than adding another layer of Python.

Analogous examples include llama.cpp for a single quantization format, Apple's MLX for a general layer, and various companies using open-source models as local workers while employing closed-source models as cloud foremen in a hybrid architecture. In terms of industry positioning, edge inference has transitioned from "being able to run" to "writing the kernel specifically for a certain MoE"; the division of labor between China and the U.S. sees Chinese labs producing sparse weights, while U.S. application companies provide chip-specialized runtimes.

Structural judgment pertains to technical substitution: pricing power has shifted from "whose framework supports more models" to "who can run this checkpoint near the bandwidth wall." The mechanism is that hybrid computing keeps private files on the local machine while leaving long inference to the cloud; the narrower the engine, the faster the local ignition, preventing the cloud from being dragged into synchronous blocking by the edge.

ABAB News · Cognitive Laws

Engines that only serve one model are often faster than those that serve many.
The runtime is open-sourced, while the subscription still sells cloud access.
Once the edge-cloud division of labor is established, local latency becomes a product threshold.

ABAB AI Insight