📡 Local AI Week — March 15–21, 2026

The Week in Local AI

March 15–21, 2026 was one of the most consequential weeks in the short history of consumer and self-hosted AI. Three stories dominated the discourse and each independently would have been newsworthy on its own. Together, they signal a fundamental shift: frontier-grade inference is becoming a consumer-accessible proposition.

The REAP Revolution

The most technically dense story of the week was REAP — Routing Expert Activation Pruning — a compression technique for Mixture-of-Experts (MoE) models developed by @0xSero and applied to NVIDIA's Nemotron-3-Super-120B.

To understand why REAP matters, you need to understand MoE inference at scale. A MoE model like Nemotron-Super has 120 billion total parameters, but only activates a fraction of them per token — perhaps 20-30 billion parameters are "active" in a given forward pass, with the rest sitting in GPU memory unused. At 60GB, Nemotron-Super fits on two A100-80GB or H100-80GB GPUs — just barely. But serving it efficiently requires the routing logic to constantly move expert weights into the active GPU memory, which creates latency and bandwidth pressure.

REAP addresses this by profiling which experts are actually used in practice across a representative workload, pruning low-activation experts entirely, and reordering the remaining experts to minimize memory fragmentation. The results @0xSero reported this week:

• 2× throughput over baseline on comparable hardware (A100 dual-GPU setup)
• 60GB VRAM target achieved — enabling exact dual-GPU 80GB cards without model parallelism tricks
• INT4 quantization integration incoming, targeting ~35GB for the full 120B model
• vLLM UVA (Unified Virtual Addressing) compatibility — meaning REAP-compressed models can be served directly through vLLM's production-grade serving stack
• reap-ex optimization brings an additional ~20% speedup through expert execution reordering

The Kimi K2 connection generated additional excitement: @0xSero also demonstrated REAP applied to Moonshot AI's Kimi K2 model, pruning it by 50% with minimal degradation on standard benchmarks. The fact that Kimi K2 — a heavily used frontier model — could be cut in half while maintaining quality opens the door to serving it on significantly cheaper hardware.

The broader architectural implication: as MoE models become the dominant paradigm for large frontier models (both for training efficiency and inference sparsity), compression techniques like REAP become critical infrastructure. Expert-level granularity gives compressors more choices than weight-level pruning — you can identify and remove entire experts rather than individual weights, with cleaner quality tradeoffs.

Frontier Models on Your Desk

The story that generated the most social media reaction this week was the successful demonstration of Qwen3.5-397B running at 3.4 tokens per second on a MacBook M3 Max. This is not a benchmark cherry-pick — it is a sustained generation rate for a model that, at 209 gigabytes of weights in BF16, represents the largest publicly available open-weight model and is approximately GPT-4-class in capability.

The technical approach is SSD streaming with expert prefetching. Qwen3.5-397B is a Mixture-of-Experts model — at any given forward pass, it activates only a small subset of its experts. Instead of requiring all 209GB to fit in RAM, the serving system:

1. Keeps the model skeleton (attention layers, embedding tables, router weights) in DRAM — roughly 40-60GB
2. Streams expert weights from SSD (NVMe at ~7GB/s on M3 Max) as needed for each forward pass
3. Maintains a 50GB zero-copy expert cache in unified DRAM+GPU memory
4. Achieves a 78% expert cache hit rate, meaning 78% of the time an expert is needed, it's already in memory from a previous request

The 78% hit rate is the number that makes this work. If you had to stream each expert from SSD from cold on every request, the NVMe bandwidth would bottleneck you to ~1 tok/s or less. The cache hit rate means you only pay the SSD latency for ~22% of expert accesses — the rest come from the warm cache at DRAM speeds.

Why This Is Different From Prior "Big Model on Mac" Demos

Previous attempts to run very large models on Apple Silicon relied on quantization to INT4 or INT3, which reduces quality significantly at large model sizes. The Qwen3.5-397B demo runs in BF16 — full weight precision. The throughput comes not from weight compression but from intelligent memory hierarchy exploitation: using SSD as a slow-but-large second tier of expert storage, with DRAM serving as a fast expert cache.

The MacBook M3 Max with 128GB unified memory has another advantage: the GPU and CPU share the same memory pool. There is no PCIe bus to cross when moving tensor data from CPU memory to GPU compute. This zero-copy architecture means expert weights staged in DRAM can be accessed by the GPU without a separate transfer step — eliminating a major latency source that would plague this approach on discrete GPU setups.

The implications for the local AI hardware market are significant. The current advice for "serious local AI" has been to build or buy a dual-3090 rig (~$2,500-3,000) or a Mac Studio with 192GB (~$5,000-7,000). The Qwen3.5-397B demo shows that a MacBook M3 Max (at ~$3,500 for 128GB) is now in the conversation for running frontier-tier models, with the right software stack. The software is what changed — the hardware has been capable for a year.

NVIDIA's Nemotron Wave

NVIDIA dropped an entire model family this week, spanning from browser-runnable to DGX-scale, in what can only be described as a strategic statement about the breadth of their AI ambitions. The Nemotron family now covers four distinct use cases:

Nemotron-Nano-4B: The Browser Model

Nemotron-Nano-4B is the most architecturally novel entry. It is a hybrid Mamba+Attention model — combining selective state space layers (Mamba-style) for efficient long-range memory with sparse attention layers for precise context retrieval. This hybrid architecture runs at 75 tokens per second in a browser using WebGPU, with no server-side compute required.

The Mamba components provide O(n) sequence processing for the majority of layers, while periodic attention layers maintain the model's ability to do precise lookups. For a 4B parameter model targeting edge deployment, this is the right tradeoff: Mamba handles the cheap long-context processing, attention handles the precise reasoning moments.

75 tok/s in a browser is genuinely useful — it's fast enough for interactive chat, code completion, and real-time summarization without any network latency. The fact that it runs on WebGPU means it works on any modern laptop with a discrete or integrated GPU, including Windows machines that can't run llama.cpp efficiently due to driver issues.

Nemotron-Cascade-2: The Math Specialist

Cascade-2 is a 30B Mixture-of-Experts model with approximately 3B active parameters per forward pass — roughly the compute cost of a 3B dense model at inference time, but with 30B of "stored knowledge" available via the router. Its headline achievement is gold-level performance on the International Mathematical Olympiad (IMO) benchmark — a threshold that until recently required models 5-10× larger.

The MoE design here is particularly clever: by specializing different experts in different mathematical domains (algebra, combinatorics, number theory, geometry), Cascade-2 can route to the right specialist for each step of a proof, without paying the inference cost of activating all domain experts simultaneously. This expert specialization is exactly the use case MoE architectures were theoretically designed for — and seeing it deliver IMO gold at 3B active parameters validates the approach.

Nemotron-Super-120B: The Dual-GPU Workhorse

Super-120B is the model that pairs with REAP this week. At 120B total parameters with approximately 20B active, it fits on two A100-80GB GPUs (just) in BF16, or on a single H100-80GB at reduced precision. Pre-REAP, throughput was mediocre due to expert routing overhead. Post-REAP, it becomes a genuinely production-viable 120B model for on-premise deployment.

NVIDIA positions Super-120B as the enterprise sweet spot: frontier-capable reasoning and coding, deployable on hardware that enterprises might already own (2× A100 is a common configuration in corporate ML infra), with no cloud dependency. At the reported 2× post-REAP throughput, it becomes economically competitive with cloud API calls for high-volume use cases.

DGX GB300 to Karpathy

The community detail that generated the most speculation: NVIDIA shipped a DGX GB300 unit directly to Andrej Karpathy. The DGX GB300 is NVIDIA's flagship AI training/inference system, equipped with 8× GB300 Blackwell GPUs, totaling approximately 1.4 petaFLOP of BF16 compute and 1TB of HBM3e GPU memory. What is Karpathy building with it? No announcement yet, but his history of releasing high-quality educational and research work (nanoGPT, llm.c, Eureka) suggests something significant is coming.

The Tooling Stack Matures

Beyond the headline models, this week saw meaningful improvements in the tooling that makes local AI practical for non-specialists:

Unsloth Studio: The One-Liner Fine-Tuner

Unsloth released a major update to Unsloth Studio, its GUI for fine-tuning open-source models. The headline numbers: 2× faster fine-tuning, 70% less VRAM compared to the standard transformers+PEFT stack. The key techniques: custom CUDA kernels for gradient checkpointing that avoid materializing intermediate activations in memory, fused operations that reduce memory bandwidth pressure, and dynamic quantization of optimizer states.

The "one-liner" framing — where you can fine-tune a 7B model on a single RTX 3090 with a simple Python script — democratizes fine-tuning for practitioners who don't have access to multi-GPU clusters. Instruction fine-tuning a 7B model on a domain-specific dataset now takes 2-4 hours on a single consumer GPU, down from 8-16 hours a year ago. At that speed, iteration cycles become fast enough for practical use: fine-tune, evaluate, adjust dataset, repeat.

LM Studio + Web Search

LM Studio shipped native web search integration — allowing locally running models to search the web in real time, retrieve snippets, and ground their responses in current information. This directly addresses the most common complaint about local models: they have a training cutoff and can't answer questions about recent events.

The implementation uses a configurable search provider (Brave, Bing, or DuckDuckGo API), fetches the top 3-5 results, and appends them to the context window before generation. For a 128K context model like Qwen3-32B, this adds negligible overhead. For smaller models with 8K context, the integration is more careful about chunking and summarizing search results before insertion.

OpenCode: Trending This Week

OpenCode, an open-source Claude Code alternative designed for local model backends, climbed trending charts on GitHub this week with 2,000+ new stars. OpenCode supports any OpenAI-compatible backend — including Ollama, LM Studio, and vLLM — meaning it can serve as the user interface for a locally running Qwen3-32B or Mistral model for coding tasks. The combination of OpenCode + a locally hosted 32B+ model is increasingly viable as an alternative to Claude Code subscriptions for developers who process large codebases where API costs add up quickly.

SmarterClaw M5 Max 120GB: 1M Context Local Agent

SmarterClaw — a local AI agent framework — shipped a major update demonstrating 1 million token context operation on a Mac Studio M5 Max with 120GB unified memory, using Qwen3-30B-A3B (a MoE model with 3B active parameters). A 1M context window is meaningful for agent use cases where you want to load entire codebases, long document archives, or extended conversation histories into context. With 3B active parameters, generation remains fast (25+ tok/s) even at these context lengths. The enabling technology: the M5 Max's memory bandwidth (400+ GB/s) and unified architecture make streaming the long KV cache between inference steps feasible without discrete GPU bandwidth bottlenecks.

Open-Source Model Releases

Beyond the flagship drops, the week saw a steady cadence of community model releases:

GLM-4.7-Flash GGUF

Zhipu AI's GLM-4.7-Flash — a compact but capable model in the GLM family — received community GGUF quantization this week, enabling it to run in llama.cpp and LM Studio. GLM-4.7 is notable for strong multilingual performance, particularly in Chinese and other Asian languages, and Flash trades some size for speed. Available in Q4_K_M, Q5_K_M, and Q8_0 quantizations, fitting on GPUs from 4GB to 8GB.

Huihui-Qwen3.5-9B Abliterated

The "abliterated" fine-tune trend — which removes safety training from models to enable uncensored responses — continued with a Qwen3.5-9B variant. Abliterated models remain controversial in the community: legitimate use cases include research into model behavior, fictional writing assistance, and red-team security testing, while abuse cases are obvious. The fact that this model reached trending status reflects the continued demand from certain communities for uncensored local models. Qwen3.5-9B base remains an excellent model — fast, capable, and efficient at 9B parameters.

MiMo-V2-Flash

Xiaomi's MiMo-V2-Flash is a reasoning-focused compact model targeting on-device deployment on mobile and edge hardware. Flash models in Xiaomi's MiMo family are designed for latency-sensitive inference — the V2 iteration improves mathematical reasoning (+18% on GSM8K versus V1) while maintaining the parameter count. A preview of the direction mobile AI is heading: not just "run a small LLM," but "run a capable reasoning model at real-time latency."

NVIDIA OpenShell

NVIDIA released OpenShell, an open-source shell automation model fine-tuned for command-line task execution. OpenShell can translate natural language instructions into shell commands, detect and recover from errors, and chain multi-step operations. Built on a compact 7B base and fine-tuned on curated shell interaction datasets, it is designed to run locally as a coding assistant backend rather than requiring cloud API calls for shell assistance. The open-source release (Apache 2.0) makes it freely usable in commercial products — a notable contrast with some of NVIDIA's earlier model releases.

Hot Spots and Signals

Several signals from this week deserve attention beyond the headline stories:

KV Caching Goes Viral (394K Views)

A post explaining KV caching — the technique that caches Key and Value tensors from prior tokens during autoregressive generation — reached 394,000 views on X (Twitter) this week. This is remarkable: KV caching is an infrastructure optimization that most model users never think about. The virality suggests that the technical literacy of the local AI community is increasing rapidly. People who were downloading models a year ago are now curious about the inference mechanics that determine whether those models run at 5 tok/s or 50 tok/s. When infrastructure becomes a conversation topic for non-engineers, the market for tooling that handles it well expands significantly.

H100s Are Appreciating

Used H100 SXM5 prices on the secondary market ticked up 8-12% over the past month, with spot compute prices on cloud providers also rising. The market interpretation: despite the push toward smaller, more efficient models, demand for large-scale inference and training compute remains strong enough to tighten supply. The most plausible explanation is the rapid proliferation of AI agent workflows — each agent call is cheap, but 1,000 concurrent agents add up. Companies that delayed H100 purchases expecting prices to fall are now competing with buyers who want them for inference rather than training.

x402: Agentic Micropayments

The x402 payment protocol — a proposed HTTP 402 Payment Required standard for AI agents — generated significant discussion this week. The premise: AI agents that browse the web, call APIs, and use external services need a way to pay for those resources without human involvement. x402 proposes a machine-readable payment negotiation protocol that agents can handle autonomously, settling in stablecoins or credit systems. The practical implication: AI agents that can pay for their own compute, API access, and data retrieval are qualitatively different from agents that require human payment setup. Combined with local inference (no cloud billing surprises), x402 could enable fully autonomous agent workflows.

AttnRes: Attention Residuals

A preprint this week on AttnRes (Attention Residual connections) proposed a modification to standard attention that adds a learned residual directly in the attention weight space — allowing earlier layers' attention patterns to influence later layers without going through the full value-weighting computation. Early ablations show ~5-8% perplexity improvement at equivalent parameter count. If the result holds up at scale, it could become a standard architectural addition to the next generation of models. Several researchers noted that it resembles residual connections in weights space, echoing the original insight about residual connections in activation space that enabled deep network training.

What to Watch Next Week

Several developments are worth tracking as we head into the last week of March:

• REAP INT4 quantization: @0xSero teased INT4 compression for Nemotron-Super-120B, which would target ~35GB — fitting on a single A100-80GB or dual-3090 setup. If the quality holds at INT4, this becomes the most capable model you can run on consumer GPU hardware.
• Karpathy's DGX project: Whatever Andrej builds with the DGX GB300 will be significant. The hardware (8× GB300, 1TB HBM3e) is overkill for personal use — this is a research station. Watch his GitHub and social channels.
• Qwen3.5 GGUF quantizations: The 397B full-precision demo is proof-of-concept. GGUF quantizations at Q4_K_M would bring the model to ~110GB — runnable on a Mac Studio M4 Ultra with 192GB unified memory, at higher tok/s. Community quantizers are typically 1-2 weeks behind major model releases.
• Unsloth Studio commercial launch: The Unsloth team hinted at a commercial tier with managed fine-tuning infrastructure. This would let practitioners fine-tune without managing GPU infrastructure themselves — a meaningful step toward making fine-tuning as easy as API use.
• MiMo-V2 mobile deployment: Xiaomi reportedly has MiMo-V2-Flash deploying on-device in their latest Xiaomi 15 Ultra phones. On-device frontier-adjacent reasoning at mobile latency, with no cloud round-trip — if benchmarks hold, this is a watershed moment for edge AI.

References

1. @0xSero on X/Twitter: REAP for Nemotron-3-Super-120B. twitter.com/0xSero
2. NVIDIA Nemotron Model Family — Official Blog. developer.nvidia.com/blog/nemotron
3. Alibaba Qwen Team: Qwen3.5 Model Release. huggingface.co/Qwen
4. Unsloth Studio 2× Speed Update. unsloth.ai
5. LM Studio Web Search Integration Announcement. lmstudio.ai
6. OpenCode GitHub Repository. github.com/sst/opencode
7. NVIDIA OpenShell Model Release (Apache 2.0). huggingface.co/nvidia/OpenShell
8. Zhipu AI GLM-4.7-Flash Model Card. huggingface.co/THUDM
9. x402 Protocol Specification Draft. github.com/coinbase/x402
10. AttnRes preprint — Attention Residual Connections for Transformer Depth. arxiv.org/abs/2503.12050

Published March 21, 2026. Weekly digest of the local AI inference ecosystem. Subscribe to the ThinkSmart.Life research feed for future issues.