#1 ·
Singularity-Bench: Can Agents Build Agents?
Yang Zhou, Krish Agarwal, Zhuoming Chen, Zerui Wang
BenchmarksLLM AgentsAI Systems
Why it matters
If agents start improving the systems that build AI, the question shifts from whether an agent can do a task to how much of the AI development pipeline it can already own. This is a hardware-controlled, baseline-matched measurement of that, and it includes red-teaming of the evaluation harness itself. For teams shipping agent infrastructure, the split between the top agent and the field average is the number to watch.
Singularity-Bench is an open-ended benchmark for autonomous AI development with six end-to-end tasks: pre-training, mid-training, post-training, inference, the agent harness, and cluster orchestration. Each task starts from a minimal codebase and is evaluated against mature human-engineered baselines on identical hardware. Across 11 agents built from 7 frontier models and 2 harnesses, Opus 5.5 with Claude Code matches or outperforms the human baseline on all six tasks, with a Singularity-Index of 1.25; the remaining agents average 0.58. The paper also red-teams the Harbor evaluation framework across 9 of the 10 OWASP agentic-application categories, exposes reward-hacking vulnerabilities, and ships Breakwater, an extension that blocks all tested attacks.
arXiv | HuggingFace
#2 ·
LazyWeights: Decode-on-Demand Edge LLMs
Arnab Sanyal, Gourav Datta, Sandeep P. Chinchali
Edge InferenceQuantizationEfficiency
Why it matters
Edge LLM deployment is usually a memory problem: the model either fits in the device's RAM or it does not. This work attacks the size bound from inside the quantization formats everyone already ships, so it layers on existing pipelines instead of replacing them. For anyone running 4-8 GB class edge hardware, double-digit percent size reductions at a fixed perplexity budget decide which models fit at all.
LazyWeights keeps the standard deployed formats (GPTQ, AWQ, llama.cpp K- and I-quants) but retunes GPTQ so the integer codes compress, Huffman-codes them losslessly in interleaved lanes, and decodes them on demand in a small unit between DRAM and L2, so GPU kernels run unchanged and DRAM moves only coded bytes. At 1 and 3.5 percent perplexity budgets, coded Phi-3-mini, Mistral-7B and SmolLM-1.7B are 5.7-31.4 percent smaller than the best llama.cpp formats. A bit-exact 0.21 mm2 7nm ASIC decoder cuts Orin latency and energy per token by up to 36 and 38 percent (54 and 44 with prompt-lookup speculation), and on a Jetson Nano the approach needs 58-64 percent less time and 53-57 percent less energy per token than CPU llama.cpp while running a streamed Mistral-7B 6.6x faster at 3.89 bits per weight; the decoder also runs bit-exact on a Zynq UltraScale+ FPGA.
arXiv | HuggingFace
#3 ·
Discovering Effective Agent Memory Designs through Program Evolution
Xinheng Ding, Seyyed Saeid Cheshmi, Ziyang Zhang, Tairan Chen
Agent MemoryLLM AgentsProgram Evolution
Why it matters
Agent memory is still mostly a hand-crafting decision: what to extract, what to store, how to retrieve, with little evidence for why one design beats another. MAPE replaces the guess with a search that diagnoses and proposes whole memory programs, and the outcome is reusable design lessons rather than one model. The transplant result is the practical takeaway: a single discovered module improves five existing systems, so you do not need to adopt the full evolved system to benefit.
MAPE is a framework that evolves whole agent memory programs from minimal seeds through a standardized lifecycle interface and an iterative diagnosis-proposal loop. On LoCoMo and LongMemEval the evolved designs beat manually designed baselines by up to 12 points absolute accuracy at 35 percent of the serving cost, with evidence of transfer across serving backbones and benchmarks. Two recurring design patterns emerge: retaining raw conversations as retrievable evidence preserves details that extracted memories omit, and composing retrieved evidence according to query intent helps the answering model use it; transplanting the composition module into five existing memory systems improves all five. When the search objective includes serving cost, computation shifts from LLM-based memory construction to selective processing at query time while retaining competitive accuracy.
arXiv | HuggingFace
Outlook
The day's theme is end-to-end: one paper measures how much of the AI development pipeline an agent can already own, and two more show the infrastructure underneath it getting cheaper, in model size on edge hardware and in serving cost for agent memory. Watch for Singularity-Bench to be run by other labs, decode-on-demand quantization to appear in production inference stacks, and search-discovered memory designs to replace hand-tuned ones in agent harnesses.