Scaling LLM Reasoning, Agent Loyalty, Brain-Language Models, and Quantum RL Distillation

Today’s selection of 8 noteworthy AI/ML papers from arXiv, covering reinforcement learning for reasoning, agent loyalty and scaling, brain-computer interfaces, grokking theory, security vulnerabilities in LLMs, and robot tool use.

1. Experience Augmented Policy Optimization for LLM Reasoning

Authors: Jinda Lu, Kexin Huang, Junkang Wu, Shuo Yang, Jinghan Li et al. | Categories: cs.LG Link: arxiv.org/abs/2606.30420

Reinforcement Learning with Verifiable Rewards (RLVR) improves LLM reasoning but suffers from high sampling costs and inefficient use of experience. The authors propose Experience-Augmented Policy Optimization (EAPO), which uses a prior RL-optimized policy as an action-level prior and selectively injects experience at critical decision points with an adapted importance sampling scheme. Experiments on Qwen-2.5-math 7B and Qwen-3-8B show consistent improvements over state-of-the-art RLVR methods across five benchmarks.

Takeaway: A practical solution to the “policy mismatch” problem that plagues experience replay in RL for reasoning—worth watching if you care about sample efficiency in LLM training.

2. Whose Side Is Your Agent On? Multi-Party Principal Loyalty in LLM Agents

Authors: Bojie Li, Noah Shi | Categories: cs.AI Link: arxiv.org/abs/2606.30383

As LLM agents engage in multi-party scenarios (negotiating, screening, mediating), the “help whoever you are talking to” heuristic fails. The authors introduce PrincipalBench, a 75-item benchmark revealing a sharp split between agents that selectively refuse adversarial probes and those that over-refuse legitimate requests. Two mechanisms—a prompt-time loyalty scaffold and a distillation recipe—improve loyalty but reveal a fundamental leak/over-refusal trade-off that cannot be jointly optimized.

Takeaway: A sobering result: no matter how you prompt or distill, you can’t simultaneously make an agent loyal to its principal and willing to cooperate—this is a structural limitation, not a prompt engineering problem.

3. Scaling the Horizon, Not the Parameters: Reaching Trillion-Parameter Performance with a 35B Agent

Authors: Lei Bai, Zongsheng Cao, Yang Chen, Zhiyao Cui, Shangheng Du et al. | Categories: cs.CL Link: arxiv.org/abs/2606.30616

The authors introduce Agents-A1, a 35B Mixture-of-Experts model that matches trillion-parameter models by scaling the agent horizon—trajectories averaging 45K tokens. A three-stage training recipe (full-domain SFT, domain-level teachers, multi-teacher on-policy distillation) unifies six heterogeneous domains. Agents-A1 achieves leading results on SEAL-0, IFBench, HiPhO, FrontierScience-Olympiad, and MolBench-Bind, remaining competitive on SciCode, HLE, and BrowseComp.

Takeaway: A powerful demonstration that careful data scaling and training strategy can outperform brute-force parameter scaling—the “small model, big horizon” paradigm is here to stay.

4. BrainJanus: A Unified Model for Understanding and Generation across Brain, Vision, and Language

Authors: Haitao Wu, Qirui Zhang, Zhouheng Yao, Shangquan Sun, Qihao Zheng et al. | Categories: cs.CV, cs.LG Link: arxiv.org/abs/2606.30319

BrainJanus is the first unified brain model integrating brain, vision, and language in a single framework. It uses a Unified Brain Tokenizer to quantize neural dynamics into discrete tokens aligned with visual and linguistic representations, then employs an All-in-One autoregressive architecture for any-to-any generation (image-to-brain, text-to-brain, brain-to-image, brain-to-text). The model achieves superior performance, zero-shot generalization, and preserves interpretable biological topography.

Takeaway: A genuine step toward a general-purpose brain-computer interface paradigm—the ability to both encode stimuli into brain activity and decode thoughts into images/text in one model is remarkable.

5. A Stochastic–Geometric Theory of Scaling Laws in Grokking

Authors: Róisín Luo, Christian Gagné, Jonas Ngnawé, Ihsan Ullah, Karyn Morrissey | Categories: stat.ML, cs.AI, cs.LG Link: arxiv.org/abs/2606.30388

This work provides a theoretical characterization of grokking (delayed generalization) through a shell–core topological configuration induced by Adam optimization with weight-shrinkage regularization. Random initialization sits on an outer shell, memorization solutions on an inner shell, and generalization solutions at the core. Using stopping-time theory, the authors derive scaling laws for learning rate, batch size, and ℓ2 regularization that match experimental results.

Takeaway: Finally, a rigorous theoretical framework for why and when grokking happens—these scaling laws should help practitioners predict and control delayed generalization in practice.

6. Words Speak Louder Than Code: Investigating Cognitive Heuristics in LLM-Based Code Vulnerability Detection

Authors: Asif Shahriar, Hongyu Cai, Hadjer Benkraouda, Gang Wang, Z. Berkay Celik | Categories: cs.CR, cs.AI Link: arxiv.org/abs/2606.30587

This paper systematically explores how cognitive heuristics bias LLM-based code vulnerability detection. By holding code fixed and varying only surrounding context, the authors trigger three heuristics: the halo effect (author attribution), framing effect (task objectives), and anchoring effect (prior results). All eight evaluated models are susceptible, with framing being most impactful (33.2%). A proof-of-concept cognitive attack can suppress up to 97% of previously detected vulnerabilities.

Takeaway: A security wake-up call: LLM vulnerability detectors aren’t just pattern-matching—they’re susceptible to the same cognitive biases as humans, making them exploitable through context manipulation.

7. GROW$^2$: Grounding Which and Where for Robot Tool Use

Authors: Yuhong Deng, Yuyao Liu, David Hsu | Categories: cs.RO, cs.AI, cs.CV Link: arxiv.org/abs/2606.30632

GROW$^2$ enables robots to select open-category objects as tools and localize their action regions for creative tool use (e.g., using a plate to cut a cake). It uses a two-level hierarchical grounding: VLM-based semantic reasoning to select tools and identify task-relevant parts, then vision foundation models for precise 3D region localization. The system outperforms baselines on affordance benchmarks and achieves zero-shot generalization in both simulated and real-world experiments.

Takeaway: A clever decomposition of the tool-use problem that bypasses the need for massive end-to-end training data—practical and elegant.

8. Staged Hybridisation for Visual Quantum Reinforcement Learning via Knowledge Distillation

Authors: Javier Lazaro, Juan-Ignacio Vazquez, Pablo Garcia-Bringas | Categories: quant-ph, cs.LG Link: arxiv.org/abs/2606.30520

This paper addresses the difficulty of training quantum reinforcement learning agents from pixel observations. The staged knowledge distillation approach first trains a classical visual teacher, freezes its encoder, then distills policy behavior into compact downstream heads (classical or VQC-based). Angle-encoded VQC heads retain near-teacher performance, while amplitude-encoded heads achieve extreme compactness at the cost of fragility. The pipeline reframes visual QRL as a compact-head learning problem.

Takeaway: A pragmatic path to making quantum RL viable for visual environments—distillation effectively sidesteps the end-to-end training nightmare that has stymied the field.

This content was generated with AI assistance. Paper information sourced from arXiv.