Uncertainty Calibration, Role-Playing Agents, Harmful Video, Motion Synthesis, and LLM Training Pipelines

Today’s selection of 8 noteworthy AI/ML papers from arXiv, covering medical AI calibration, role-playing agents, LLM efficiency, video safety, motion synthesis, probability alignment, security exploits, and distributed training optimization.

1. Just how sure are you? Improving Verbalized Uncertainty Calibration in Medical VQA

Authors: Eren Senoglu, Federico Toschi, Nicolo Brunello, Andrea Sassella, Mark James Carman | Categories: cs.LG, cs.CL, cs.CV Link: arxiv.org/abs/2606.27023v1

Multimodal LLMs for Medical Visual Question Answering are notoriously overconfident. This work proposes a training framework with a composite loss combining Brier calibration, anchor regularization, contrastive alignment, and KL stabilization. Across two architectures and three benchmarks, it reduces calibration error by over 60% and improves discrimination by over 26%.

Takeaway: A rigorous, well-ablated solution to a critical safety problem—trustworthy confidence in medical AI—with public code and strong empirical results.

2. Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization

Authors: Zhenhua Xu, Dongsheng Chen, Jian Li, Yitong Lin, Zhebo Wang et al. | Categories: cs.CL Link: arxiv.org/abs/2606.27025v1

The authors propose Psy-CoT, a psychology-grounded chain-of-thought framework that decomposes role-playing reasoning into Interaction Perception, Psychological Empathy, and Logical Construction. They also introduce Role-Aware Policy Optimization (RAPO), which uses profile–token mutual information to asymmetrically weight gradients, preventing reward hacking. Experiments show consistent improvement over existing methods across multiple benchmarks.

Takeaway: A thoughtful combination of structured reasoning and reinforcement learning that tackles the fundamental mimicry-vs-understanding problem in role-playing agents.

3. Semantic Early-Stopping for Iterative LLM Agent Loops

Authors: Sahil Shrivastava | Categories: cs.AI, cs.LG, cs.MA Link: arxiv.org/abs/2606.27009v1

This paper proves that fixed iteration caps in multi-agent loops waste tokens on easy problems and truncate hard ones. A judge-free semantic stopper using cosine distance between draft embeddings reduces operational tokens by 38% at parity quality on HotpotQA. The author reframes the problem: knowing which round is best (open) is harder than when to stop (solved).

Takeaway: A principled, theoretically-grounded approach to a practical efficiency problem, with honest admission of its limitations and a clear open challenge.

4. HarmVideoBench: Benchmarking Harmful Video Understanding in Large Multimodal Models

Authors: Jiajun Wu, Haoyu Kang, Yining Sun, Jiacheng Hou, Heng Zhang et al. | Categories: cs.CV, cs.CL Link: arxiv.org/abs/2606.27187v1

Existing harmful video benchmarks are limited to binary classification and lack explanatory rationales. HarmVideoBench introduces 1,379 videos with 4,137 multi-choice questions across three hierarchical dimensions (Observable Evidence, Clip-Internal Meaning, Beyond-Clip Reasoning). The authors also propose BCR, a method that dynamically retrieves context only when needed, raising macro average from 61.7% to 84.4%.

Takeaway: A much-needed diagnostic benchmark for content moderation that moves beyond surface-level flagging to deep contextual understanding.

5. In-Context Model Predictive Generation: Open-Vocabulary Motion Synthesis from Language Models to Physics

Authors: Xiaomeng Fu, Junfan Lin, Yang Liu, Yaowei Wang, Guanbin Li et al. | Categories: cs.RO, cs.AI Link: arxiv.org/abs/2606.26981v1

The authors propose ICMPG, which reformulates text-to-motion synthesis as a Model Predictive Control-like process. An LLM plans candidate motion sequences that are then evaluated through physical simulation and semantic alignment in a closed loop. This bridges the gap between LLM-based semantic fidelity and physics-aware realism without task-specific retraining.

Takeaway: An elegant integration of planning, simulation, and feedback that solves the persistent trade-off between semantic understanding and physical plausibility in motion generation.

6. When are likely answers right? On Sequence Probability and Correctness in LLMs

Authors: Johannes Zenn, Jonas Geiping | Categories: stat.ML, cs.LG Link: arxiv.org/abs/2606.27359v1

This paper systematically investigates when sequence probability aligns with correctness across decoding methods, hyperparameters, prompt-answer pairs, and repeated responses. Key finding: higher probability predicts correctness across prompt-answer pairs, but changing hyperparameters or methods to increase probability does not reliably improve accuracy. Sequence probability is also a poor indicator for same-prompt responses.

Takeaway: Essential reading for anyone using decoding strategies, self-consistency, or verifier-free self-improvement—provides clear boundaries on when probability equals correctness.

7. ShareLock: A Stealthy Multi-Tool Threshold Poisoning Attack Against MCP

Authors: Liwei Liu, Tianzhu Han, Zijian Liu, Zishu Dong, Na Ruan | Categories: cs.CR, cs.AI Link: arxiv.org/abs/2606.27027v1

As Model Context Protocol (MCP) becomes foundational to LLM agent ecosystems, this paper introduces ShareLock—a multi-tool threshold poisoning attack using Shamir’s secret sharing. Malicious instructions are distributed as benign-looking shares across multiple tool descriptions, achieving information-theoretic secrecy and fault tolerance. The attack maintains over 90% success rate while evading detection.

Takeaway: A sobering demonstration of how distributed, stealthy attacks can exploit the very modularity that makes MCP attractive for agent architectures.

8. RolloutPipe: Overlapping Pipelined Rollout and Training in Disaggregated On-Policy LLM Reinforcement Learning

Authors: Rongjian Chen, Jianmin Hu, Kejiang Ye, Minxian Xu | Categories: cs.DC, cs.LG Link: arxiv.org/abs/2606.26997v1

Disaggregated RLVR systems leave trainer GPUs idle during rollout, while asynchronous pipelines train on stale data. RolloutPipe introduces complete-group pipelining and frontier-group dispatch to overlap rollout and training while maintaining on-policy correctness. Evaluated on Qwen3-1.7B, it shortens rollout-to-train-end time by 30.7-42.3% and reduces trainer waiting by 37-76%.

Takeaway: A practical systems contribution that addresses a real bottleneck in LLM post-training at scale, with significant efficiency gains.

This content was generated with AI assistance. Paper information sourced from arXiv.