Trial by Trace
Benchmarking AI agent harnesses for reliability and failure attribution. TraceTrial-Bench examines 95 tasks across four harnesses and nine model configurations.
From the lab
Milestones
Reports, papers, and moments that move our work forward.
View the full timeline-
arXiv v1 submitted
-
Proceedings
MM-VeriRubric: Verifiable Evidence-Grounded Rubrics for Multimodal Reward Models
Grounding multimodal reward-model rubrics in task-relevant visual evidence and verifying their consistency.
-
Proceedings
Disentangled Privileged States for Simulator-Assisted Multi-Turn Dialogue Policy Training
Separating privileged state tracking from simulated user utterances to support multi-turn dialogue policy training.
-
Article published
TraceLite AI in InfoQ’s AICon Shenzhen recap
InfoQ’s conference recap lists TraceLite AI among the organizations represented by speakers at AICon Shenzhen 2026.
-
Proceedings
SimRPD
Optimizing recruitment proactive dialogue agents through simulator-based data evaluation and selection.
-
Proceedings
Look Less, Reason More
Rollout-Guided Adaptive Pixel-Space Reasoning.
-
Conference year
Absorbing Gradient Conflicts: Modeling Semantic Variance via Kent Distributions for Cross-Modal Hashing
Modeling class proxies as Kent distributions on the hypersphere so that conflicting multi-label gradients are absorbed by directional variance instead of shifting the proxy.
-
Conference year
Select Less, Reason More: Prioritizing Evidence Purity for Video Reasoning
Selecting relevant frames and resampling visual evidence for video reasoning.
-
Conference year
CausalStep
A Benchmark for Explicit Stepwise Causal Reasoning in Videos.
Featured work
Research
Papers
MM-VeriRubric: Verifiable Evidence-Grounded Rubrics for Multimodal Reward Models
Most multimodal reward models emit only an outcome reward, which is sparse and misses the structure of human preference. MM-VeriRubric builds training data for multimodal rubric generation at scale: task-aware visual clue extraction grounds rubrics in task-relevant image evidence, and rubric-aligned rejected responses with controlled error dimensions let each rubric be verified through consistency checking. The verified data trains both a rubric generator and a rubric-based reward model, improving preference judgment by 16.4% on VL-RewardBench and Best-of-5 selection by 13.1% on MMMU-Pro.
Read the paper
Disentangled Privileged States for Simulator-Assisted Multi-Turn Dialogue Policy Training
Simulator-assisted reinforcement learning is a practical way to train multi-turn dialogue agents, but policy optimization gets misled when the simulated user gives inconsistent feedback across long trajectories. DPS decouples task-conditioned privileged state tracking from surface utterance realization; those privileged states stay hidden from the agent but are used during training to derive dense, state-aware process rewards that complement sparse outcome signals. The work also introduces RecruitmentBench, a persuasion dialogue benchmark grounded in anonymized real-world logs, and reports improved measured simulator fidelity and stronger fine-tuned policy performance.
Read the paper
Absorbing Gradient Conflicts: Modeling Semantic Variance via Kent Distributions for Cross-Modal Hashing
Proxy-based deep cross-modal hashing usually treats each class proxy as a fixed point in the embedding space. With multi-label data, co-occurring labels pull that point in conflicting directions, causing gradient contention and proxy collapse. KDPH instead models each proxy as an anisotropic Kent distribution on the hypersphere: the proxy keeps a stable mean direction and absorbs conflicting gradients by adjusting its directional variance, while a loss built on the Cayley transform keeps the distribution parameters orthogonal during training. It is the first framework to bring Kent distributions into cross-modal hashing; on three benchmark datasets it mitigates proxy collapse and oscillation and outperforms state-of-the-art methods.
SimRPD: Optimizing Recruitment Proactive Dialogue Agents through Simulator-Based Data Evaluation and Selection
Proactive dialogue agents steer recruitment conversations toward concrete business outcomes, but training them is bottlenecked by the scarcity of high-quality, goal-oriented domain data. SimRPD is a three-stage framework: a high-fidelity user simulator synthesizes large-scale multi-turn conversations, a Chain-of-Intention evaluation framework assesses the simulator and selects high-quality data using both global-level and instance-level metrics, and the agent is then trained on that selection. In a real-world recruitment scenario SimRPD outperforms existing simulator-based data selection strategies.
Look Less, Reason More: Rollout-Guided Adaptive Pixel-Space Reasoning
Pixel-level visual operations help Vision-Language Models handle fine-grained detail, but they are routinely overused — costing efficiency and pulling attention toward irrelevant regions. This paper proposes the first framework for adaptive pixel reasoning, which decides per query whether a pixel operation is warranted: operation-aware supervised fine-tuning establishes baseline competence, then a rollout-guided reinforcement learning stage uses feedback from the model's own responses to learn when to invoke tools. The result reaches 73.4% accuracy on HR-Bench 4K at a tool usage ratio of only 20.1% — 66.5% less tool use than prior methods, with higher accuracy.
Select Less, Reason More: Prioritizing Evidence Purity for Video Reasoning
Uniform frame sampling dilutes evidence in long videos, and existing pixel-space video agents lack reward mechanisms that enforce evidence purity or let them look beyond pre-sampled frames. This work introduces EARL, an evidence-aware reinforcement learning framework that turns the model into an active interrogator of evidence: it selects the most relevant frames and re-samples locally around them to recover fine-grained temporal detail. Across five video reasoning benchmarks the EARL-trained 7B model sets a new state of the art among open-source Video LLMs, reaching 59.8% on LongVideoBench, 69.0% on MVBench and 64.9% on VideoMME.
CausalStep: A Benchmark for Explicit Stepwise Causal Reasoning in Videos
Existing video benchmarks mostly test shallow understanding and let models lean on global context, so they cannot tell whether a model really reasons step by step about cause and effect. CausalStep segments each video into causally linked units and enforces a strict stepwise question-answer protocol: questions must be answered in sequence, which blocks shortcut solutions, and every question carries distractors built from an error-type taxonomy. The benchmark has 100 videos across six categories and 1,852 multiple-choice QA pairs, scored with seven diagnostic metrics. Experiments with leading proprietary and open-source models, alongside human baselines, show a significant gap between current models and human-level stepwise causal reasoning.
Blog
Explore the blog
Notes
What we are working on, and what we plan to build next.
Roadmap / Environment
What is an agent's environment made of?
We want to describe an environment the way we describe a model: by listing the factors that change what the right move is.
Read the note
Roadmap / Evaluation
Evaluating the harness, not just the model
The same model can behave very differently inside different agent harnesses. We want that difference to be measurable.
Read the note
Roadmap / Training data
Turning failures into training data
Agents fail all the time on real tasks, and most of those failures are thrown away. We want to keep the useful ones.
Read the note