AI signal, minus the noise.

NVIDIA and Artificial Analysis have released AgentPerf, the first benchmark designed specifically for agentic AI infrastructure. Unlike traditional benchmarks, AgentPerf measures performance when an AI agent chains dozens to hundreds of model calls, uses tools, gathers context, and iterates until task completion. The initial results highlight that NVIDIA Blackwell delivers 20 times more agents per megawatt compared to the previous NVIDIA Hopper architecture.

A V2EX user posed a distributed systems question: after sequentially writing keys 1 through 5 in ZooKeeper, can three clients simultaneously observe the sequences [1-5], [2-5], and [3-5]? Different AI assistants gave contradictory answers. The user’s own reasoning, based on ZooKeeper’s sequential consistency guarantees, argues it is impossible because seeing a later write implies having seen all earlier writes. The user then fed their reasoning back to the AIs, which then agreed, highlighting that LLMs can flip opinions and lack reliable reasoning on nuanced technical topics. The post is framed as an informal evaluation of AI judgment and intelligence.

A recent paper compared clinical AI tools (such as OpenEvidence) with general frontier large language models (LLMs). The evaluation showed that frontier LLMs outperformed the specialized clinical tools in all three assessments. The clinical AI tools' performance was comparable to that of Google Search's auto-enabled AI Overview on the RCQ benchmark. This finding challenges the widespread push for adopting purpose-built medical AI tools, suggesting that general LLMs are already more capable for medical queries.

The paper presents Claw-SWE-Bench, a benchmark designed to standardize evaluation of OpenClaw-style coding agent harnesses. It provides 350 GitHub issue-resolution instances spanning various programming languages and repositories, along with a Lite version for rapid validation. An adapter protocol is introduced to decouple agent logic from harness execution, and experiments show that adapter choice significantly impacts agent performance. The results highlight the critical role of harness design and cost in fair comparisons, offering a reproducible and cost-effective reference set for coding-agent evaluation.

A study proposes a framework that employs large language models to automate the assessment of research reproducibility in the social and behavioral sciences. The framework aims to reduce time, effort, and human biases associated with manual reproducibility checks. By leveraging LLMs, the method can streamline the evaluation of whether study results can be reliably reproduced. This innovation addresses the ongoing replicability crisis in these fields, potentially fostering more transparent and trustworthy research practices. The paper discusses the technical approach and its implications for improving scientific credibility.

In a recently conducted agent evaluation, the highest difficulty tier proved insurmountable: every tested agent scored zero. No model was able to earn any points on that level, highlighting the extreme challenge posed by the benchmark.