🎉 iSWE-Agent is now Rank #1 on the SWE-PolyBench (Verified) Python leaderboard — 58.41% resolved (66/113), topping Atlassian Rovo Dev (54.87%). 🧵

🚀 SWE-Bench-Arena now speaks 8 languages. Patches are pulled from 3 benchmarks: ✅ SWE-bench Verified (Python) ✅ Multi-SWE-bench (Java, TS, JS, Go, Rust, C, C++) ✅ SWE-PolyBench (Python, Java, JS, TS) Pick your language, review AI vs human patches → swebencharena.com #SWEBenchArena #AIEvaluation

A coding agent can solve the right bug and still fail a benchmark for choosing the wrong unstated identifier. I call this a hidden naming contract. New post: jatinganhotra.dev/blog/swe-age...

The paper behind #IBMResearch iSWE-Agent is now out on arXiv. If you saw posts on iSWE-Agent topping Multi-SWE-Bench and SWE-PolyBench for Java, this paper shares the system behind those results: agent design, workflow, and Java-aware tooling. arxiv.org/abs/2603.11356

1/ OpenAI: SWE‑Bench Verified is no longer a good frontier eval — test/spec mismatch + contamination. openai.com/index/why-we...

🚀 IBM Research's iSWE-Agent is now #1 on the SWE-PolyBench (full) Java leaderboard🎉 On the Verified subset, iSWE-Agent scores 46.38% on Java — matching Atlassian Rovo Dev and significantly outperforming Prometheus (33.33%). More details: jatinganhotra.dev/news/ #AI #Java #SWEPolyBench

(repost welcome) The Generative Model Alignment team at IBM Research is looking for next summer interns! Two candidates for two topics 🍰Reinforcement Learning environments for LLMs 🐎Speculative and non-auto regressive generation for LLMs interested/curious? DM or email [email protected]

🧵 1/4 Current AI coding benchmarks miss the mark. Claude 4 Sonnet hits 72.7% on SWE-Bench, but industry data shows code clones rose 48% (8.3% to 12.3%) and refactoring rates dropped from 25% to 10% since AI adoption. (GitClear: gitclear.com/ai_assistant_code_quality_2025_research)

🔍 AI models hit 72%+ on coding benchmarks, but there's a hidden problem... Recent data shows concerning trends since AI adoption: • 48% increase in code cloning • Refactoring dropped from 25% to 10% • Developers report "missing context" as #1 issue Are we optimizing for the wrong metrics? 🧵

🚨 New Blog Post: AI agents collapse under visual complexity. A 73.2% performance drop when images are introduced in SWE-bench Multimodal. Here's why this matters — and what it tells us about the future of AI in software engineering: 🧵👇

Like the tale of the Emperor's new clothes, sometimes we need fresh eyes on familiar benchmarks. SWE-Bench Verified shows 73% success rates, but focusing on discriminative subsets reveals a different story: 11% What really challenges AI agents? Analysis: jatinganhotra.dev/blog/swe-age...

Fascinating finding: When you remove the 156 problems that 61+ agents solve, performance drops dramatically Top agents: 73% → 11% This isn't about making things harder - it's about measuring what matters 🎯 jatinganhotra.dev/blog/swe-age...

1/ "What gets measured gets improved" - but are we measuring the right things? SWE-Bench Verified has driven amazing progress, but with most agents solving 350+ same problems, we need new targets @ofirpress.bsky.social Enter: discriminative subsets that highlight genuine challenges 🧵

With OpenAI releasing Codex, a SWE agent openai.com/index/introd... and evaluating on both "SWE-Bench-Verified" AND "OpenAI Internal SWE tasks", I thought it's important to once again highlight issues with SWE-Bench-Verified. I've compiled all my analysis here: 🧵

🚀 Advancing AI-Powered Software Testing! 🚀 SWE-Agents can now generate code to resolve GitHub issues—but how do we ensure these fixes are robust and reliable? Introducing **Otter & Otter++**, two innovative test generation approaches leveraging LLMs with self-reflective action planning. 🦦🔍

🌟 New Benchmark! 🌟 Do you work on RAG? Are you interested in Multi-Turn conversations? Very excited to share the new MTRAG benchmark we have released! Data: github.com/ibm/mt-rag-b... Paper: arxiv.org/abs/2501.03468

Can AI handle multi-file coding challenges like humans? This post dives deep into SWE-Agents vs. human engineers on SWE-Bench Verified. With 14.2% of tasks needing cross-file changes, are AI systems taking shortcuts or truly solving the problem? jatinganhotra.github.io/blog/swe-age...