Buying a testing agent: easy. Knowing if it works, recalibrating it, watching its reasoning, drawing its authority line: not in the box. The five questions I'd start with: https://kato-coaching.com/fighting-ai-with-ai-easy-part/ Paid partnership w/ SmartBear.

Holy guacamole. Long-ish read, pretty jaw dropping. bobdahacker.com/blog/fifa-hack

You can't manually test AI-generated code at AI-generation speed. That math isn't mathing. 40% of teams generate 41%+ of their code with AI. 92% still test by hand. Source:

Everyone in QA can describe their AI's bad habits without thinking. Nobody finishes "what I really want from this tool is…" If you've been generating tests with AI and correcting the same handful of things every week, what would actually help?

The research on code review has been consistent for over a decade: it's strong at knowledge sharing, weak at catching defects. When teams insist on "humans review every line" of AI output, they're doubling down on the thing we know doesn't work. kato-coaching.com/human-in-the-loop-code-review

The AI keeps making the same test mistake because the decision to stop is buried in a Slack thread it's never read. Write the rule. Put it in the file your AI reads at generation time. Walkthrough: https://youtu.be/ccFaZdGUSkw #AIinTesting #QA

> 65% of QA teams describe AI output as "noisy, incorrect, or requiring rework." > > Mapping where the verification time actually goes. 5-min survey, no opt-in: >

Last night Alan and I talked about the new cool kid on the block, and how it's changing testing. Or maybe not?  I'll link Apple podcasts, but it's available everywhere you get your podcasts. 

Our free web development bootcamp launch party is today! Join: @vivaldi.com's @brucel.bsky.social will be showing us the lovely side of HTML Me for how good notes drive learning @freecodecamp.bsky.social's @quincy.bsky.social for a fireside chat about learning We would love to see you there!

It’s no secret that I love working with claude code. If you’ve not tried it yet but would like to: turns out I’ve got three passes for a free week of it. Grab it here claude.ai/referral/WY0...

I have a Claude Code skill that creates google slides for me, and then adds speaker notes. Since the update to Sonnet 4.7 it constantly "forgets" how to do the latter, and tells me it's incapable of it. Anybody else have this problem? Suggestions?

The "testing mindset" isn't a personality trait. It's trained. Developers often don't think about quality the way QA engineers do because they haven't had the same deliberate practice. That's fixable, but only if the culture reinforces it too.

Before you conclude AI isn't worth it, measure how long correction actually takes. One experiment, a real task, less than a day.  Free guide: resources.kato-coaching.com/5-ai-experiments #SoftwareTesting #QA #AIinTesting

The Easter Bunny has 10,000 eggs to hide across 47 gardens before sunrise. His team suggests using AI to plan the hiding spots. Before handing a task to AI, run it through a simple framework. Here is how the Easter Bunny uses this framework to avoid a disaster. 🐇🥚

Getting the task type wrong makes the model choice irrelevant. The scorecard tells you what kind of solution actually fits before you commit to one.  https://resources.kato-coaching.com/scorecard #SoftwareTesting #AItools #QA

TIL: it's much easier to get Claude Code to play nicely with the Google workspace MCP than it is to get Gemini to use it. What the hell? Yes, I am experimenting with a multiple agent setup, but man, it's painful.

AI gave you 12 scenarios. Now what? The list is raw material, not a test plan, and your judgment is what turns it into one. New video on how to triage what AI gives you: https://youtu.be/0awmamL1l2c #softwaretesting #AIinTesting

Agile gave us JIRA. DevOps gave us pipelines. Neither answered the harder question. AI evals is doing the same thing: the tooling is excellent. The question underneath it remains.  https://kato-coaching.com/the-hard-part-of-ai-evals-isnt-the-tooling/ #AITesting #QualityEngineering #SoftwareTesting

Most AI pilots produce impressions, not evidence. An acceptable error rate isn't a feeling. It's a number you write down before the experiment starts.  https://resources.kato-coaching.com/scorecard #SoftwareTesting #AItools #QA

"The profession keeps arriving at “quality is systemic” as though it’s a fresh revelation, every time a new technology makes the verification frame obviously inadequate. It happened with continuous delivery. [...] It is happening now with AI agents."

Every AI demo works because the vendor picked the use case. When you're back at your own codebase, the conditions are different. Define your problem before you evaluate the tool.  https://resources.kato-coaching.com/scorecard #SoftwareTesting #AItools #QA

The vague prompt gets you a generic list. Two minutes of context - scope, users, risk angle - gets you scenarios worth thinking about. Short video on how to close that gap.  https://youtu.be/55SLTvKq7rE #softwaretesting #AIinTesting

The AI evals field calls it a rubric. If you've done serious testing work, you probably know it by a different name.  https://kato-coaching.com/the-anatomy-of-a-metric/ #SoftwareTesting #AITesting

When AI adoption is handed to IT, you get a deployment. Deciding what good output looks like and how to verify it belongs to the people doing the work. https://resources.kato-coaching.com/scorecard?utm_source=Bluesky&utm_medium=post&utm_campaign=scorecard  #QualityEngineering #TechLeadership

When teams say the AI tool isn't working, I ask about the process before the AI arrived. Usually the requirements were always vague. The tool is just making that visible. https://resources.kato-coaching.com/scorecard #SoftwareTesting #AIinTesting #QA

Six months into an AI pilot with no success criteria is just six months of accumulated impressions.  You can't pass or fail a hope. 

I am working on a food tracker for my very specific needs, and Claude just turned into every developer I ever worked with.

"The hard part is scalability, not automation." That line from session one of "AI evals and analytics" confused me. Session two explained it.  Full write-up in my blog: https://kato-coaching.com/the-ai-evals-field-chose-a-flawed-tool-and-stuck-with-it/ #AIEvals #SoftwareTesting #QA

Updated my website this week — it should finally be clear what I do and how to work with me. Courses, workshops, and 1:1 coaching for QA professionals.  kato-coaching.com

If the output is slop regardless of how you phrase it, the problem isn't the prompt. It's the use case.  Free scorecard:

Before I wrote today's post, I defined what good looked like: gives value, doesn't rely on outrage, sounds like me. Writing to a clear brief changes the experience. So does diagnosing a draft. Testers do this before they run anything.

The AI correction loop usually starts before the tool is opened, at the moment someone chose the wrong use case for it. Testers already know how to ask whether a tool suits a problem. That skill just hasn't been applied here yet. More soon.

The skills QA professionals already have (defining success criteria, testing behaviour, not trusting metrics at face value) are exactly what's missing from most AI integrations. I'm learning AI evals to understand why.  Session one: kato-coaching.com/what-i-dont-understand-about-ai-evals-yet/

Ran my workshop "Deciding Fast" on Tuesday with a software team in Sweden. Everyone in one room sharing computers, no breakouts. It's built for remote. I adapted. 6/8 Good or Excellent. Best response to "what will you do differently?": "Set clearer success condition." #SoftwareTesting #AITesting

"Tell me what you're uncertain about." "Push back if my assumptions are wrong." Only 30% of people give instructions like these. The model defaults to confident and agreeable.  https://www.anthropic.com/research/AI-fluency-index #SoftwareTesting #AITesting #AILiteracy

The strongest predictor of AI fluency, per Anthropic's research: iteration. Treating the first response as a draft, not an answer. 5.6x more likely to question reasoning. Familiar territory if you work in testing.  https://www.anthropic.com/research/AI-fluency-index #SoftwareTesting #AITesting

Six months in and nobody can say whether the AI is actually working. Not anecdotally, but with evidence. That gap is the most common thing I see in QA teams right now.  How do you measure if the new licence is worth the money? #softwaretesting #QA #AItools

18% of testers I surveyed said their top AI frustration is not bad output. It is that the tools have no sense of test strategy. The AI is not wrong. It is indiscriminate. #AITesting #SoftwareTesting #TestStrategy

I timed myself running tests manually, then gave the same task to an AI tool and timed that too. The result was not what I expected. Full breakdown with video later this week.

You ask the AI to generate a few test cases for a feature you know inside out. Save yourself twenty minutes. What comes back looks reasonable. Then you start reading. Read the full story in my blog:

Why do so many testers call AI "exhausting"? It is not the learning curve. It is the correction loop. You spend 45 minutes fixing output that was supposed to save you an hour. Five experiments to figure out which tasks AI actually improves:

I asked testers what worried them most about AI. 24% said management expectations, not the tools. "Pressure to deliver has increased dramatically because management thinks we must be twice as productive now." Argue with evidence:

I surveyed 17 testers about their biggest AI frustrations. 65% said the same thing: the output is unreliable. One called it "slop." Another described the correction loop as exhausting. I wrote a free guide: 5 small experiments

I have blogged about why I think the rise of AI tooling is not the end of the world, as many in the testing community see it. 

AI can generate 500 test cases in an hour. But if you don't know what decision you're supporting, you're just generating noise faster. I'm updating the QED course on this. Quick survey if you've worked with AI testing tools:

AI tools won't fix your testing problems if you're solving the wrong problems. The QED framework: start with what's breaking, who feels it, and what it costs. Then run a two-week experiment. Works whether you're writing tests by hand or using AI.

AI generates test cases fast. Most of them are useless.  Here's how to filter before you generate: 

The AI review debate feels similar to early reactions to spellcheck. Tools that scan for specific issues at scale can raise the baseline, as long as the rules are clear and judgement stays with the human.

As a German living abroad, with many friend all over the world and specifically in the U.S., this summarises well how I feel about the current situation.

AI did not break testing. We were already stuck. Watching QA argue about titles while AI becomes the new panic topic feels familiar.  We already know how to deal with uncertainty and risk. We just keep choosing not to apply it.