AI Test Manager for Everyone
I’ve learned some valuable lessons about testing and software quality over the years. I’ve always tried to share these insights with other testers and engineers. I’ve written books, conducted webinars, given keynotes, and led interactive tutorials. But most software testers are simply too busy to read or learn because there’s always another thing to test. When they want to apply what they’ve learned, they often lack the time or authority to try out new ideas. The demand for new testers and software testing is growing faster than we can keep up with education. Things just aren’t scaling.
Digitizing Me
You might have noticed I’ve been quiet recently. I’ve been working on digitizing all my learnings in the form of an AI Test Manager bot. A bot that can test people’s apps the way I would recommend. A bot that sounds like me, thinks like me, and tests like me. I’ve even incorporated learnings from other expert testers and taught the bot to avoid the many false starts and antipatterns in software testing. Essentially, I’ve been building an AI Test Manager that manages a team of AI testing bots.
Broad Coverage
Rather than teaching different topics to 50 people at a time, I’ve taught separate bots to specialize in diverse testing areas: accessibility, usability, privacy, performance, content, API, console logs, UI, UX, security, GDPR, OWASP, exploratory testing, visual differences, qualitative feedback, feature sets, and competitive considerations. Even the largest teams don’t have enough specialists to test all these aspects of quality, and each specialist can take years to become an expert.
Test Generation
I’ve taught the AI-Test Manager how to reliably generate test cases for any app. They can now test websites I’ve never even seen, creating novel test cases that often surprise me with their creativity. I’ve trained them to create useful test cases, including positive, negative, boundary, user flows, edge cases, and even the boring but necessary tests. They can even emulate the test thinking of other schools of thought as a curiosity — bizarre they may be.
The test case descriptions generated by the bots sound as if I had written them, including the motivation for the test, detailed reasoning of why it’s important, and even hints on how to evangelize the need to fix the issue if the test fails. Instead of writing a guide to great test design, the bots can now generate great test coverage for almost any app.
Test Execution
I’ve also taught these AI Testing bots to dynamically execute the tests. They can code and execute the tests they generate. The tests are written how I would write the code — super defensive, with logging, checking at every atopic step, and recording details of the whole flow. These are automation code best practices I could write up in a blog post or book, but now the bots simply do the right thing on every website.
Qualitative Feedback
The AI Testing bots create user personas and generate qualitative feedback on areas like design, emotional connection, competitive analysis, visual appeal, and clarity. The bots can even suggest features that are missing, but they would expect. Instead of teaching classes about testing more than just functional and quantitative quality, the AI test manager can synthesize user personas and deliver this feedback to every website in the world.
Bug Reporting
When the AI Testing bots find an issue or bug, they can also guess at the root cause, often suggesting a code fix. This is a skill that usually takes years of testing and coding experience, requiring an understanding of the full stack. Rather than mentoring folks for years, the AI Test bots can do this out of the box, on every website.
Grading and Summarizing
The AI Test manager looks at all the test coverage, issues, performance, and competitive apps and gives the app a simple letter grade, and a pointy-headed-test-manager-type text summary of the app quality for management. The AI even factors in what type of app it is and weighs the info based on analysis of millions of app store reviews to know what quality issues real world users care about most. (from all the data in the App Quality Book).
Smarter than Me
Surprisingly, the AI Test Manager is getting smarter than me — I was hoping for just ‘OK’. The AI/LLMs not only know everything I’ve written and everything written by other testers, but they can reason about all that knowledge at once and apply it to the testing problem at hand. I can’t remember everything I’ve learned or experienced. Embarrassingly, sometimes, I remove some of my specific guidance, and the AI bots get smarter. For example, when building a bot to check for visual differences, it handled dynamic content changes (like Google’s doodles) better than I had anticipated.
Speed is King
The AI Test Manager is faster than me — testing an app in about 30 minutes. It would take me days to a week to plan the testing and deliver the same level of coverage. It’s also far less expensive than my consulting rate, making it an accessible option for many companies.
An AI Test Manager for Everyone
In most respects, the AI Test Manager has surpassed me. It has tested more apps and found more bugs than I have. That’s a happy surprise, but I’m still dealing with it emotionally. Now, I’m looking to share this AI Test Manager bot and its team of testing bots with every app in the world. If you’d like an AI Test Manager to help you with the testing of your website, sign up @ https://www.checkie.com.
If you’d like to peek at some of the test results, check them out HERE
— Jason Arbon
