AI Generates Test Cases
This isn’t a revolution, or the end of manual or automated testing— just some test-nerd fun.
No one really likes to create the 100th test case. I’ve spent a good fraction of my life on this planet entering test cases into a test case manager or spreadsheet :) Some of us have dared to wish there was a point-and-shoot robot that looks at a web page, and automatically creates test cases. This is starting to happen thanks to advances in generative AI. Who thought machines would be able to do this just a few years ago?
People ask ‘where is the AI in testing’? And, how can it help outside of automation? This is another example. My goals in sharing these results are to answer some of those questions with ‘data’ rather than marketing speak or hand-waving. I also hope to get some feedback on the generated tests.
The AI-Generated Test Cases have the usual things like a title and a description and a priority. They also have two properties not usually in test cases:
- Why Important: AI explains why this test case exists. Helpful when it fails.
- BDD / Cuccumber: Actual ‘steps’ that are easy to read, or import into other automation.
Efficiency: With paralellization turned off, it still runs in about 1 hour per page. It costs about $1 per page for ~600 test cases. Just interesting metrics.
Someone smart suggested that it might even be useful for testers to scan areas they have already tested to see what things they might not have thought of already — or spark creativity.
I’d love to hear what people think about the AI-generated test cases below. Really, any opinion, technical or emotional, positive or negative, would be interesting and invited. These are interesting times, and I’m hoping the testing community can collaborate a bit while navigating these waters. Also, if you want to see what test cases the AI generates for your webpage, just let me know and share your URL in the Feedback form. Keep in mind these are the early days — but things are moving fast :)
Quality of Generated Test Cases
I’ve only skimmed these test cases. Not because there are a lot of them, but in the interest of sharing some raw, unvarnished data, and looking for unbiased/altered feedback.
I would suggest the best way to measure their quality would be for YOU, the reader, to write down test cases for these various pages yourself. At least think about the categories of tests you would write, and the compare that with the coverage from the AI. Consider what you missed, consider what the AI missed, and consider what was superfluous.
I’ll proactively say that anyone that and disparages the AI without doing such a basic test — may not be all that much of a tester :). I’d really welcome skepticism, even a negative/tough bias, but at least be a professional tester and “test” it to some degree. It is amazing to me how little testers are testing the delta between AI and humans in these interesting times. #irony #lonely
Oh, and for folks that don’t religiously believe in test cases, or scripted testing, AI does an interestingly swell job of pair-exploratory-testing testing. You might want to check that out and test what it is like to pair with an AI to enhance your testing. Let the world know what you find.
Google Home Page
I had the AI generate test cases for the Google.com home page, since we are all pretty familiar with it. The AI generated over 600 tests! I was thinking maybe 50., and saw some in there that I wouldn’t have thought of myself that I probably should have.
The raw spreadsheet export containing the test cases is HERE.
STARWEST
The STARWEST testing conference is coming up and should be interesting. I can’t make it this year, but figured the next best thing a tester could do is generate test cases for it :-) Here are the STARWEST home page test cases.
PNSQC
I also don’t think I canmake it to the Pacific Northwest Software Quality Conference (PNSQC) either, even though they have quite a few AI-Testing sessions. It woud have been fun to give a talk on these test cases, but why go through the middleman?
The spreadsheet of AI-Generated PNSQC home page test cases.
I’d love to hear what people are thinking about AI Generating test cases in general and any suite or test-specific feedback. This is some test-nerd-cool-fun stuff. The world of testing might just be changing after all :-)
— Jason Arbon
