The Lost Interview: Testing Trends 2020

A testing/agile company asked the following questions but didn’t publish the QnA — perhaps because my take on ‘agile’ wasn’t in-line with their overall message? Maybe it wasn’t ‘vendor-friendly’ enough? Most likely they just forgot or didn’t care enough about the answers to bother publishing it. I prefer to imagine a conspiracy :)

What were the trends that have come to pass this year with regard to test automation tools and best practices?

AI for software testing is now a ‘thing’. Some want to ignore it, some embrace it, but it is no longer just a curiosity and it can’t be ignored. In 2018 there were a few brave souls applying AI to their testing problems. In 2019 there are now multi-person test teams at the largest corporations turning to internally developed AI solutions, or applying vendor-based solutions. 2019 was the year to educate yourself on the basics of AI — if you don’t at least have an intelligent-sounding opinion on AI for testing, you are falling behind as a leader and technologist.

Selenium 4 and Appium continue to dominate test infrastructure. We all owe a huge hug and thanks to the opensource engineering teams that power most test automation on the planet — they’ve both made great progress this year with new features and stability improvements.

Shifting left and right. Over the past few years, folks have tried to shift testing “left”, earlier in the process, others suggested shifting “right”, toward the customer experience. Both have been proven right in 2019. With the adoption of more agile and continuous integration, lower-level tests need to be written earlier as there is no other time to write traditional automation or (manual regression passes) before the software is deployed. The rise of the importance of user experience (UX) as most software products and companies are commoditized by the app stores and creativity in design has become more incremental the importance of testing from the perspective of the end-user (human or machine) is more important than ever. In the wake of shifting left and right, is the unmentioned failure of all that traditional testing ‘in the middle’, which either has very low returns on investment or often produce little to no results at all. The absence of ‘middle testing’ is the unspoken trend of 2019.

Not enough great testers. There just aren’t enough great testers in the world, especially in 2019. Too many of the competent people leave for different disciplines in search of higher pay. Too many teams are moving too fast with such low engineering quality that there aren’t enough humans to catch all the bugs. Despite all the hype around automation, AI and the end of dedicated testers, the demand for testers is growing something like 20% every year, and percent of total engineering budgets spent on testing has kept increasing to about 23% today. This trend will continue.

What were the duds?

Vendor marketing teams that claimed there was ‘AI inside’, forcing their engineers to scramble and hack, and disappointing potential customers, most vendors failed to meaningfully add AI to their products. This is of course expected at this point of the hype cycle, but it was painful to watch in 2019.

Lack of creativity in how to apply AI to testing problems. Too many AI curious folks focused on incremental value versus re-thinking the entire notion of how to perform testing in this brave new world. This is a dud because it led to so many disappointments, versus high expectations in AI-powered testing in 2019. Fixing element selectors when they break is incremental, test selection is incremental, logs analysis is incremental — AI is a transformational technology, not an incremental one.

Agile is now ‘tired’. There are so many varieties of practices named ‘agile’ that engineers have simply grown weary of it. During 2019, agile has devolved into the simple message of just shipping as fast as possible. Agile has taken its toll not just on software quality and design creativity, but frankly, there has been a human toll. Old school waterfall is making a fashionista comeback, if only so engineers can plan their personal lives and not live in continual stress. I’ll add that I think agile is keeping the best and most diverse talent out of our testing profession — only so many people can tolerate 24/7 engineering accountability and availability. At this point, we should rethink all this ‘agile’ business in the interest of humans.

Specific testing philosophies and schools of thought have grown boring in 2019. Evangelists of single approaches to testing have either died on the vine or the new ones proposed have been met with a large consensus yawn. The testing community has seen too much and knows too much as of 2019. The best testers of today masterfully blend of aspects of all approaches their specific situation. Humility is in, hubris is out. To ironically cast this as a philosophy, I think of it as ‘Postmodern testing’ given its similarities to postmodern art which draws upon all previous schools of design to build something even more interesting.

What will the most important top trends and challenges be for 2020 with respect to both open source and commercial test automation tools?

Open source tools will continue their dominance in adoption given they are the right price of ‘free’ as many test teams don’t have budgets and don’t know how to ask for budgets, for testing tools, and it is super easy to get started.

There will be more, but little used, ‘open source’ testing tools as the new resume for test automation engineers is to have a Github.

Open-source tools will continue to suffer from a lack of resources. The folks at the core of these projects are genuinely just very talented and good humans, but so few dollars are actually invested, and the open process is so slow (congrats to folks for W3C standardization BTW: https://www.w3.org/TR/webdriver/), opensource is at a large disadvantage with the rapid evolution of AI-based approaches.

The new innovations in AI are very expensive to build and monetizable so there is a significant disincentive to opensource this new wave of test automation advances in 2020.

The blame for ‘flaky’, ‘slow’, ‘expensive’ tests based on opensource frameworks flared up in 2019, but will continue into 2020. The frameworks blame ‘bad programmers’, the ‘bad programmers’ blame the framework design that encourages bad programming patterns. As for me, you can infer my opinion, but I try to stay out of the debate and simply claim the root problem is that people have to program at all to write great automation :)

More hybrid tools that blend opensource work, with some proprietary tweaks so they can market and monetize it. These tools will be forgettable in the test of time.

A couple vendors will go ‘all-electric’ and have truly AI-first solutions in 2020. The rise of AI-first tools built by leading test teams for application-specific or internal testing challenges. These will remain quiet because they won’t need much marketing, or viewed as secret as they are competitive advantages.

More hype around AI. Despite fumbling in 2019, the marketing of AI-based automation claims will only increase. We’ve already seen claims of tools that can generate near “100% coverage”, or “100X faster”, or “neural nets can’t be used for testing”, or “we use over 100 different AI technologies”, or perform other miracles in 2019. Just like reality television, AI-marketing hype will move from cringy in 2019, to full-blown guilt-inducing entertainment in 2020. Get some popcorn, it will be a memorable year. Focus on what demonstrably works and what the smarter test teams are doing.

More fragmentation in the test automation tools world leads to more confusion. The adoption of commercial tool solutions will be slower than many hope. Testers in 2020 will have too many incompatible partial solutions. Testers and their managers really just start dreaming of one platform to do it all.

RPA (robotic process automation) will add to the fragmentation and confusion in 2020. RPA companies have raised a lot of marketing money and are exploring the testing market. RPA is good at automating apps that rarely change, often legacy systems, and doing the same sequence over and over. Great test automation, on the other hand, has to deal with constantly changing apps never seen before and still delivering coverage, not repetition. It will take a full year for the market to figure that out.

The real trend in 2020 will be less focus on the AI in systems, and more on the raw benefit of these solutions and teams looking for help from vendors to combine the best of AI and traditional solutions to deliver value at speed.

2020 will see the rise of quality focus by the platform teams (the Googles, Microsofts, Apples, Amazons) to start defining, enabling and requiring basic testing. In 2019 they all made progress in basic testing and security scanning when apps are submitted for release, but this trend will pick up speed in 2020. The platforms care more about the quality of apps than the app teams…they have the money, they have the tech, they won’t leave app testing up to the little adhoc and underfunded test teams spread around the world. Get ready for massive progress from the ultimate vendors in test automation — the platform companies.

— Jason Arbon, CEO @ test.ai