Testing Just Shifted To the Future

I’ve been chatting with test professionals, investors, CEOs, and engineering teams the past few weeks to see how the recent work-from-home and macro environment has been impacting testers and testing activity. It seems that many previously slow-moving trends are accelerating, and will likely have a significant residual impact after things seem to be headed back to ‘normal’.

There have been major shifts before; desktop, web, mobile, and cloud. You could also argue waterfall, agile and everything in between were also shifts. But those were simply technological and process shifts. The world of testing just got nudged pretty hard by world events. Everyone has felt the shift, but may not realize the long term impact on the profession. We are in the acceleration phase of a new shift, one that will impact who, what, where, when, why and how testing will be done in the near future. The recent mass experiment of working from home, under pressure, under economic shock, with realtime changes, under government watch, the software now considered the critical infrastructure, and a lot of ambiguity will change the face of testing forever. Sure things will eventually normalize to a degree. But, these shifts were already underway, and the world just accelerated them for us. These shifts will be the new normal soon. Best to get ready for these shifts today.

Who is doing the testing

In this new quick-turnaround, high-pressure world, the developer, product managers, and users are doing far more of the testing. The dedicated testers are there, sure, but often checking deployments after the fact because their manual test passes take too long, or the changes in their applications broke their regression automation and didn’t have reliable test results for the team to evaluate quality before the world and the business needed to ship. Developers are testing their code before checking in, running unit tests, and manually checking staging and production for risky aspects of the software related to their changes, hand in hand with product managers.

The testers are doing their normal work in the background, often after the software has been released. Yes they find issues after days of grueling manual regression or updating their test scripts, and the issues get fixed — but they are getting fixed after the fact in many cases right now. The old adage of ‘bugs are more expensive to fix when found later’, is being replaced by ‘it is more expensive not to ship quicky, then fix a few things later’.

The long-trending shift toward testing in production is accelerating to the point that many critical issues are being found by the users. Users find websites are erroring out when the load on them is high. Some companies are experiencing a surge of 3–20X usage, and these users are encountering horrible usability, security, and reliability issues. Some of these apps even have the FBI issuing warnings. Many of these issues were likely known, but quiet and rare, or the testers were so busy running regression tests that they didn’t do load or exploratory/negative testing. The end-users, even the government, are now testing software in production.

An important question is who are the best testers for this environment — for the future. Testers that are highly adaptive, creative, great communicators, and not stuck to old, or any particular dogmas are the best testers to have on board. Some testers debate the minutiae and miss the forest for the trees. Some testers argue on automation versus manual when the answer is both. Some testers debate the semantics of ‘testing’ vs ‘checking’, or point out that automation is actually manual because coding is ‘manual’, or endlessly discuss what really is ‘AI vs Machine Learning’ or argue that ‘no product can be fully tested due to combinatorics’. The world needs more testing regardless of dogmas and ego. I’ve not seen any high-quality product teams recently where these debates are raging. The world needs testers who can analyze all aspects of the problem, familiar with all forms of testing and quality activities, apply creativity under time constraints, and ensure that software can work reliably under very demanding circumstances for users, for the business, and for the world. Oh, and teams are realizing they can get by with fewer testers.

What is being tested

With all this transformation and chaos, what is being tested is changing. The urgency to update software today means there is little time for exhaustive testing. Exhaustive testing really never was a thing because it is combinatorially nigh impossible. The shift to ‘agile’ has been painful as there wasn’t much time in a sprint to get the testing done anyway. Today’s world is highlighting this problem as teams must ship on external timelines such as ‘yesterday’ — there is little to no time to test the software before it is deployed. In this world there are now three primary threads of testing by the app team:

1. Developer written and executed Unit and Integration tests, run before checking code in and on every new build/deployment.
2. Whole-team testing of high-risk areas in the product based on recent changes, often in production.
3. A background thread of software testers doing more complete testing, after deployment, and the bugs found are simply resolved as hotfixes.

Formal testing now happens on an asynchronous, longrunning, lower-priority, background thread. Testers still test the bulk of the application, but not every build, and not every new feature. Their bugs are found and triaged often after release and only the most critical bugs warrant a hotfix in production — or even a fix at all.

Testing is shifting into three modalities: every more developer testing, exploratory testing around the latest changes, and background testing of the rest of the application which ‘should’ still just work but needs to be eventually checked just in case.

Where is testing is happening

Lately, far less testing is happening in the office, and much more is done remotely from home. The best testers often build relationships with the developers and product managers by hanging over their cubicle walls, trying to sit with them at lunch in the cafeteria or the cafe down the street — all in an effort to understand what is changing and building rapport to get their bugs fixed. All that automation running on the cool looking lab machines in the corner doesn’t look so cool when the rest of the team can’t see it. Test automation is shifting dramatically to cloud test execution, and for applications that need to be tested on-premise, only the ‘lights-out’ labs will survive. Manual and automated test execution is shifting radically.

Manual testing is now done in the isolation of testers’ home offices, and word is that they feel far more productive without all the interrupts of daily office life. But, testers are failing to realize that their peers are losing contact with them. Testers are losing the friendship and confidence they often need to argue for bug fixes or convince the team that there is a risk with shipping the new build. Online tools are great on the surface, but we are all still emotional, physical animals at the end of the day — heck we even respond to pheromones!

Testers may feel they can create or execute more tests every day — but again, the rest of the team frankly doesn’t notice. They never really did notice or care about how many tests were created or executed, but now they don’t even see the testers toiling away early morning or after-hours in their cubicle. The primary output of testers that the team responds to are bugs — meaning most contact testers have with the rest of the team are on bug reproductions and triage, also known as ‘bad news’. This is further shifting the test/developer relationship into one of negativity and ‘throwing it over the wall’. Testers need to be acutely aware of how they are perceived when testing is happening out of the view of the rest of the team. Testers should focus on as much non-technical communication and comradery as possible with the online tools used by the team, and invest time in making sure their effort is visible.

Test automation is most often performed in a lab behind the corporate firewall, or on public cloud infrastructure. With many on-premise labs in empty office buildings with no testers to physically attend to rebooting the machines, adding more machines, or debugging them, many of these labs are dying on the vine. For those companies with permissive security rules, many testers are moving their basic test execution to the cloud lab providers. This has been a trend for many years, but many teams are now forced into this shift. For those teams who must have their test labs on-premise, rather than let their test automation grind to a halt, they are working on configurations of their labs that enable ‘lights-out’ operation. This is a combination of remotely addressable powerstrips, monitoring, container-based environments, direct connections to hardware output, and easy product-specific remote configuration and debugging. Test automation is rapidly shifting to modes where the tester needs zero physical access to the machine executing the test cases.

When is testing happening

Testing is now happening continuously, with a blend of ad-hoc testing in production, pre-production, automation labs, and long-running manual regression and exploratory tests. When software is so critical and when shipping so quickly, what used to be controlled and serialized test cycles are now blending into a continuous mesh of testing activity.

In waterfall testing, the test cycles could be months-long coordinated efforts where testers never could create or execute enough test cases. In agile, these turned into weeklong sprints, with testers worried about even having a chance to get some tests developed for the product before it releases. Today, due to urgency and speed, the software is being shipped without much of a chance for formal testing at all. Instead, software teams are relying on monitoring of operations, user feedback, and testing in production to identify the most recent regressions and bugs.

Testers are often completing the manual regression and exploratory testing *in production*. When bugs are found in the product, the related features are either: hot-fixed in production, turned off via feature-gating, and/or the ‘flight’ of the new build is halted or rolled back to avoid the issue affecting any more users. Teams that have invested in these technologies the past few years have been well prepared for these extreme times. Other teams are playing catchup quickly. All teams will need to be continuously testing and deploying in the near future.

Why is testing happening?

In regulated industries testing is compulsory — everywhere else, the need and justification for testing have varied wildly. Some app teams invest in testing because they think they need to, some as a reaction to bad ‘bugs’ in the past, some out of habit, some to avoid accountability when the product does fail, more because of some loosely defined business/revenue risk concerns, and even fewer because they took pride in the quality of the software they deliver. There really never has been a mathematical justification for testing.

That has all changed today. When customer habits are changing this rapidly, downtime, security, privacy, functional, and even performance bugs can lead to a huge lost opportunity. Customers are choosing video conferencing services for example, based on the quality of the user experience, reliability, and security — quality can mean a difference between a 2X and a 20X increase in user growth at this time. All those testers who seemed worrywarts about testing their product under extreme load and in production are now seen as heroes as their products are not only holding up but letting so many new customers in the door. Quality is now also seen as a relative thing — the best video conferencing software will win out — you need to not just be great, but better than the competition, which means testers need to understand the quality of those competitive apps. This is the beginning of the hero tester — the ‘test pilot’.

Even more importantly, the world is now realizing that software isn’t just a thing that runs when you turn on your computer and forget about it when it is turned off. Software is now critical infrastructure for companies, countries, and important for humans to connect and collaborate with each other. What used to be software annoyances are now interfering with the operation of society, talked about in the news cycle, mentioned by politicians. The CEO’s of companies are now focusing on quality above features for once! Some operating systems have put a full stop to all new feature work, and focusing only on quality and reliability as millions of students turn to remote education — these teams not only need to capitalize on the business opportunity, but also feel a moral responsibility to ensure their software is as high quality as possible.

Testing, and more importantly the quality of software in general, has finally become more important than new features — perhaps a first in the history of software (other than the Chrome Browser, yeah, I had to plug that, and they won on quality before it was cool). The teams that realize they now work on a utility, a public service much like water, electricity and policing functions, and that quality is more important than feature creep will be the best prepared for the world to rely on them in the future.

How testing is done is shifting

The world of testing has been incredibly fragmented, with many different tools, languages, best practices, degrees of testing, on-premise, cloud — really testing has been up to the individual tester a team happened to hire and their particular view on the world. Some test teams super focus on manual testing, some automated, a blend, or no testing. All aspects of how testing is done is changing.

Manual testing is just too slow for regression testing. Period. Modern software teams leverage CI/CD and have wanted to know the quality of each build, even before the code is checked in. Many did it because this was cool, or knew this was the future. Manual testing couldn’t keep up. Now we are seeing software teams dealing with the need to change functionality on an external, urgent timeline highlighting the problem. Recently, many companies have had to update their web pages and apps to deal with these near-real-time changes — the only thing they could do was cross their fingers. Rideshare apps had to update their features and app flow to remove the sharing of rides, banks are rolling out new forms to apply for federal loans, games are adding as much purchasable content as possible for all those folks pretending work from home. Software used to be updated when it was ready, now the software has to be updated when the world needs it updated, and manual regression testing can’t keep up, and has left these companies deploying code with minimal manual testing, and lots of risk. Manual testing has its place in usability, creative exploration, and risk analysis, but, as software becomes critical infrastructure for society, manual regression testing has little place in the engineering flow for qualifying builds before release. This has been the inevitable trend, and it is obvious now.

Automated testing as it is today is too flaky, too expensive, and takes too long to write and maintain. We all know this, whether we admit it or not. This new realtime lower-budget world can no longer justify 6–18month investments in test automation only to have it the automation break when a change is made in the application. Teams need to reassess their practices and move to next-generation infrastructure for test automation that is resilient to changes in the application, easy to write, reusable, reliable, and fast. <no…ad…goes…here>

Quantification of Quality

The real world has been filled with daily updates of graphs, statistics, and phrases like ‘flattening the curve’ and even ‘exponential growth’. Nearly every person, even outside of engineering, is now familiar with these terms. For years, testers have tried to quantify and communicate the quality of our products but our ‘internal customers’ often glazed over when we showed them dashboards and trend lines of bug reports and measures of quality over time. Not only are our peers accustomed to these dashboards now — they are starting to expect it.

Those teams that cannot communicate quality through quantification and visualization now look archaic and out of touch. Testers that have invested in great dashboards and quantifying quality are now enjoying the ability to communicate quality without the reliance on real-time or face-to-face interactions, and their teams are benefiting from 24/7 quality updates and the ability to decide when and how quickly to ship — without waiting for the test lead to lean over their cubicle.

Summary

The testing world has been trending in all these directions the past decade, slowly shifting, but today’s macro-environment is forcing all of us to see what the future of testing will look like today. The Who, What, When, Where, Why and How of testing is being re-examined in real-time. Things will get back to semi-normal for a while, but we’ve all seen the truth of what the future holds and should use this time to prepare for that future. The faster we get there, the better-prepared testing, software quality, and the world will be.