Testing Value

The most difficult problem in software testing isn’t testing, or creating value — it is communicating the value of the testing. Knowing what to communicate, how to communicate, when to communicate, and who to communicate with, is an essential skill that few have mastered. Too often testers make communication complicated in an effort to disguise the lack of value, or they just overcommunicate. Ironically, less is more, so it is easy to communicate testing value with just a few principles.

What Testing Value to Communicate

The value of testing really boils down to 3 simple things:

1. Bugs
2. Coverage
3. Quality/Risk Assessment

Testing and test results are boring —even to most testers. Sorry to break the news. No one really wants to hear what was clicked or what API was called or look at some dashboard with disappointing CSS. Even when actionable, bugs just mean more work and delay to the rest of the team. The more you try to explain the value of testing — the less value the team will perceive. Here are some examples of communicating the value of testing:

Good Examples:

1. Bugs: “Two blocking bugs, both in the signup, but note they are only in the new marketing page flow.”, and let the team ask for more details.

2. Coverage: “You asked about the quality of the shopping cart functionality? We don’t have great coverage in the shopping cart area, so I can’t really say, but that is next on our list”.

3. Risk Analysis: “This build seems to have more risk than value, recommend we delay ship, but OK if there are other reasons to push.”

Anti-Examples:

1. Bugs: “We have 134 Active, 28 Resolved, and 10 issues still not triaged. The active count is going up. The time to resolve issues is going up too. Let's walk through the top priority issues.”
2. Coverage: “We have 95.6% of our exciting tests executed. And, we’ll be adding more next week, which will bring us to 81.5 % code and 90% feature coverage.”
3. Risk Analysis: “We should not ship this build. It is very risky for the business and we aren’t sure what might break in production.”

How to Communicate Testing Value

Surprisingly — testers aren’t usually the best communicators. They are rarely the orators or poets. Worse — the marketing of testing tools and vendor services that promise 100% percent coverage or generate 1000's of test cases, or pay-per-bug, or ‘10X faster’, just confuse the consumers of testing data and measuring value in general. All those ROI calculators are really just sales tools. Don’t sound like them.

Most of what testers read is about over-communicating their value to the stakeholders. Don’t. It's just annoying and makes folks who allocated the budget for you and hired you and your team start to question why they did that in the first place. In software testing — not communicating is often the best option.

That said, always be ready to report on the value of testing. Always have the answer to the following:

1. If the current build is better or worse than the released version
2. If there are any possible blocking issues and their current status.

By default, don’t communicate anything else. Not wasting people's time with irrelevant or uninteresting data is better than no data at all.

Be super concise and accurate too. Never use percentages unless they are 100%, and even then, qualify that by “..of the things tested “. If you hear yourself talking — stop. If people want more info, they will ask. When they ask; it is valuable for them to know, you haven’t told them something that wasn’t valuable, the person asking will feel smart for asking, and you will look smart for knowing.

Don’t be sad when no one appreciates all the testing work and dedication — you don’t appreciate all the software design and frustration with code fixes or arguments around business justification either. You cannot competently interpret it anyway — you are focused on testing and quality. It's OK, you should focus on being a great tester, and that is an infinite job and a largely unsolved problem. And, trust your peers to do the same.

Communicate automatically, when possible, via dashboards, or email updates. Spend your face-time with developers, product managers, and executives understanding their assessments of quality and risk — not updating testing status. When you incorporate and address the things they care about — they will especially value your work.

A Story of Communicating Test Value

When I used to work on the Chrome team, the engineering director decided when to release. We released by default internally every day, and externally every week. I ran the automation side of the house. Our test results and the test results from the dev team’s tests were all automated and summarized in a quick dashboard. The engineering director would load this dashboard in the comfort of his office, when he was in the mood to look at it, and had brewed his coffee.

In contrast, the manual testing team worked feverishly in cubicles nearby, obviously wanting to be seen working hard. On public release days, after looking at the dashboard summary of some 20k automated results, the engineering director would wander by the manual testers and ask if there were any ship blockers. The test manager would talk about how they hadn’t yet finished the full test pass, and there might be an issue but still investigating, and you could tell his mind might be elsewhere. She hadn’t answered his question — the simple question related to the manual testing team’s value in his eyes. He would sometimes walk away while the full test pass details were being rattled off, and press the ‘ship’ button from the quiet of his office.

When I dropped by this same office every few weeks, he was happy to chat. I used the time to ask what he was worried about. He said his biggest stress was political — when the builds broke internal beta products it was embarrassing. He also was concerned about the seemingly random, and unreproducible issues reported from anonymous users/usage in production, but it felt so out of our control. In response, I met with the internal product teams and asked for their tests so we could run them before releasing new Chrome builds internally. I also spun up crowd testers at Applause.com to try and reproduce end-user reports and test with “dirty/real-world state” in their browsers. Management obviously valued the efforts and results, and the product was far better for it.

Testers will be valued more, and deliver more value if they talk less and listen more.

When to Communicate Test Value

It is surprisingly obvious to know when to communicate the value of testing — when something very significant regarding quality changes, or when the team asks for updates. The worst timing is when the testers want attention for attention's sake. When testers share possibly relevant info, but at irrelevant times, it will be viewed as a negative value to the team.

Even when it is a relevant time, say the end of a sprint, don’t repeat yourself — if nothing has changed significantly, just report “no change” by default. Don’t list everything for listing's sake. Don’t mention things that aren’t relevant (e.g. new low-priority bugs). It is very tempting for testers to fill time and space and feel they need to ‘talk’. Less is more and more valuable.

Sometimes it can make sense to proactively communicate testing value quarterly or after a significant release, especially for testing vendors. The best example I saw of this was someone at Applause, a crowdsourced testing company, where the test manager would proactively offer an optional quarterly assessment of testing. Only people who cared needed to show up and he made sure that folks that didn’t care did not show up. He presented some basic trends on bugs, coverage, and risk over the past quarter. And most importantly, a quick slide highlights the biggest risk avoidance due to either bugs that were found and fixed, or coverage that was built or executed. He’d point out that things could have gone wrong but they either were detected or could have been detected — a summary of risk mitigation, which is the ultimate value of testing. Proactive communication of value is super important when in a vendor/ contractor role as the dollars spent on testing are counted more precisely and scrutinized more diligently. But still, brevity should be king.

Who to Communicate Test Value With

Testers often feel unappreciated, but instead of adding value, and speeding up the team, they add negative value by sharing updates on their value with people who don’t need to know. Not everyone needs to know about testing value, just like not every tester needs to know about marketing or finance value. Most testers don't even care about other testers’ work — I have some scary stories on that topic!

OK, I will share one of those stories, but I probably shouldn’t. Keep in mind this is obviously just my myopic and biased perspective of what happened. Back at Google, when I worked on Google Desktop, we had these regular meetings of testers across products. It seemed like it made sense for us to share learnings, maybe code, maybe processes, who knows. This went on for a little while and seemed semi-helpful, but really just the comradre and empathy were what people were there for. As with all engineers, even the testers soon started wanting fewer meetings and a more automated, and less synchronous way to share status. So, it was proposed we build a tool to let folks ‘tweet’ their status, put that into a feed and people could quickly scan and ignore what wasn’t relevant but if they saw something interesting, they could follow up.

Participation in the tool was weak, so in the next regularly scheduled meeting, people were talking about how to increase engagement. The idea of making the tweets a requirement was proposed. All seemed okay, but it started feeling a hint Orwellian and a tad Lord of the Flies. One person spoke up pitching that not everyone should be required to tweet their testing status — sometimes there wasn’t anything worth tweeting beyond their product team. The room started to grumble and argue that it was ‘just a tweet’, it best to keep them regular/required on the off chance there was something useful. This person, who worked on a smaller, more isolated team, seemed to get upset. So, our hero says ‘Hey, maybe we should just leave it up to people to decide whether to tweet. Heck, people like <redacted> might work on teams where no one needs to know what they are doing some weeks, and they know that no one would care.”.

Uh oh…I awoke a giant and I was pinned against the window in the conference room with chairs on both sides. The only door out to the climbing wall and reception area was blocked. <redacted name> stood up, pointed at me, and loudly proclaimed “People do care about my work and what I do!!!”. And <redacted> seemed to literally leap across the table. Not kidding, if I was in range, I would have got a black eye. The other folks in the room quickly shuffled and stood up, and three kind humans stepped in between and worked to cool tempers down. I was confused and shrugged mumbling “I was just trying to stick up for <redacted>….”. Folks ushered <redacted> out of the room, red-faced and all, toward the lunch room. Yes, this person had larger issues. That was close, I would have had to take the punch :-)

Lesson to take away — not even testers want to hear testing updates, and most do it to feel important, not for the people they are updating. Keep it brief and only share with the people who need to know and would care!

Even worse, some testers want to talk about things that are irrelevant (not one of the three topics of testing value above), and instead want to talk about “learning the product”, “learning how to code”, or making sure other disciplines know how hard they are working. No one cares about your personal growth story except maybe, maybe your manager will feign interest.

Only tell those who need to know, when they need to know about your testing value.

Some Additional Communication Don’ts:

Don’t leave your lane. You will hear some folks suggest speaking in terms of the “business”, or in terms of business value or risk. Just don’t. You will never quantify the business value of your testing — because no one else has legitimately ever figured that out, anywhere. The team already questions whether you have a handle on the true business priorities — don't open your mouth and remove all doubt.

Don’t use metrics and graphs — you may think they make you look smart, but in testing, the underlying data is often on shaky ground as the samples are small and constantly changing. Use metrics and graphs for yourself, maybe, but they at best distract your audience and expose the qualitative nature of your work.

Don’t discuss ROI. Everyone knows this is a fake number. If you could figure this out quantitatively— you might literally win a Nobel Prize in economics, and the Turing Award to boot. It’s actually okay not to know the ROI. Note that it is also difficult to value the new feature work of engineers or even determine the value of the company itself, even with detailed financial reporting. Admitting you can’t calculate ROI it will only earn you the respect and trust from people who matter. They want to know you are doing your smartest and best effort to reduce risk.

Don’t Invite Yourself. You will hear people suggest you try to participate in more planning or strategic meetings. Your ego likes the idea — but that’s not you. You will only look like a fish out of water. You will be invited in if you would add value and communicate well. If you ask to be included, know that it is almost always just a courtesy and an annoyance that they let you in. More cringe and less value.

Don’t Avoid Assessing Risk or Quality. It is a difficult thing, but other folks on the team, and the collective won’t do as good a job as a great tester in assessing and communicating quality and risk. Don’t shy away from the responsibility. Shipping software is more a social than a computer science — don’t be too stubborn avoiding assessments of risk and quality or you will lose the trust and collaboration you need to do your job.

When Testing Value Isn’t Appreciated

If you work somewhere that doesn’t care about your bugs, coverage, or risk assessments at all — their software and the business probably aren’t great either. It definitely won’t be a fun, pleasant, or growing experience for you as a tester. If you follow the rules above but no one cares when you do communicate the value of your testing, you should find a better team or company to work with. Be in an environment where you actually value, communicate it, and be appreciated for it.

Summary

Overall, it is pretty simple. When communicating testing value: less is more and focus on adding actual testing value. The value of your testing should speak for itself.

— Jason Arbon