Image for post
Image for post

Much like the book, Thinking, Fast and Slow, testing often falls into one of two buckets, ‘fast’ and ‘slow’. Fast, unconscious-like testing consists of quick checks, with little or no thinking to determine if a test should pass or fail. Fast tests click through test case steps with haste and obviously complete quickly but only focus on what is necessary to complete the test. Slow testing is where the computer, or human, takes a bit more time to measure, compare, or consider how to execute the test case, and to decide if a test should be marked pass or fail. …

Image for post
Image for post
TestOps in DevOps

How do the most well-funded, sophisticated software test teams on the planet test everything? — they don’t! There is a new way to test that delivers great coverage, finds important bugs early, and ensures that core functionality works on every new release. This new way isn’t well described because it feels like a capitulation to some, but it is a direct response to delivering high-quality software that ships every month, week, even every day. The key to delivering modern software with new features and not breaking existing functionality is “TestOps”.

“TestOps”, much like DevOps, is the convergence of testing with operations and development. We know most testing today is too slow and unreliable to keep up with development and operations — even in the best of situations. Yes, DevOps often handles API testing and unit testing that quickly verify isolated portions of a program. Still, we all encounter major bugs and regressions in production software. What has been missing is quick, automated, end-to-end testing where the entire product is tested to make sure that it all works together and for the end-user. …

The AITF (AI Test Framework) Community Edition has been in the hands of a few brave souls the past two weeks. Our enterprise customers have the team to get things setup and running seamlessly, but we are deliberately letting the Community Edition folks try it without our help, to make sure it is ready for the masses. …

Too much energy, creativity, and time are spent debating ‘testing versus checking’ and ‘manual versus automated’. Testing philosophers wander town to town, blog to blog, webinar to webinar, sowing confusion and angst. The motivation is not the pursuit of Truth. They are squeezed by modern trends in software testing and trying to preserve their role as experts from a time long gone, rather than embrace and extend modernization. The tragedy is that these debates really do confuse and detract from the art and practice of software testing.

Those senior testers, who dress to stand out, need people to revere them. They don’t believe people will appreciate their intellect if they are associated with rote, repeatable testing. No one will pay for their seminars or training. This view is both wrong and demeaning to the very audiences they are appealing to. Ironically these testing philosophers are far better at testing software than they are at Socratic debate. …

I’ve been chatting with test professionals, investors, CEOs, and engineering teams the past few weeks to see how the recent work-from-home and macro environment has been impacting testers and testing activity. It seems that many previously slow-moving trends are accelerating, and will likely have a significant residual impact after things seem to be headed back to ‘normal’.

There have been major shifts before; desktop, web, mobile, and cloud. You could also argue waterfall, agile and everything in between were also shifts. But those were simply technological and process shifts. The world of testing just got nudged pretty hard by world events. Everyone has felt the shift, but may not realize the long term impact on the profession. We are in the acceleration phase of a new shift, one that will impact who, what, where, when, why and how testing will be done in the near future. The recent mass experiment of working from home, under pressure, under economic shock, with realtime changes, under government watch, the software now considered the critical infrastructure, and a lot of ambiguity will change the face of testing forever. Sure things will eventually normalize to a degree. But, these shifts were already underway, and the world just accelerated them for us. These shifts will be the new normal soon. …

Image for post
Image for post
Automated Game Test Steps

I used to tell my team that “games are just too hard to tackle right now” and would tell potential investors “we don’t test games…” Today, video game teams and investors are glad they ignored me.

Video games are perhaps the toughest pieces of software to test. Unlike apps or web pages, many games have an almost infinite number of states, incredibly custom interaction models, and release updates all the time to a finicky audience. Luckily, in the past year, AI has evolved to address many of these testing problems. …

I landed at the Googleplex with an idea of how to make Google search just a little more relevant for everyone. The opportunity to impact the discovery of information and ads, at the scale of Google was electrifying — but would anyone listen? Every Noogler was expected to spend the first week indoctrinated in class rooms. Instead, I walked the halls of building 43 physically hunting down the heads of search engineering with a PowerPoint outlining my naive ideas.

I soon got an audience with the head of all search at Google. On slide 3 of my PowerPoint explaining some of the math and possible benefits, he politely asked, “Did you come from Microsoft?” I nodded, blushed, and knew this was going to be cringy. “At Google, we speak with code and data. If you can run some of these experiments and get promising results, just reach back and we’ll talk then.” I wasn’t embarrassed, I was invigorated. I’d found my tribe. I hate PowerPoint. A few weeks later we met again, discussed my code and data, and I soon had a team of 10 engineers working on the project to improve web search. …

A testing/agile company asked the following questions but didn’t publish the QnA — perhaps because my take on ‘agile’ wasn’t in-line with their overall message? Maybe it wasn’t ‘vendor-friendly’ enough? Most likely they just forgot or didn’t care enough about the answers to bother publishing it. I prefer to imagine a conspiracy :)

What were the trends that have come to pass this year with regard to test automation tools and best practices?

AI for software testing is now a ‘thing’. Some want to ignore it, some embrace it, but it is no longer just a curiosity and it can’t be ignored. In 2018 there were a few brave souls applying AI to their testing problems. In 2019 there are now multi-person test teams at the largest corporations turning to internally developed AI solutions, or applying vendor-based solutions. 2019 was the year to educate yourself on the basics of AI — if you don’t at least have an intelligent-sounding opinion on AI for testing, you are falling behind as a leader and technologist. …

Image for post
Image for post

It is time to get serious about the field of software testing. In the last decade, software started to power our day-to-day lives, infrastructure, and the economy. The software testing community fragmented as it grew. Under-resourced testing teams, scattered around the world in isolated cubicles, started each new project from scratch, and worked on increasingly complex and diverse software. Software is far more useful and amazing than ever in 2020, but people encounter software failures nearly every day. It is time to get serious about software testing in the 2020’s. If you are reading this — the world needs you.

The term “Postmodern” is borrowed from the world of art and architecture. Like testing, these worlds have had different ‘movements’, tools, fads, and schools of thought. In the world of art think of “Impressionism”, “Baroque”, “Romanticism”, “Futurism”, and even “Dada”. Similarly, software testing has “Analytical”, “Risk-Based”, “Quality Assurance”, “Modern”, “Context-Driven” and other schools of thought. Each school of thought has its insights but also has its blind-spots. Postmodernism in the art world means artists free of any single school of thought, leveraging the greatest aspects of any historical school, and combining them with newfound creativity to build something unique. …

Image for post
Image for post
Credit: Microsoft’s Concept — Future vision 2020

“I wonder if the baristas deliberately slow things down to make it feel special…”

“You are always looking for trouble Angela!”

“Once a tester, always a tester. Just a healthy bit of skepticism Dave. Thanks for the latte, it is a great way to start the day.”

AI Assistants

Both Angela and Dave’s watches blinked with the same simple message: “New build, low risk, verifying. — the bots.”. Angela smiled. She knew Dave was thinking the same thing. Angela used to be the first, and last, in the office, always a little frazzled, always a bit stressed, worrying about everything that could go wrong with their team’s new social app. …


jason arbon

blending humans and machines. co-founder @testdotai eater of #tunamelts

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store