Testers Discuss AI
Thanks to Marcel Veselka , Juraj Žabka for a fun conversation that spanned AI, and the future of testing.
makeITfun YouTube vid @ https://www.youtube.com/watch?v=qUa0hdKtlNE
**Marcel**
Okay, so hi Jason, can you hear me? Can you see me? Welcome to our show. Yes, and thank you. Okay, so let's start with a very quick introduction. I'll do my best and then hopefully you will help me to fix it because I believe there will be a lot of gaps or misunderstanding.
So you, **Jason Arbon**, are the ex-Microsoft ex-Google ex-Applause tester. But last seven years you are an entrepreneur in testing. May I say that in that way? I don't know. Yep. And I would be interested, but we could discuss later whether you are still tester or developer of testing tools.
**Jason Arbon**
I write tests every day and like literally every day, I write tests and testing things. Okay. and writing testing code to test code to test code. You are also author of book or co -author of book, right?
**Mixed**
How Google test books. Two books, okay. That's a self -help book. Okay, self -help book. Another book on how to grind coffee. Really? No. No, it's a joke. A good one. You're a scientist. So another thing is that you are also co -chair of in the association for artificial intelligence for software testing.
**Marcel**
I don't know whether this is still alive or how it goes. I was refreshing the page or exploring the page last days and it looks very same like I saw it a couple of years ago. So that would be maybe also interesting topic to discuss later.
Sure. And I don't know anything about your hobbies if there are any. Maybe testing is not sure I do either. But okay. So that's it. What I prepared and hopefully you could enhance and improve and and fix what I said.
**Jason Arbon**
Okay. I learned from previous ones that I'm not supposed to answer until you've declared yourself finished with the list to be discussed. So are you ready? Yeah. Okay. You got it right. Okay. As my as my family calls me called ex-Microsoft ex-Google.
I'm basically like a serial quitter. Just when my family thinks i have got a decent job, I quit it. That's basically the story like kind of literally my family sees it. I like the term serial quitter.
Yeah. Yeah. And in the serial entrepreneur is don't want to go back to where I quit. What else can I possibly do? So yeah, I would say the interesting, maybe interesting thing to people about those....
... I got into testing because of some projects in college and I was testing the stuff that was supposed to be, maybe it's not very politically correct, but stuff that was supposed to work. I did some projects for missile stuff for Lockheed Martin back when I was in school and it made me really think about code quality, like very seriously code quality, like the stuff like satellite communciations and navigation, you want to make sure that the code's good.
And so I became kind of obsessed about kind of realized how dangerous code is during that senior project with Lockheed. And so I went down the testing path. For many people it's pretty accidental, but for me it was pretty deliberate.
And I thought it was also I'll add, while we are geeking out with test people here is I think it's the hardest problem on the planet. Like it's actually more difficult than a lot of nuclear physics research, string theory, quality or testing.
Yeah, I'm talking about the code quality or testing or well, all of that. How do you know that the software, it means it's the requirements even, right? How do you know requirements are correct? How do you know that the software operates correctly?
How do you know that it operates within parameters, within expected contexts? Yeah, we're geeking out forever. But software powers our lives so much more than when I graduated college, right? Like it's now like our daily lives are run by software, we're on software right now talking.
And I think it's also the stakes have gone up much, much higher than I ever even expected. Like it's more the world runs on software. So we need to make sure the stuff works. But that was a very, very technical.
**Marcel**
It could be very technical. So you can easily to judge whether it works or not. But if you look on the from the human perspective, it might be very subjective. Whether this is good or not. Yep, then the subjective stuff is, from a business perspective, almost more important.
And then, I would argue one thing is that the more technical you are on software testing, the harder the problem is. Like, the more that you think it's a miracle, the stuff works at the end of the day.
Especially when you have all the entanglement between multiple systems these days. Like the complexity we built into these things is insane. If you wanted to build software to be unreliable, you would probably be doing what we're doing today.
**Mixed**
In terms of just training systems and client server, blend software, and now AI. Okay, yeah, yeah. We will come to AI soon, we will come to AI soon. We don't have to keep it in a structure. And so, I was asking about the hobby.
**Marcel**
Do you have any hobbies or the testing is your hobby as well?
**Jason Arbon**
Testing is as actually a hobby like like on Saturday night at 1 in the morning I am usually or Sunday morning or whatever. I'm usually writing some test code Which is embarrassing, but I do like to go to National parks with my kids like we drove through seven or eight national parks in the last 12 months, you know everything from you know freezing winter, you know hikes to Over the Olympics down into you know that literally I wanted to go we went to Vegas and and death Valley In the heat and summer and my kids asked me to leave them in the desert So I I shouldn't probably talk about this publicly, but they want to be left alone Yeah, of course I did so But they well they took pictures of themselves on laying on the ground on the desert sand reaching for water Like that was there fun, but we went through the high of one of the highest peaks in the United States in the lowest peaks in last 12 months.
**Marcel**
I was called that kind of a hobby, but oh, but I don't other than that really not much Okay, cool. So nature traveling and maybe hiking in nature would be the Yeah Okay, I see okay, let's move on now to the hot seat questions Yeah, I will let you write it to run it.
**Juraj**
Yeah, we prepared some question. Well, a hot seat question that you have to choose maybe One option or two short answer from your side that we can start really quickly And the first one is: "will manual testing still be here in 10 years."
**Jason Arbon**
No Well, we have the fall I had a follow question then in 20 years, but I have don't need to ask you 20 years old That's the sad truth, that's the sad truth. Oh no. AI or human? Just in general, those two words.
Yeah, that's in general. AI or human. That's not a tough one for me because like my profile setting, which everyone really follows, there's always been blending human and machines. So literally like the midpoint.
But between the two, I would actually choose AI. I had to make a forced Boolean cast. Okay.
**Juraj**
Yep. Okay, tester or developer?
**Jason Arbon**
Tester. Oh. I don't know if it's me you're talking about or just in general, but 100% tester.
Developers, development is actually pretty boring. You just build what other people tell you to build and then hope it works and people just send bugs your way all day. It's a pretty lame job actually.
**Juraj**
Interesting.
I expected a lot of like feedbacks under the video. Very good. Yeah, why do you think I've been quiet the past year? Okay, corporate or startup?
**Jason Arbon**
Oh, 100% startup. Yeah, 99% startup. Okay, good. But every large corporate was a startup, so kind of the ultimate goal of every startup in a weird way, so it's entangled.
But yeah. Okay. And what was the last time that you created or reported a defect? To somebody other than myself. I'm actually being serious, like I'm not being... Yeah, it was a couple months ago, I filed a bug with OpenAI on their API.
And they... I wrote an actual blog post on it. It was kind of fun because I wrote... I had the AI... Like, I found the bug. I'll talk about it or not, but probably not. But, details. But I had the... I was sort of writing up the description for it, right?
Like why they should fix it and why it's a problem. And I just DM'd the API guy on Twitter or whatever. But I had GPT write the bug report and the justification. And ithe AI actually had a point in there that I didn't think about, and it was far more articulate than me.
And it was the first time, I think, maybe in the history of the planet that I'm aware of, where... Where the AI actually technically kind of filed the bug against itself. And then the humans decided to fix it.
So the humans are actually now in the realm of the kind of the AI in a way. That's kind of... Yeah, that was the first... That was the last probably external bug I filed of interest.
**Juraj**
Okay, good. Microsoft or Google?
**Jason Arbon**
It's a Boolean, right? I mean...
**Juraj**
Yeah, sorry for this question, but we have to do...
**Jason Arbon**
No, this is good. Sorry, and then we're asking today in... I was trying to integrate over time. Okay. And I will additional one probably last one that ChatGPT or Bard?.
ChatGPT.
**Juraj**
Okay. Yeah. That's it. It looks that I have to prepare next time better because you are absolutely okay that it looks that are not really hot seats so that I have to improve it for the next time.
**Jason Arbon**
Well, it's just I have no shame, no dignity. Now it's the time to move on because I over the next 15 minutes if you think of some new ones write some more other ones down I'm happy to answer those are kind of fun.
**Marcel**
Quick questions.
So I was doing some research to make it sound cool. And I found that you quit. Google in 2011 December 2011.
**Jason Arbon**
Oh, you found that?
**Marcel**
It was in your LinkedIn profile. It's not so complicated. But there is a follow up question.
**Jason Arbon**
Is that correct or not? Was that correct? Yeah, actually it's crazier than that. I'll give you the tease. The real tease is this. So I started a startup-- I was working at Bing back when it was MSNSearch just converted to Bing I think at the time.
Back in the day. It was a small team. But we were actually funny thing. We're using AI and Google wasn't using AI at that point. That's kind of funny. Anyway, inside baseball. But the yeah, I created a personalized web search thing because I was so frustrated that my dad would get the same search results that I did.
That was ridiculous. And my dad was searching for like I would search for say "bush", right? And I wanted like Bush the band. And instead of got Bush the president and my dad is a gardener. He wanted like bushes-- things with leaves.
And so I thought I want to solve that problem so that web search was actually personalized for everybody back in the filter bubble kind of days. Yeah. And then I actually funny thing I still own that IP that kind of core IP, what it matters, but but the financial crisis hit and it's a long story.
They got an entrepreneurial story, but I ended up going to Google and working on. I didn't talk about it publicly. I worked on the search. I had a search team on the search team or at Google, which is very.
**Marcel**
Was the was the James Whittaker your boss or not?
**Jason Arbon**
Yeah, for part of the time and actually, yeah, yeah. I mean, absolutely for most of the time. Yeah, but I also had a team that was not in testing, like it was just in the search team was a weird arrangement
**Marcel**
because there was in October 2011, there was a conference.
Google Test Automation Conference. I don't know what you remember. You think and I found one funny video and I would like to share it. 13 seconds or so with you. And then I would like to discuss that.
**Jason Arbon**
I think I know what this is. I mean, is this, was I wearing, was it a onesie or a bikini? Let's see. I would work history videos. Yeah. We all do.
**Marcel**
That is the right, this should be the entire screen.
**Jason Arbon**
Is this the "test is dead" thing?
**Marcel**
Okay. I'm guessing right, but I would like to share it with other, other audience. This is the video.
**Jason Arbon**
I shared an office with James for a long time. I actually used his, when I was at Microsoft, his college kids were interns on my team.
**Marcel**
Yeah, so now I stop it. Hopefully now you can't see that, right? Yeah, I can't see it, correct, yeah. Okay, so now we can continue. So this is one video from the conference and Google had this keynote about testing is dead on this time.
**Jason Arbon**
Yeah.
**Marcel**
And now we are 13 years from there. And not too much in that area.
**Jason Arbon**
You're wrong, actually, like, so this is what, oh, I'm saying you're wrong, I have an opinion. Usually people like to bring that up in the sense that, "oh, I'm still here".
Number one, the best thing ever, if there's only one line answer to that, it's called survivor bias. If you're familiar with the survivor bias thing? Uh -huh, uh -huh. Like, for everybody who doesn't know, it's basically, I don't remember, the basic of the story is that there's one story of survivor bias, which is like in one of the wars, people like to have wars, I guess, but in one of the wars, the airplanes that came back were all shot up, right?
And so they realized they were shot up like in certain parts of the plane. So they made, they were thinking about making those parts of the plane stronger because they were getting shot up. The problem was, those were the holes that were okay to get shot up in, because the people can't.
If you got shot at the other places, you didn't come back. So the fact that you, that plane survived, or that a tester in this mapping, a tester has survived today, doesn't mean that a bunch of other people are not testers any longer.
It also, by the way, doesn't account for all the testers that should be around us. And look at us, are we all spring chickens?
**Marcel**
Probably not. Really?
**Jason Arbon**
Look at the conferences, look at the talks. James, in those old videos, James looks like a...
a spring chicken then. So the key, my key point is that the test is that is interesting. I used to go on walks with James as a, as a, as a, as a shallow copy of the talks that Zuckerberg would give people around the Facebook campus to try to think that we're cool.
But we talk about this thing, these meta kind of problems all time. And a lot of that stuff came from the core notion that talk, it's like a whole thing, but like improvements in engineering process and CI/CD and observability and all these types of things.
..were going to take up a lot of good improved software, by the way, without testing help, it's just going to make software better. But it's also going to like relieve a lot of the burden on testing because these things are coming into existence and they were, you know, they were newer back then.
So the idea is when I propose on this, I haven't really said it much, but I see it a lot of time. People think this is that that oh, it's, you know, it's not dead. If you look at how much. more software there is just the growth in market cap of Microsoft and Google alone
**Marcel**
Mm-hmm.
**Jason Arbon**
If they kept the same percentage of testers. There should be hundreds or millions more testers like. We should do the math There should be millions more testers than there are instead a lot of us are aging in place And and it's funny because the teams that don't need those testers where test is dead See, this is why I can't get on the internet guys you Ah, Thanks. But here's the thing: the thing is that all those teams should have testers and they don't anymore.
They're not there, but the teams that want humans, that need humans because they haven't adopted all that latest technology, still need human beings. So I think it's a huge survivor bias thing and Like literally just look at the math percentage of people that are doing testing It testing grows the CAGR the growth year-over-year as an entrepreneur, total money spent on software testers and software testing services, is like growing it like I think it's an 11% Rate this year.
It was like 20% maybe like six years ago. You know these numbers, guess what the overall growth in in tech is and just big Over 20% still
**Marcel**
I think the metric which says 20 to 40 percent of all the spending in IT igoes to quality assurance.
**Jason Arbon**
Okay, it's changing what and how we are doing this quality we're numbers based on which report they're doing so like Sometimes they account they count developer activities that are related to testing and most developers actually spend probably like 40% of time testing locally They could change
**Marcel**
My point is that this trend is not did not change for a couple of years when you ask or check the reports and analysis it used to be 20.
**Jason Arbon**
No, no used to be 20. This would be depending on the, depending on Gartner or whoever you're listening to, it was like between 20 and 23% about four or five years ago. I had to make a pitch deck and get through VC analysis and all that stuff.
And, and that number's been dropping about one and a half, 2% a year for the last like,
**Marcel**
Okay, this is interesting. If you have any source for that, I would like to see that because I felt like this is all for the longer time.
**Jason Arbon**
Yeah. So then there's, and if you look at the reports, they also anyway, they mess with the numbers because it's a product. Yeah. The data is a product. And so like, here's it. Here's the best one. I'll give it a question back to you.
Like these quick challenges. So we've seen all the garden reports, right? That like, guess what? 70 something percent of testers are using AI today. Yes. And this is about before ChatGPT, by the way, too.
**Marcel**
Yeah, that I that's I think it's bullshit and we could see such a bullshit on many places because if you look on the reports five Sometimes you could see the small letters they ask about are you using or planning to use these tools?
**Jason Arbon**
They're all scammed. Yeah Because then they want to because then all the companies have to pay them to get the data on how people are planning to use AI
**Marcel**
and the key point you set in that in that topic was that they data are they product If they want to improve their product, they need to be slightly influenced by the vendors who are trying to sell something to the market.
**Jason Arbon**
Everyone they're selling stuff to the vendors who are selling stuff to people and the people don't want to, people also self report strangely because they don't want to say that they're not up to the cutting edge. Like so they'll don't make stuff up to like they lie to themselves.
Those numbers are very cagey at best but the ones that have been consistent and persistent over time It's the percentage of spend on QA is actually been dropping and if I'm trying to raise money I shouldn't be telling the world that but it's just the reality.
**Marcel**
I'll ask you the last question in this section about the testing is dead and future and so on. So what do you think what what what is going on out there and where we will end up or you said in 10 years there will be humans in test or there won't be manual testing manual testing is not here anymore in 10 years you said that in that question.
