Tester Tales: Origin
Driving back through my college town, my mind drifted back to visions of tumbling Satellites lighting the sky on a spring evening. A submarine captain hits the periscope because his underwater message can’t get through. A missile lands hundreds of meters off target. In the digital age, a single one instead of a zero can lead to real-world consequences.
“Awesome work , the folks at Lockheed will really appreciate this.”
It was the end of the spring quarter. We had declared our project done, but all my mind was doing was thinking about what could go wrong as I walked across campus.
An incredibly patient manager from Lockheed Matin was supposed to infuse us with a bit of real-world engineering in our junior year of college. The project had us learning entirely new math, the kind of math that made sure messages sent by satellites couldn’t be scrambled by interstellar radiation or enemy electronic interference. If there were errors in transmission, Galois Field Arithmetic could detect the errors — and magically repair them. Pretty cool.
Most of Lockheed's projects were highly dependent on reliable communications and they needed a chip designed to implement it faster than their current software designs. Our student engineering team went to work on some VHDL software that could turn programmable chips into perfect, lossless, messaging devices. Like all college students, one member of our team was always too busy with other classes to help. We awkwardly caught him working on his Pie Pizzaria assignment — probably for the better. The other two were bespectacled and got right to work on the algorithm in the lab, not knowing how it worked, but working to get the project completed as quickly and efficiently as possible. Our hero spent several weeks trying to understand how the underlying math actually worked, alternately figuring it out and then getting confused again for weeks.
Magically, we submitted a final merged solution in the days before finals. It worked. We tried a few messages, ones with no errors, some with a small number of errors and it worked. We even tried inputs with more errors than the system should be able to fix, and verified it said ‘doh, resend’, as expected. We were excited it was working after only a few minutes of testing and looked to enjoy our summer vacation.
Walking out of the bleak NSA-looking engineering building, I started thinking of all the messages we hadn’t tested. All the messages that ‘could’ be sent through the system in the future — would they work too? Would some errors not be detected? Would some be fixed incorrectly? I started thinking of how many potential messages could be passed and quickly realized it was nearly infinite — more than the number of atoms in the universe. An interesting intellectual problem, I started having visions of Lockheed Martin confidently adding our chip designs to satellites, submarines, missiles, and jet fighters. My stress level increased.
Back to the lab. The college campus was quiet that first day of summer break. In the computer lab —it was just me, another goober who was always in the lab, and the aloof System Administrators behind the glass wall. All the cool kids were there. Friends were off hiking, some just napping, and let’s just say my girlfriend wasn’t too thrilled about the idea either.
I spend the next few days writing programs to generate all sorts of input test messages with different lengths, different content, and different numbers of errors, different patterns of errors — verifying the chip design correctly processed and identified all of them correctly. I looked into algorithms to generate statistically interesting permutations of inputs, leveraged my understanding of how the algorithm was supposed to work to target possible weaknesses, and how to build a test framework automatically execute it all. My sun workstation and a couple of others I ‘borrowed’ were soon heating up the lab. I also stressed about how I might have to eat crow and contact Lockheed if I did find a bug. After just short of a week of nerddom, I hadn’t found any bugs but had convinced myself with a lot of computation that the chip did actually work as designed.
I learned a few things about the world, and myself in those few days:
- Most real-world software isn’t tested all that well
- Learning to test the algorithm was more difficult and interesting than the algorithm itself.
- There was very little existing research on software testing methods — almost none.
Those few summer days turned me into a Software Tester. Writing code is relatively ‘easy’, Testing that code is the hardest and most important problem in Computer Science. For all I know, the code we delivered was never used. At worst it was probably tested by Lockheed — but I knew it worked with 99.9% confidence, and I could sleep at night. And, if it was used, it probably worked. When a Microsoft Test Manager landed on campus the next year, teasing an intern opportunity to work on testing embedded devices running Windows CE, I was more than ready to dive in.
Writing code is relatively ‘easy’, Testing that code is the hardest, and most important problem in Computer Science.
Note: These algorithms used to only be applied to space and military technology, but are now found in all computer devices/storage. Errors weren’t always caught, and when they were, the data was simply corrupted and lost. Today, our devices detect the error, fix it, and humans never notice.
— Jason Arbon, a Tester
