Selenium Selectors that Suck Less
When I made the move from Microsoft to Google, the culture shock was immediate. When I asked for the goals of the next software release, the response wasn’t more features, it was ‘suck less’! I learned a lot about software at that moment.
Selectors are a key part of the magic of Selenium, and a key source of pain. Selenium selectors are the methods that let millions of test automation scripts find and verify elements in web pages. As we all know, when the page changes, even invisibly, those selectors often break. What if those selectors were a bit smarter? What if they had an automatic backup to try and fix things when they go wrong? Instead of adding more brittle features like ‘find the element to the left of this other element’ that might break even more often, what if selectors simply sucked less?
Selector’s Suck
Each automated test case has a half-life — it is just a matter of time before it fails. It's also just a matter of time before most test-development energy, time, and money are spent on maintenance fixing up all those broken selectors. Many software testing vendors' entire value proposition is having tons of hard-coded heuristics to try when selectors fail, claiming self-healing, or calling a bunch of IF-THEN’s “intelligent’ for some reason. Maybe try to ignore the ID and look for the class name when the ID is missing? Perhaps look for a unique combination of attributes that matched the element last time? Frankly, so much time and talent are spent fixing broken selectors that it is mind-numbing. Manually re-inspecting the element and building a selector or having a simple list of checks and variations doesn’t make our profession look all that sophisticated, and it's not all that effective.
That's a long preamble, but many in testing are right now too busy writing their first selectors with optimism, or too busy fixing old selectors to see if there is a better way.
Simple and Low Impact
Let's get to the point. Some test developers won't even care how or why this works, they will just make their selectors suck less in a few simple steps.
Just PIP-install a library, import it and add one line of code. That's three simple steps and can be done in 3 minutes.
python3 -m pip install test-ai-selenium
JAVA support is ready too, just click below for access. JavaScript/Node, and others coming in a few days. Appium, Playwright, XCUITest in a few weeks.
Now, just import the test_ai library
from test_ai import test_ai
That was easy. Now just add this line of code:
driver = test_ai.TestAIDriver(driver, “API_KEY”)
You can get an API_KEY for free at: https://sdk.test.ai . You can signup for targeted early access and ask questions below.
That’s it. All your selectors are now automatically ‘backed up’ with a smarter version.
How it works
Firstly, it works all automatically from here. Some automated ‘training’ happens in the background to create backup versions of all these selectors. Then, when the tests run, the backups are only called if the original selector breaks.
Let's talk about training. After adding those three lines of code, The next time the tests are executed, some magic happens. If this is the first time the test has run since the test.ai code was added, the library spends about 1.1 seconds for each selector. The test.ai driver:
- Looks at the original, old-school CSS/XPath selector you are using to find the element
- Then, lets the usual Selenium selector code find the element as normal,
- Then trains a little selector bot to find the element in the future. The image and other attributes of the element and screenshot are sent to a test.ai server to train a dedicated neural network (actual AI/Machine learning) to find the element just in case the first one breaks.
Every time the script is run thereafter, the old-school selectors you defined still run as normal and no new training is required. This means your code runs just like it did before, just as fast— the backup selectors aren’t called unless needed.
When, and it's only a matter of time, the old-school CSS/XPath selector breaks because the application changed, the new backup selector kicks in.
The test.ai driver, which wraps the standard Selenium driver detects the selector break and instead of throwing an exception and stopping your test execution, invokes the backup selector and tries to find the element using visual cues and ‘AI/Machine learning’ finding the element and letting the script continue as normal.
How does the backup selector try to find the element? Well, a lot happens behind the scenes, but only takes about a second, and only when the default selector breaks. The driver sends a screenshot and the original selector to a test.ai server. The server looks up the previously trained neural network for this selector, chops up the screenshot into elements, then runs the neural network against all elements. The element most like the original (color, text, #edges, position, size, etc.), with a threshold, is returned back through the wrapped selenium driver. The driver code writes out a warning that the backup had to kick in (so humans can tell that the original selector if they like), and returns an element object back to the test code so it can keep on testing.
Just like you don’t need to know how Selenium works internally, you don’t need to know AI or ML to know how the backup selectors work. These neural network classifiers work like your own brain does when an element selector breaks. You would look at the screen, find the updated and correct element based on what it used to look like and ‘do’, where it is, its color, its text, etc., and then update the selector and re-run your test. With the backup selectors, all this magic happens while you are sleeping.
Why We Built It
At test.ai, our long-term vision is to build bots that will ultimately automatically test 90% of the functionality in apps. We aren’t there yet, but we are making progress. These backup selectors for Selenium accelerate that vision by collecting more and more screenshots, elements, and which elements are interesting to click from a test creation perspective. We also just care about the world of testing and want to make it just a little bit better for every tester out there. We were tired of selectors breaking too!
By default, the test.ai servers collect all those screenshots and use them for training data to make the bots even smarter over time. We thought it seemed like a fair tradeoff to share some data, in exchange for free selector backups and a better lifestyle.
We also built this as a bridge between the two major types of testers, those who code, and those who don’t. Our other product is a low-code solution that enables non-coding testers to build awesomely reliable test automation with a drag-and-drop interface. Most interesting to us though is that the test.ai SDK for Selenium can communicate with the low-code tool, letting the two applications share this training data. Coding and Non-coding testers can finally collaborate on automation! More on that later…
For teams that want or need support, or need to run locally/privately, that is available too.
If you want to get started, it is super easy. Today it works for Python and JAVA with Selenium running on macOS. In the coming weeks, Javascript, Appium, and XCUI, and Windows support will be ready.
Click here to signup and get access so your selectors can suck less!
https://forms.gle/fqMWFYUgWhBMVpfp8
— Jason Arbon
CTO, test.ai
