Google: AI-Checked
Google builds amazing software that makes access to the world’s information super simple, right? Google’s home page is the fastest, right? The Google home page is used by many to test their internet connection, and I’ve been using it as a fast, simple page to test the Checkie.AI testing agents naively assuming the page would be the highest quality and there wouldn’t be many issues to distract. It seems some cruft has crept in at Google lately, and the competition hasn’t stopped improving.
Let’s walk through the AI’s software quality analysis of the Google home page, and in the process see how human engineers can interact with AI bot data to perform better quality analysis than I know I’ve ever done in the past — and better than Google too :)
Design Checked
The Google home page is so familiar, and the result of design genius that only billions of dollars can buy, right? I thought so too. Now I have a new perspective thanks to the AI-checking bots that found the stark design to be less than ideal for all users — maybe too stark and minimalistic.
The AI found 3 issues users might notice and Google might want to fix.
- There is no link to help or support. I know Google doesn’t want to deal with pesky humans at scale, but there are feedback mechanisms on other Google apps. Maybe at least let users talk to those fancy new AI chatbots?
- In the interest of minimalism, some text on the page is just text, sometimes the text is a single word and used like a link, but doesn’t look like a link. Sometimes it is a few words or a sentence — with no visual separation of the words or indication of whether the text is a link or just text. Not the end of the world, but with billions of users, I’m sure this confuses at least a few million.
- The page mentions something about a decade of climate action. Congrats, but it’s kind of odd to be placed on the home page which is generally so sparse, and the user has to click it to understand what it is about. A 1%’er problem, but also probably not the most understandable or useful design. Looks like someone just said ‘Put a link to my project on the home page’ :)
Qualitative Feedback
As expected, Google’s home page gets high marks, but no one is perfect. Checkie.AI generates relevant synthetic users with a name, profile, and things they’d like to accomplish with the web page they are checking. The design and usability of the page are a fantastic 9/10, as we’d expect.
Many of the synthetic users said they wanted more quick and direct links to other Google services. Some of those links are hidden in the ‘menu’ icon — minimalism gone too far? I remember a design ethos from when I worked there to avoid submenus and have to drill into things — maybe that has changed? Some synthetic users said the page is too ‘sterile’ and lacks emotional connection. True.
Functionality Checked
The AI Checking agents automatically analyze the page and create functional test cases. As expected, most functional tests worked well.
The AI performed some basic interaction tests, similar to what a human tester would do. Interestingly the AI not only looks at the page, and determines what can be done on the page, but it can also check if the resulting behavior seemed correct. All on its own, without human input.
One of the interaction tests showed a warning. When clicking on the GMail link, the resulting page didn’t look like the GMail page it was expecting. Interesting. Maybe worth Google considering. The actual gmail.com page when you aren’t logged in just presents you with a simple login page, not a marketing page.
The AI also tried some goal-driven, multi-step tests. As an example, the AI wanted to check for the ability to search for some ‘technical documentation’. The AI was able to define a test with the query “python for comprehension”. The AI automatically loaded the Google home page, entered the query, and importantly visually checked the results page to make sure the results were “relevant”.
The AI created these functional tests, executed, and verified them, all without humans in the loop.
No Selenium or JAVA was used, or harmed in the process. No humans had to think up clever test cases over a cup of coffee, manually execute the tests over the weekend, craft a test script over a day or two, or get the budget to set up infrastructure to execute the test. This was all done, end-to-end, by AI.
Of course, the human, me in this case, can check that the AI did the right thing based on the test definition and screenshots, and indicate that with a thumbs up or down. Those humans can now do more complex testing — or convince the PM and developers to fix the other issues found by the AI.
Performance Checked
We think the Google home page is the fastest on the web. Most test teams can barely test their site, let alone the competition’s web pages. They set their own performance goals and try to meet them. With Checkie.AI, all the competitors can also be checked for performance issues. The AI found that Bing (0.3s) is consistently just as fast to load vs Google (0.3s) these days. Google is still fast, but not ‘the fastest’ anymore.
Digging into the network and API calls under the hood, we can look for what might be slowing the page load time down to get even faster than Bing. Firstly, there are a lot more API calls (53!) than I would have guessed for such a seemingly simple web page. And more data than I’d have expected.
We can see that the slowest of API calls aren’t all that slow — they are super fast at 13–22ms, they probably can’t get much faster and wouldn’t be noticed by mortals anyway. That said, one of the payloads is surprisingly large at 877kb, and when you click on it, it looks like a bunch of ‘Gunk’ that probably doesn’t need to be in there, or can be trimmed down.
I asked Claude.AI what all that javascript does, and yup, looks like a lot of that code might not need to be in there. For example, maybe only download the browser-specific shims based on the user agent string? Maybe only include libraries needed for the home page instead of for all other Google apps? And that’s just a few seconds of human analysis with a ChatBot.
Coverage
Google’s home page is a high-quality site and they have lots of testing resources, but even Google can’t test for everything. The AI-checking agents checked thousands of things on the Google home page for issues:
- User Interface / Visual
- Console Logs
- Networking / API Calls
- Document Object Model and Javascript
The AI checked all these aspects of the application for issues such as accessibility, usability, privacy, performance, content, API, console logs, UI, UX, security, GDPR, OWASP, exploratory testing, visual differences, qualitative feedback, feature sets, competitive considerations, and general issues. Most test teams don’t even know how to test all these things.
Each aspect of the Google home page was checked for a broad array of possible quality issues — not even the biggest and best teams check for all of these types of issues today, and if they do, it can take days to weeks, have to communicate with different vendors, and is expensive.
Analyzing only 10 Google pages in this quick run, the Checkie.AI bots checked over 1600 network calls, noticed some errors in the console logs, provided 23 suggestions for improvement, and found 19 issues/bugs.
AI Summary
The AI also generates a high-level summary of the software quality of the Google home page, including a Grade (B), quality relative to competitors, and a test-manager-like summary in text.
Note that the grade isn’t a simple % calculation, it is based on an analysis of all the quality metrics above and does some sophisticated consideration of the types of issues that are most and least important for that particular type of app based on analysis of millions of app store reviews. Most humans haven’t read 1000 reviews.
It should make you feel better knowing no one is perfect — not even Google. But, with the progress of AI, it is now embarrassingly easy to discover similar issues with your website. Checkie.AI is working to democratize software testing and quality so that every web page can have the same level of testing as even the largest companies — even teams without testers. Testers should focus on the more difficult and complex work and analysis of the AI bot results.
AI is already delivering more coverage and in more areas than even the best testing teams.
If you’d like to dig around in the results yourself, you can — just click HERE.
If you care about software quality, but wish it was automated and delivered far more coverage just sign up HERE to get AI Checking for your website — the bots only need to know your URL.
— Jason Arbon
