Mabl is a scriptless, cloud-based test automation tool that has machine learning capabilities. Through the use of “Journeys” (that can be recorded via a chrome extension and edited manually if needed) it builds a visual model of pages and then compares these over time to subsequent runs, identifying any changes. It also captures broken links and JavaScript errors and can be integrated with CI/CD tools such as Jenkins to run post-commit along with other common tools such as Slack and Jira.
They offer a free 14 day trial so I figured I’d give it a try on the new agl.com.au release.
Setting it up
Once you’ve registered an account and logged in, you’ll be presented with the Mabl app dashboard, see below.
The first step is to create a new Journey. You’ll be required to enter an application URL and optionally some credentials if your website requires authentication. Unfortunately due to our IAM reCaptcha this posed a challenge, one that I’ll follow up with Mabl to see how it can be bypassed. For now, I excluded MyAccount and related SPAs from the scope of Mabl’s testing.
Once you’ve created your Journey, by default it will have 2 Plans associated with it. The first plan essentially crawls every link, taking screenshots of each page, logging javascript errors and finding any broken links. The second simply visits the homepage as a health check. The data from both plans (e.g. screenshots, errors etc.) are presumably harvested by the machine learning algorithms and used to determine potential regressions during the next run. Plans are scheduled to 4 hours by default, but can be set up to initiate after a commit or from CI (there’s a few plugins already available such as Jenkins).
Results
After the plans have completed, you’re provided with Insights detailing whether any issues were detected during execution. You can also click into the results for a detailed breakdown of the logs, screenshots etc. See below.
Broken links are provided as a list stating the page where the links was found and the href that returned an HTTP error code e.g. 404.
Below are the metrics captured from run results pre/post release. For context, this was a substantial release involving both infrastructural changes and a complete information architecture rework (apparent from the reduction in links/pages in the results). To some extent this defeats the purpose of Mabl, in that it’s intended to detect and raise small changes between releases over time, but I figured the other features were well worth continuing the trial and it was a good test of Mabl.
Pre-release
Post-release
We can see from the results that the number of pages was significantly reduced (by over 50%) as well as the crawled links (~ten fold reduction). This led to a reduction in broken links (the majority of which from the first run were pages linking to: http://www.microsoft.com/internetexplorer/ which now 404s) as well as in test execution completion time (also around the 50% mark/2 hours). I checked the broken links and forwarded them to the team for investigation.
Verdict
Given the ease of use of Mabl it’s well worth trialing it for free at your organisation at the very least. It literally took 10-15 minutes to set up and whilst the dashboard interface could do with some CX/UX improvements, overall it’s fairly intuitive. I can’t comment on the pricing as I was using the free plan and the ROI is very context specific (e.g. potential to replace other tools, cost per defect, hours saved on automation test script setup, maintenance and infrastructure). I also can’t talk much to the ML-driven aspect as I didn’t intentionally introduce any defects to see how they were handled. Documentation on the ML-driven feature is also pretty thin (perhaps intentionally to protect their IP), so it would be good to see some much deeper demonstrations to ensure it’s not just a marketing ploy.
A few issues were encountered while using it. Error messages/pages were quite common, particularly when accessing previous test runs. I was also unable to test our login and account management applications as these have reCaptcha enabled. This is normally bypassed by setting a cookie with a token returned from an HTTP request in our tests, however due to the limited “scriptless” nature I was unable to add this in. Perhaps it would be possible to engage the Mabl team directly to add in more sophisticated test scripts actions like this.
Overall, there is no doubt in my mind that tools such as Mabl (scriptless, cloud-based, heavily automated) will become increasingly popular as they reduce the barrier to entry and complexity of automation enormously.