If you’re choosing testing tools for test automation, five criteria will shorten a long list quickly: you must be able to evaluate the tool yourself, it has to run reliably, it has to fit your process, the vendor has to be a fair partner and the tool has to be intuitive to use. In our selection, those five criteria took us from ten candidates down to two.
By the book, we had gathered our functional requirements for a test automation tool in countless workshops. Every requirement was neatly organized in Excel (because everybody can use Excel somehow, from the tester to the managing director). Thanks to tool databases, finding the candidate tools was easy. So what do we do with the ten tools that were left?
How about a counting rhyme?
Testing Tools You Can’t Evaluate Yourself
“Ten little testing tools, all ready to shine,
one we couldn’t try ourselves, and then there were nine.”
We work with standard technology, code in a standard programming language and use a standard GUI without a lot of bells and whistles. (You can still do exciting projects that way, honestly!) So there is no reason why we shouldn’t be able to evaluate a tool ourselves, instead of having the vendor “set it up” and “customize” it for us, possibly for a fee.
Stability: Crashes Overnight
“Nine little testing tools left running until late,
one crashed once too often, and then there were eight.”
This one nearly got knocked out before the final, because it simply refused to install (again on a boring, freshly set up standard PC). Well, the support team was friendly, and in the end it worked. But it never ran reliably, on any of the three workstations we used. Friendly support doesn’t make up for that.
The Tool Has to Fit Your Process
“Eight little testing tools, but bending’s not our style,
one didn’t fit our process, seven left in the pile.”
We’re happy to adapt our processes when there’s a good reason, and a tool can be a good reason. Every vendor has presumably thought hard about how the workflow and features are built, and you can usually count on a lot of experience behind those decisions. But the change has to be justifiable. If it isn’t, the tool doesn’t fit.
Vendor Support and Sales as a Partnership
“Seven little testing tools, what’s next in our tricks?
Support, sales and the offer, and then there were six.”
When you buy a tool, you’re also entering a partnership. It’s usually a long-term commitment, so the relationship has to work. A Christmas card from the sales rep (on paper, maybe even handwritten) is a nice gesture. But when sales calls every single day (!) after you’ve downloaded the trial version (“Any questions? Can we help?”) and sends emails on top, it gets pushy. Even if you do have a question, you no longer want to ask it. Not even the Christmas card can fix that. You want the same fair partnership when it comes to support, choosing training courses and so on. Sadly, not every vendor delivers.
Usability: Intuitive Testing Tools Instead of Training Marathons
“Six little testing tools, now let’s play them through,
four of them we just didn’t get, and then there were two.”
Test automation tools can be complex and take in-depth training to master. They don’t have to be, though. If seasoned software developers and testers can’t work a tool using the documentation and their own experience, it’s doubtful the tool will hold up in daily use. And there is another way, as the two remaining tools show. Both are so intuitive that working with them is a real pleasure. Word has it that team members actually enjoy writing tests with them.
Two Finalists
“Two little testing tools are left for us to choose,
one of them will win the race, the other one will lose.”
The two finalists were neck and neck, and both had to show what they could do. The final decision was due in early 2014. Until then, I enjoyed the Christmas cards.
