Choosing an end-to-end testing framework starts with clearly defining your requirements before comparing tools. Those who prioritize Safari support, execution speed or debugging convenience will get different results. Among the frameworks compared, including Cypress, Playwright, Selenium, TestCafe and Nightwatch, Playwright stood out for its speed and low resource consumption.
Key Takeaways
- Anyone who introduces a testing framework without first defining concrete requirements risks blind spots: Mesut Durukal discovered too late that the tool he was already using did not support Safari browsers as a matter of principle.
- The choice between Cypress, Playwright, Selenium, TestCafe and Nightwatch depends on the project goal: Playwright scores with execution speed and small Docker images, Cypress with an integrated dashboard and its own debugger.
- Requirements for a framework cannot always be precisely quantified: speed, for example, has no fixed boundary value, but must be consciously named and prioritized as a criterion.
- If you set up automation code from the outset without duplicates and with reusable auxiliary classes, you can change the framework later without having to rewrite all test cases.
- Tool decisions should not be a one-off process: As frameworks are continuously evolving, it is worthwhile to regularly check existing decisions against new requirements and alternatives.
Why the choice of end-to-end framework determines test coverage
The test automation framework determines what you can test in the first place. If the decision is wrong, you won’t cover cases that are central to your product. This was Mesut Durukal’s experience when he joined an ongoing project and took over the chosen tool.
Automation is part of quality assurance because a large number of test cases cannot be handled manually. However, this does not give carte blanche to any tool. The selection and setup of the environment determine the limits of what automation can achieve.
In concrete terms, this was demonstrated by a missing browser. The existing tool did not support the execution of test cases in Safari. In the region where Mesut was testing, the majority of traffic comes from iOS and macOS devices, i.e. Safari. A browser that accounts for the majority of users was therefore completely excluded from the test coverage.
Determine your requirements before comparing tools
Start with a list of the features your test cases really need, not the tool. Many teams do the opposite: they buy something, try it out and later realize that it doesn’t fit.
Mesut fed his list of requirements from two sources. The first was his own experience with the specific application: What attributes occur on the web pages, are there iframes, do interactions open new tabs or pages? The second source was conversations with the product team and developers to close gaps and record known risks.
For a comparatively simple web application with few pages, text fields and buttons, the list remained short. Nevertheless, there were clear must-have criteria:
- Execution in the Safari browser, because that was where the main traffic lay
- Support for device emulators to simulate mobile views directly in the browser instead of on real devices
- Ability to customize requests to target different use cases
Not every requirement can be hard-coded. There was no fixed specification for the execution speed, such as “less than ten seconds”. The framework should execute test cases as quickly as possible, but the threshold remained a gray area. Such soft criteria also belong on the list, but with the awareness that they have no hard limit.
What a structured tool comparison looks like
Compile the frameworks widely used in the community and evaluate their strengths and weaknesses against your list of requirements. Mesut chose four to five of the most frequently downloaded frameworks from NPM: Cypress, Playwright, Selenium as one of the oldest, TestCafe and Nightwatch.
There is no single best tool. Each has different strengths and weaknesses. The decisive factor is which function has the highest priority for your project. In Mesut’s case, Safari support was at the top of the list, and it was precisely this criterion that shifted the result.
The following overview summarizes the features that stood out in the comparison:
| Framework | Strength | Weakness |
|---|---|---|
| Cypress | Built-in dashboard and own runner, results incl. duration and flakiness without additional effort, debugging in own window | At test time no execution in Safari (WebKit only as beta) |
| Playwright | Three to four times faster than the alternatives, small Docker image, resource-saving in the pipeline | No noteworthy in the project |
| Nightwatch | High readability and simplicity of code | Weaker community support, open tickets, spotty documentation |
| Selenium | Very extensive adaptability, own solution can be built on the basis of the framework | (not evaluated in detail in comparison) |
At Cypress, Mesut emphasized the reporting function. After execution, all results automatically end up in the dashboard, now called Cypress Cloud, including execution duration per test case and indications of unstable cases. With other tools, you have to compile such evaluations yourself.
Playwright scored with speed. The small Docker image loads quickly in the pipeline, and execution was significantly faster than with the alternatives.
Weaknesses need to be put on the table openly
Every framework has its weaknesses, and these need to be included in the evaluation. At the time of testing, Cypress did not support execution in Safari. Later, beta versions for WebKit, the driver behind Safari, were released, but at the time this was the decisive exclusion for Mesut’s requirement.
With Nightwatch, the problem was less in the code than in the environment. As an open source project without a large company or community behind it, open tickets remained unresolved and there was no clear solution for some problems. Cypress is older than Playwright and therefore has more material in the community, which makes troubleshooting easier.
I can’t say that this tool is the best. Each has different strengths and weaknesses. The most important thing is to find out which function is most important to you. Mesut Durukal
Architecture decides how expensive the tool change will be
A reusable architecture makes switching from one framework to another inexpensive. If common operations are cleanly encapsulated, a tool change does not affect them.
Mesut did not delete the test cases when switching, but converted them. Deleting would have meant that the cases were redundant, and they were not. The conversion effort was minimized by a decision he had made from the start: avoid duplicates.
The login example shows the principle. The login process is used in almost every test case. If the login logic is located directly in the spec files, you have to change it in every single file when switching. If, on the other hand, it is in a pure JavaScript file in an auxiliary class, it remains unchanged when switching from Cypress to Playwright. The functions are then normal code in a programming language, reusable regardless of the framework.
Machine learning platforms can also be used for part of the conversion. You specify the spec files and have them converted. The accuracy is not sufficient for complete automation, but it provides a basis and reduces the manual effort.
The tool decision is never final
Regularly check whether your end-to-end framework still fits your requirements. A decision made is not a final decision. Requirements change, and the frameworks themselves continue to develop, with new versions sometimes on a daily basis.
This results in two tasks that stand side by side. Before making a selection, you clarify which function has priority for you and whether a framework is suitable for it. Then keep an eye on whether the chosen tool will still be viable in the future. There are always areas where improvement is possible, so it’s worth keeping this question open at all times.
