Testing against multiple browser versions in 2021
A great question came up today on Stack Overflow from user zanona. It is indeed a very elementary question and gets more relevant each year.
What's the benefit of E2E testing only against latest browser versions? [...]
Any insights about why considering only recent browser versions and forgetting about the rest is a good idea, by using Playwright instead of a tool which would allow targeting tests against a specific browser binary[?]
I was consistent about using the latest Chromium-based browsers in my end-to-end tests in the past years, still it is a very valid assumption to run our tests on as many browser versions as possible for wider support of our product.
Do you remember Browsershots.org with its long list of specific navigators?
Those days are gone as the way how browsers are updated changed over time.
Well of course, as always browser version usage really depends on our needs. Let's see.
Puppeteer, Playwright with specific executable
First, let's clarify that it is possible to launch Puppeteer and Playwright with other executables than the bundled one in the case of Chrome/Chromium.
Puppeteer: puppeteer.launch([options])
const browser = await puppeteer.launch({ executablePath: '/path/to/Chrome70' });Playwright: browserType.launch([options])
const browser = await chromium.launch({ executablePath: '/path/to/Chrome70' });Second, even if 'testing' is there in the list of features in the case of Puppeteer and Playwright, they are not testing frameworks like Cypress which is a testing tool primarily.
What's the benefit of E2E testing only against the latest browser versions?
Some years ago it was crucial to test across all browsers and as many versions as used by a mass of users.
Over time it has changed.
What's new in 2021?
- Only Google Chrome and Safari (Webkit) browsers have 10%+ market share. Statcounter.com, Wikpedia
- For the vast majority not even Firefox support is required (with its 3.68% market share worldwide)
- "Google Chrome on Windows and Mac auto-updates itself on a regular basis. The auto-updating procedure is performed by Google Update, which is based on the open-source Omaha project. Auto-updated provides fixes to sometimes critical issues, limiting exposure." chromium.org
- Due to the auto-updates the number of users with not the latest Chrome is very marginal. The previous versions (e.g. the last-but-one) go out of circulation in 1-2 months and remains under 1% share for a short time before vanishing. Desktop Browser Version Market Share Worldwide, Statcounter.com
Conclusion(s)
I.) It means for most products it is fairly enough and the best option to test against the latest browser versions.
For example Chrome 70 is a relatively old version, released in October 2018, for most products it is not required to support it, they are in the same bucket as Internet Explorer 11 or legacy Edge.
II.) For some products, you may have to support specific (older) browser versions, in that case, the usage of specific executables can help.
By the way, if you'd decide to run tests against multiple browser versions it will be still possible with Puppeteer or Playwright as well, you just need to provide the right executables while the test suites are iterated over (e.g. in case of Jest: describe.each() can run the same tests or test suites with different configs and test data).