Best practices for preventing flaky visual tests

In some cases, you might encounter instability in your visual tests due to inconsistent behavior of your application UI on certain environments. This is something you need to take into consideration when designing and maintaining your visual tests.
 
Flaky tests can happen due to a number of reasons; either your network is not stable enough, or maybe your application servers encounter loading delays. You might also depend on third-party components that have different response times. Last but not least, your client environment could be facing performance bottlenecks such as CPU or memory load.
 
As a result of these issues, your automation tests might run into synchronization challenges. If you are experiencing some of these difficulties related to UI stability, here are some best practices to help you deal with them better.
 
The first recommended practice you need to know is WebDriverWait. This mechanism enables you to set an explicit time (in seconds) during which your test will try to meet a certain condition (e.g. wait for an element to be visible). If the test does not meet your condition within the predefined time period, then your test will not continue and thus will fail.
 
Here’s an example of such a command:

WebDriverWait(webDriver,5).until(ExpectedConditions.visibilityOfElementLocated(By.xpath("xpath_to_search_textbox")));

If you wish to learn more about dynamic loading times, you can review this blog post from Dave Haeffner.
 
In addition to the first method, there are other alternatives. Applitools provides a visual synchronization mechanism controlled by the MatchTimeout setting, which allows you to define the max amount of time (in seconds) that Applitools Eyes will wait for your app to stabilize to a point that it is similar to the baseline image. When using this setting, Applitools Eyes will try to immediately compare the page against your baseline. In the case that there is no match, it will retry several times until there is a match or until the timeout will expire. (So if the page will stabilize after 1 second, the test will continue after 1 second even if you set the timeout as 10 seconds.)
The default waiting time is 2 seconds, but if your pages take longer to load/stabilize, you would need to change it accordingly and set a higher limit.
 
Since this method relies on your baseline, you need to ensure that your current baseline represents a stable state of your application in order to compare your future tests with it. If this is not the case, please be sure to accept a new baseline after another execution of your test.
 
You are welcome to read more about Applitools’ Match Timeout here.
 
The third option is to add a wait or sleep command to your code. For example, in Ruby, in order to add a 10 second wait previous to your test run, you simply would add sleep(10) at the relevant line in your test. Note that among all the recommended methods, this is the least recommended one due to its rigid nature. For example, if your page has finished loading after 5 seconds, you will still have to wait 10 seconds. On the other hand, if your page was unexpectedly loaded within 15 seconds, your test will still fail.
 
These methods focus mainly on issues that involve UI instability, but other reasons can cause flaky visual tests, such as working with dynamic content or using different GPUs that render your app differently. You can try dealing with these issues with Applitools’ built-in solutions, such as Floating regions or Ignore regions.
 
For additional advanced best practices, please contact Applitools support team.