When automated tests seem to pass or fail randomly, we say that they are unstable, non-deterministic, or flaky. Beyond the annoyance this can be, fixing them is in fact critical to the success of any software product, as I will explain later.<\/p>\n\n\n\n
Flaky tests present themselves in different ways. For example, a couple of weeks ago, I encountered an automated test that would randomly fail in my team’s repository. The test was supposed to check that we could filter a list of users by their name. It was something like this:<\/p>\n\n\n
\nlet!(u1) { FactoryBot.create(:user, name: "Bill") }\nlet!(u2) { FactoryBot.create(:user) }\nlet!(u3) { FactoryBot.create(:user) }\n\nit "filters users by name" do\n all_users = users.get_all_from_db\n filtered_users = all_users.filter_by_name("Bill")\n\n expect(filtered_users.length).to eq(1)\nend\n<\/pre><\/div>\n\n\nThis test uses FactoryBot to create database records for our test. We pass arguments when needed, or use defaults otherwise. In our case, we just needed to specify the name of the first user to test the filter functionality.<\/p>\n\n\n\n
Simple as it is, this test sometimes failed because the filter returned two users instead of one.<\/p>\n\n\n\n
In my experience, automated tests are usually unstable due to tests not properly isolated (dependencies on other tests, components, or the environment), or the execution of non-deterministic operations in the code being tested, or in the test.<\/p>\n\n\n\n
Given how the test was written, I was confident the issue was not about dependencies, so I started looking for any non-deterministic operations, but the code was immaculate. However, there was an issue with the test itself.<\/p>\n\n\n\n
It’d be very boring (and sometimes inconvenient) if all test users had the same name. For this reason, our user factory sets the default name using Faker, a convenient utility that can generate names, emails, and other useful strings at random<\/strong>. <\/p>\n\n\n\n