ixnzcw
Last Updated: July 16, 2019
·
602
· macob0

Discover flakes by running new tests many times

Tests that only sometimes succeed are a huge nuisance:
- They can lead to developers rerunning their build over and over trying to get the build green, costing your company real money.
- They can mask real issues - maybe one test is sporadically failing where it wasn't before because there's a new bug?
- Worst yet, as any bug they are really hard to patch after the fact.

The most best way to kill flakes is by not letting them happen. By running new tests in PRs over and over many times to see if they fail even once.

The code

Here's how you can do that:

  # flake_discoverer.rb
module FlakeDiscoverer
  # This file should be generated on CI, with a command such like
  # `bundle exec rspec --format json --dry-run --tag ~skip > spec/master_tests.json`
  # on master branch
  def self.known_test_descriptions
    @known_test_descriptions ||= begin
      data = JSON.parse(File.open('spec/master_tests.json').read)
      data.fetch('examples').map { |row| row['full_description'] }.to_set
    end
  end

  def self.run_new_test_multiple_times?(example)
    !known_test_descriptions.include?(example.full_description)
  end
end

# rails_helper.rb
config.around :each do |ex|
  if FlakeDiscoverer.run_new_test_multiple_times?(ex) && ENV['CI'].present?
    puts("New test, running 50 times: #{ex.full_description}")
    50.times { ex.run }
  else
    ex.run
  end
end

To make this work, spec/master_tests.json must be generated before tests run on CI.

Integrating into CircleCI

To integrate this into CircleCI, you can add this step before your tests in .circleci/config.yml

- run:
        name: Gather known tests
        command: |
            git checkout origin/master
            bundle exec rspec --format json --dry-run --tag ~skip > spec/master_tests.json
            git checkout "$CIRCLE_SHA1"

10 Responses
Add your response

32188
32288
32956

Everybody' isn't right about that. Overlooking testament issues makes SSL fundamentally shaky, particularly defenseless against man-in-the-center assaults [url] http://eventeca.com/ [/url]

2 months ago ·
33014

Thanks for sharing such great tip. I'll definitely try this see what happen.

about 2 months ago ·
33078

I really appreciate the kind of topics you post here.
http://fnafworld.online/

about 2 months ago ·
33166

nice great idea https://mywifiextnets.net/ best part of the article that it gives a good effort

about 2 months ago ·
33275

tks you

about 1 month ago ·
34365

I have read through some similar topics! However, your article gave me a very special impression, unlike other articles. I hope you continue to have valuable articles like this or more to share with everyone!
https://vex-3.com

13 days ago ·
34489

This blog is definitely entertaining additionally factual. I have picked up helluva helpful tips out of this amazing blog. I ad love to visit it again and again. Thanks!
https://hotmailhelper.com/hotmail/hotmail-login

3 days ago ·