Discover flakes by running new tests many times

Tests that only sometimes succeed are a huge nuisance:
- They can lead to developers rerunning their build over and over trying to get the build green, costing your company real money.
- They can mask real issues - maybe one test is sporadically failing where it wasn't before because there's a new bug?
- Worst yet, as any bug they are really hard to patch after the fact.

The most best way to kill flakes is by not letting them happen. By running new tests in PRs over and over many times to see if they fail even once.

The code

Here's how you can do that:

  # flake_discoverer.rb
module FlakeDiscoverer
  # This file should be generated on CI, with a command such like
  # `bundle exec rspec --format json --dry-run --tag ~skip > spec/master_tests.json`
  # on master branch
  def self.known_test_descriptions
    @known_test_descriptions ||= begin
      data = JSON.parse(File.open('spec/master_tests.json').read)
      data.fetch('examples').map { |row| row['full_description'] }.to_set
    end
  end

  def self.run_new_test_multiple_times?(example)
    !known_test_descriptions.include?(example.full_description)
  end
end

# rails_helper.rb
config.around :each do |ex|
  if FlakeDiscoverer.run_new_test_multiple_times?(ex) && ENV['CI'].present?
    puts("New test, running 50 times: #{ex.full_description}")
    50.times { ex.run }
  else
    ex.run
  end
end

To make this work, spec/master_tests.json must be generated before tests run on CI.

Integrating into CircleCI

To integrate this into CircleCI, you can add this step before your tests in .circleci/config.yml

- run:
        name: Gather known tests
        command: |
            git checkout origin/master
            bundle exec rspec --format json --dry-run --tag ~skip > spec/master_tests.json
            git checkout "$CIRCLE_SHA1"