Speed up Travis-CI build preparation time by 800%

///===

update 2013-10-22

Travis-Ci introduce caching - http://about.travis-ci.org/docs/user/caching/
So far it is only available for private repositories, still it is worth to look at this instead of custom solution that I've drafted in that document. Thanks for Travis-CI guys for finally implementing it :-)

///===

We want to start using travis-ci our customers projects, unfortunately preparing build was taking 8 minutes, and it is the time before test are starting. Another drawback from this, was fact that parallelism builds have no value (each instance will take their 8 minutes before running tests), and we really want to use this Travis-Ci feature ( http://about.travis-ci.org/blog/2012-11-28-speeding-up-your-tests-by-parallelizing-them/ )

So I spend last day's in order to improve this - result: preparation is taking less then a one minute :-)

here is our final travis.yml file

language: ruby
rvm:
  - 1.9.3
env:
  - TAG_FILTER=~js DB=postgres RAILS_ENV=test
  - TAG_FILTER=js DB=postgres RAILS_ENV=test
install: touch ~/do_not_run_bundle
before_script:
  - "./script/travis/setup_database.sh &"
  - "./script/travis/configure_rspec.sh &"
  - "./script/travis/bundle_install.sh"
  - "bundle exec ruby script/travis/bundle_cache.rb &"
script: "bundle exec rspec --tag $TAG_FILTER && sleep 1"
notifications:
  email: false

1) Caching .bundle directory

the most time consuming tasks was installing gems (6 minutes), so we thought what it would be if we will cache .bundle directory :-) Compressed directory is about 110MB - downloading it from us-west-1 takes 30 seconds in average, from eu-west-1 50 seconds. The most fastest is keeping your file in hetzner server if you have any ( 5 seconds in average).
In case you wish to encrypt your credentials, please read - http://about.travis-ci.org/docs/user/encryption-keys/

script/travis/bundle_cache.sh

require 'fog/storage'
file_name = 'bundle.tgz'
s3_file_path = "travis/#{file_name}"
local_file_path = file_name
storage = Fog::Storage.new({
  :provider => 'AWS',
  :aws_access_key_id => '<KEY>',
  :aws_secret_access_key => '<ACCESS_KEY>',
  :region => 'us-west-1'
  })

# prepare archive

`rm -rf #{local_file_path}`
`tar -cjf bundle.tgz .bundle`

# put on s3

bucket = storage.directories.new(key: '<YOUR_BUCKET>')
file = File.open(local_file_path)
bucket.files.create(body: file.read, key: s3_file_path, public: true)

3 things worth noted are
* we upload file with public permission, thanks to that we are able to download file with authorization (.bundle directory is usually nothing secret)
* we prepare cache in the background - so it's not delay starting tests
* we skip installing gems that are are used only in development

script/travis/bundle_install.sh

#!/bin/sh

curl -o bundle.tgz https://<HOST_NAME>/travis/bundle.tgz
tar -xf bundle.tgz

bundle install --path .bundle --quiet --without=development

exit 0

2) loading database without rake

just set to dump you schema in sql format and you will save another 60 seconds :-)

config/application.rb

config.active_record.schema_format = :sql

script/travis/setup_database.sh

#!/bin/sh

psql -c 'create database <DB_NAME>;' -U postgres
psql -U postgres -q -d <DB_NAME> -f db/structure.sql
cp -f config/travis/database.yml config/database.yml

3) separate configuration for rspec ( spec_helper.rb .rspec)

in development we use spork - for ci we don't, so we remove few unnecessary lines from spec_helper and prepare separate file (more readable then many if's) - it will give you a few seconds more of speed up

It's is definitely possible to write it in better - still I see that idea is worth of sharing. Hope that in some day travis-ci will give possibility to use somekind cache directory between builds out of the box :-) (for Pro Users - ask support if you don't have yet: support@travis-ci.com)

There is one more challenge that I have with Travis-CI, but this is something for the next holiday time :-)