Solving vagrant authentication problems

Vagrant.configure("2") do |config|
  config.ssh.private_key_path = "~/.ssh/id_rsa"
  config.ssh.forward_agent = true
end

Then add you public key to ~/.ssh/authorized_keys on the Vagrant VM