NGINX / UNICORN / CENTOS 6.6 / RAILS TROUBLESHOOTING

Welcome to hell

As my first deploy on a VPS Centos 6.6 i was a little excited to test my sysadmin skills but oh boy I was so wrong.

The following guide uses most of a Digital Ocean guide:
https://www.digitalocean.com/community/tutorials/how-to-deploy-rails-apps-using-unicorn-and-nginx-on-centos-6-5

So I will jump some realy easy and solvable steps for common setup on any system/machine.

For the first steps we need:

• RVM
• RUBY
• RAILS
• UNICORN GEM
• POSTGRESQL
• NGINX

All this can be installed following the above guide.

When we have a basic Rails installation ready, our application will be deployed to the folder:

/var/www/appname
# change appname for your application folder

NGINX

The first setup is the Nginx configuration, we will use a single application server so we will use our default configuration file inside nginx installation and put inside it the following code:

# /etc/nginx/conf.d/default.conf
upstream app {
  server unix:/tmp/unicorn.myapp.sock fail_timeout=0;
}

server {
  listen 80;
  server_name localhost;

  root /root/my_app/public;

  try_files $uri/index.html $uri @app;

  location @app {
     proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
     proxy_set_header Host $http_host;
     proxy_redirect off;
     proxy_pass http://app;
  }

  error_page 500 502 503 504 /500.html;
  client_max_body_size 4G;
  keepalive_timeout 10;
} 

Here most of the code explains by itself but I will go a little deep in some of the parameters which are prone to miss understanding and wrong pathing for beginers like myself.

Now the explanation for this code:

# "myapp" will be any name, I encourage you to use it on lowercase just remember it because
# through this exact file NGINX and unicorn will comunicate

server unix:/tmp/unicorn.myapp.sock fail_timeout=0;

The root path:

# this path will be for the public assets of your application, as we know (or may not) when assets are precompiled they go straight into:
# "public/assets/"
# and NGINX will serve our assets so unicorn will take care of the internal processing
root /root/my_app/public;

The port:

# theres not much to explain here but if you have to change the port on your server remember to change it here too
listen 80;

Unicorn

Now inside our application we will create a unicorn configuration file:

# Yes this is a ruby file
/config/unicorn.rb

and put the following code inside it:

# Set the working application directory unicorn will load the application workers from, as copies of this folder (your application folder)
working_directory "/var/www/appname"

# Unicorn PID file location if you dont have the pids dir just mkdir it
pid "/var/www/appname/pids/unicorn.pid"

# Path to logs. If you don't have the /log dir just mkdir it but it is part of the Rails core.
stderr_path "/var/www/appname/log/unicorn.log"
stdout_path "/var/www/appname/log/unicorn.log"

# Unicorn socket (THIS WILL BE THE SAME SOCKET ON NGINX)

listen "/tmp/unicorn.myapp.sock"

# Number of processes or workers to spawn this will be processes or daemons on your server
worker_processes 2

# Time-out for server workers
timeout 30

Troubleshooting

Now we are going to run everything with:

# As unicorn_rails is a gem, remember to have your rvm environment loaded bundle install it, rvm gemset use it when running this command or inside unicorn logs it will display a missing gem error and it will timeout. 

# NOTE: -c is for directory, -D for Daemon running it, -E for environment running.
unicorn_rails -c /var/www/appname/config/unicorn.rb -D -E “production"

# some times nginx don't get installed as a service so we need to create it or use the /etc/init.d/ variant.
sudo service nginx start

/etc/init.d/nginx restart

As first time the service and everything will look like working but if you look at the logs on (if you cant access by normal user, log into "su -")

tail -n 200 /var/log/nginx/error.log

We will notice from timeouts to permision-denieds, to solve this there are several steps that I will explain (most of them problems):

1.- var/www permissions:

If we look at the folder's permissions with the following command (or any other)

sudo -u nginx stat /var/www

we will see that the folder is not owned or accessible by nginx, thats one of the main reasons why nginx can't access the public folder or gives us a Bad-Gateway response

to solve this we will add our current user and nginx user to the same group, remember it doesn't matter if we do chown or chmod to a folder NGINX AND UNICORN need to be able to use our application's folder and chown nor chmod will do the trick as they only change the owner and change the writing/reading permissions at shallow level:

# www-data is the name of a group we are creating a group so we can assign it to the folder and our users can access it
group add www-data

# add users to the group, my_user must be the user running unicorn's daemon.
usermod -a -G www-data nginx
usermod -a -G www-data my_user

# then we add access on the folder to the group. (-R will make all the folders inside www accessible to the group). If you want to make only your app accessible (multi app server) just add the group to your apps folder
chgrp -R www-data www/

Now when running unicorn we may get a can't write folder problem, this can be solved by a chmod but with the folder assignation should be sufficient (just remember to NEVER 777, only just 755 or 775 permission levels)

2.- Off to the socket issue

Some times you may get a (access denied) on your nginx's log pointing to the socket file, this can be solved in many ways, just remember your socket is in:

# Described in both configuration files for NGINX and Unicorn
/temp/unicorn.appname.sock

One way:

(one you may have found already) is to chmod or chown the sock file, I don't recomend it because the sock file is a temporary file generated every time your unicorn starts so just leave it under /temp/

Or another way:

# find the failing access or denied exceptions under logs
sudo cat /var/log/audit/audit.log | grep nginx | grep denied | audit2allow -M nginx

# (if you don't have semodule installed install it!!!!!)
# the above command will tell you to run a script when it finishes creating the policies, then run it
sudo semodule -i nginx.pp

After this just restart your nginx and kill/restart your unicorn processes,

NOTE: You may find other problems on access permissions but they are solvable by assigning the folder to the group above described on case 1.- .

Conclusion

As far as I have encountered, this are the major problems one may find which are not scripting errors like a wrong path or stuff like that, most of this are just permissions problems very common on CENTOS distributions mainly because of security reasons, just remember your server is meant to be secure and I encourage everyone to learn some security policies and directory information on your server as every each of them is meant for a purpose or job on different distributions like Ubuntu server or CENTOS.