Deploying and Scaling Zero Downtime NodeJS application

<a href="http://nodejs.org">NodeJS</a> is an event-driven and non-blocking I/O javascript platform. Now, nodejs have <a href="http://nodeframework.com/">30+ frameworks or libraries</a> that you can use to build a lightweight web application. The question is, how to deploy our nodeJS app to staging or production server?

---

Make your app scaleable

I was thinking about how to make scaleable application based on total cpu in my server. I need cluster module to bootstrap my application. For example case, by default I will have 1 master run to process request and then N workers fork the master to deal request with forked processes. Thansk to Johnny Domino who built node-clusterap, but i need to revise small thing due to issue of umask and sock file permission.

example clustering in express.io

var express = require('express.io'),
    Clustrap = require('clustrap'),
    app = express(),
    httpOptions = { 
      workers: 3, 
      port: 3000
    },
    environment = process.env.NODE_ENV || 'development';

if(environment !== 'development'){
  httpOptions.sock = './tmp/sockets/app.sock';
}

app.http().io();

app.get('/', function(req, res, next){  
  res.send('I am in '+environment);
})

var service = new Clustrap(app, httpOptions);
service.listen();

The snipped code above tells the application to fork 3 workers and listen on port 3000 for http server and socket io if environment is development, but listen to socket file and skip port listening if the environment is not development. If you do not provide number worker, it will fork worker as much as total cpu in the server. I prefer to use sock's file because I dont need to think about ports firewall to run my application.

---

Setup deployment script

They are some alternatives to deploy nodejs application:

<ul>
<li>
<strong>Capistrano</strong>
<ul>
<li>speak: ruby</li>
<li>source: http://github.com/capistrano/capistrano</li>
</ul>
</li>
<li>
<strong>Mina</strong>
<ul>
<li>speak: ruby</li>
<li>source: http://github.com/mina-deploy/mina</li>
</ul>
</li>
<li>
<strong>Minco</strong>
<ul>
<li>speak: nodejs</li>
<li>source: http://github.com/dsmatter/minco</li>
</ul>
</li>
<li>
<strong>Mina JS</strong>
<ul>
<li>speak: nodejs</li>
<li>source: http://github.com/CenturyUna/mina</li>
</ul>
</li>
</ul>

Because I speak Javascript, I don't want install ruby or any engine in my local and server machines only for a deployment. Then my decision goes to <b>Mina JS</b>, it's built on Minco and inspired by Mina Ruby. The Capistrano is awesome and full-stack deployment, too much complicated for newbie like me and for me ruby is not JS.

step 1: install mina in deployment machine

$ npm install -g mina

step 2: initialize deployment package

$ cd /path/to/dev_app
$ mina init

step 3: setup deployment package with zero downtime when restarting application

{

  "server": "user@127.0.0.1",

  "server_dir": "/path/to/production_dir",

  "repo": "git@github.com:youraccount/project.git",

  "prj_git_relative_dir": "",

  "branch": "production",

  "force_regenerate_git_dir": false,

  "shared_dirs": [
    "node_modules",
    "logs",
    "tmp",
    "public/uploads"
  ],

  "history_releases_count": 5,

  "prerun": [
    "npm install",
    "npm test",
    "ln -s config/database.js /path/to/production_dir/shared/config/database.js",
    "cp tmp/pids/app.pid tmp/pids/app.pid.old || true"
  ],

  "run_cmd": "NODE_ENV=production node server > /dev/null 2>&1 & echo $! > tmp/pids/app.pid; kill -15 `cat tmp/pids/app.pid.old` || true"
}

The code above is written in deploy.json file after you run mina init, the format is json. Let me explain little bit about the keys of json above:

• server is an address to deploy your application. It will use ssh login, you can add your ssh key to .ssh/authorized_keys in your server, so when deployment you will not be asked to enter a password.

• server_dir is a path address to deploy your app to this dir on server.

• repo is a git repository address, only support git right now.

• prj git relative dir is project location in your git repository if you have more than one project in the same git address.

• branch is a branch to be checkout and deploy from your github address.

• force regenerate git dir is forcing to reclone repository each time development, by default is false.

• shared_dirs is directories of your project in this array will use a symbolic instead create every time when run deploy. The shared directories are usually used by every release version, so that you will not every thing in this folder, for example log files in logs folder, pid file, sock file, cache files in tmp folder, or node_modules.

• history releases count is about how many release snapshots keep away from auto cleanup, by default is 10 releases if not provided.

• prerun is list of command lines to run customize scripts before run application. In prerun, I wrote cp tmp/pids/app.pid tmp/pids/app.pid.old || true, that means I copied the existing process id file to another file, so that I can used it to terminate the active server after the new version is released.

• run_cmd is a command to start your project. You can use Forever to ensure your node server runs continuously, instead of nohub or bash script with &.

---

setup deployment directories

Let's leave development or local machine at the moment and login into your deployment server. I assume you have user and directory for deployment, please do not use root for security reason. Go to your user's home directory and create production or deployment directory, I usually build some sub-directories in my deployment directory.

- production_dir
   - releases
   - shared
      - config
      - tmp
         - sockets
         - pids
         - caches
      - public
        - uploads 

Let me explain what those folders for:

• shared/config is directory to store configuration of your application, for example database.js, twitter_info.js, redis.js, and etc.

• shared/public/uploads is directory to store uploaded files if your application has process business to upload file and store it in server

• tmp/sockets is directory to store sock file that will be created after http server run.

• tmp/pids is directory to store pid file

• tmp/caches is directory to store cookies or cache files if you use file storage rather than memory storage.

• releases is directory to store history application by release version.

---

prepare assets precompile

I suggest you to use nginx as your main http server, because it supports event-driven, asynchronous, and 10K friendly. I am also interested to implement assets precompile like Ruby did. You can use asset-smasher to precompile your static assets files in local machine. But it can not handle run time precompile dynamic assets.

To handle dynamic assets precompile, I found ngx_pagespeed to automatic optimise assets at the first time page opened. Compiling assets is important to cache asset files in your browser by its version.

---

setup nginx's load balancing

To install ngx_pagespeed module, you can follow instruction on its github page. Once your nginx is installed, you need to create config file for your website, here is my configuration in /usr/locals/nginx/sites-available or /etc/nginx/sites-available directory after that restart to your nginx:

upstream nodejs_stream {
  # comment this line if you use port
  server  unix:/path/to/production_dir/current/tmp/sockets/app.sock fail_timeout=0;
  # uncomment this line if you use port
  #server 127.0.0.1:3000 fail_timeout=0;
}

server {
  listen              80;
  server_name  your-website.com;
  root                /path/to/production_dir/current/public;
  try_files           $uri /system/maintenance.html @nodejs_stream;

  location @nodejs_stream {
    proxy_set_header  X-Real-IP $remote_addr;
    proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header  X-Forwarded-Proto $scheme;
    proxy_set_header  Host $http_host;
    proxy_set_header  X-NginX-Proxy true;

    proxy_redirect    off;
    proxy_pass        http://nodejs_stream;

    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";


    access_log      /path/to/production_dir/current/logs/nginx.access.log;
    error_log         /path/to/production_dir/current/logs/nginx.error.log;

    # remove or comment app pagespeed configuration if you not use it
    pagespeed on;

    # it will precompile new assets each time application released, for best result please use tmpfs directory
    pagespeed FileCachePath /path/to/production_dir/current/public/;  

    #  Ensure requests for pagespeed optimized resources go to the pagespeed
    #  handler and no extraneous headers get set.
    location ~ "\.pagespeed\.([a-z]\.)?[a-z]{2}\.[^.]{10}\.[^.]+" { add_header "" ""; }
    location ~ "^/ngx_pagespeed_static/" { }
    location ~ "^/ngx_pagespeed_beacon$" { }
    location /ngx_pagespeed_statistics { allow 127.0.0.1; deny all; }
    location /ngx_pagespeed_global_statistics { allow 127.0.0.1; deny all; }
    location /ngx_pagespeed_message { allow 127.0.0.1; deny all; }
    location /pagespeed_console { allow 127.0.0.1; deny all; }
  }
}

---

Setup Zero Downtime Server with Monit

I use Monit to monitor system process and error recovery in the deployment server. Monit is easiest to install through apt-get:

sudo apt-get install monit

you can add monit configuration files in /etc/monit/monitrc/

Monitrc Script for NodeJS application

# save to /etc/monit/monitrc/monit_nodeapp.conf
check process monit_nodeapp with pidfile /path/to/production_dir/current/tmp/pids/app.pid
  group staff
  start program = "/path/to/production_dir/current/config/monit_nodeapp.start"
    as uid deploy and gid staff
  stop program = "/path/to/production_dir/current/config/monit_nodeapp.stop"
    as uid deploy and gid staff
  if failed unixsocket /path/to/production_dir/current/tmp/sockets/app.sock
      protocol HTTP request "/?status" for 5 cycles then restart

Monit binary script to start your app if terminated unexpectedly

#!/bin/bash
# save in /path/to/production_dir/current/config/monit_nodeapp.start 
echo "Start NodeJS App:"
echo "  [+] Enter directory /path/to/production_dir/current"
cd /path/to/production_dir/current
echo "  [+] Run NodeJS script"
exec `NODE_ENV=production node server > /dev/null 2>&1 & echo $! > tmp/pids/app.pid`
echo "  [+] Done !!"

Monit binary script to stop your app if no response or hang

#!/bin/bash
# save in /path/to/production_dir/current/config/monit_nodeapp.stop 
echo "Stop NodeJS App:"
echo "  [+] Enter directory /path/to/production_dir/current"
cd /path/to/production_dir/current
echo "  [+] Run Termination script"
exec `kill -15 `cat /path/to/production_dir/current/tmp/pids/app.pid` || true`
echo "  [+] Done !!"

You can check here to install Monit Configuration with start and stop binary scripts for Nginx, PostgreSQL, Redis.

Then reload your monit

sudo monit reload

---

It's time to deploy

Back to your development machine and app folder then run this command

mina deploy

Once the process is completed, you can see in deployment server a folder "current" in your deployment folder is linked to released version's folder after that open your domain in the browser. For the example codes of this sharing knowledge you can check my github

Many thanks to visit my wall, if you like this please share this wall. If you have any comment, please leave me a message. cheers ^_^