God and Boot Order Dependencies

I noticed quite a few questions on how to support boot order dependencies using God. Below are instructions on how I used the God DSL to ensure process boot order.

Conditions of God

A new God Condition is needed and it must inherit from one of the God Condtion classes.
A new condition has the following requirements:

• Condition class name is camel case
• Is derived from one of three Condition classes

PollCondition - poll the system based on a defined interval
EventCondition - condition occurs based on a system event
TriggerCondition - when a task is moving from one state to another

I used a PollCondition to validate that the middleware is up and running.

def middleware_up?
  s = God.status
  (s['mongo'][:state] == :up) && (s['apollo'][:state] == :up)
end

class MiddlewareRunning < God::PollCondition

  def valid?
    super
  end

  def test
    middleware_up?
  end
end

Use the God.status class method to get the state of the mongo and apollo servers.

Task State Transitions

Using the God DSL you create a Watch for each server. A Watch can be in one of the following states:

1. :init - When God creates the watch
2. :start - God is attempting to start the process
3. :up - Process is running
4. :restart - God is attempting to restart the process

Watch instances depedent on the middleware will need to be held in the :init state until the middleware is in the :up state. So, I set the transition as follows:

w.transition(:init, :start) do |on|
  on.condition(:middleware_running) do |c|
    c.interval = 3.seconds
  end
end

Notice that the :middleware_running symbol is the camel cased class MiddlewareRunning smoothed out.

w.transition(:up, :init) do |on|
  on.condition(:process_exits)
end

Also, if the process exits we move back to the init state and ensure the middleware is up and running before moving to the :restart state

Adding Dependents to the Watch

Now that the middleware is up and running there are a few processes where I want to ensure boot order.

A member for dependents is added to the Watch. The array of associated dependents is assigned to the new dependents Watch member:

# This adds a dependents member array to the watch instance and then adds the dependents to the member
def add_dependents(klass, *dependents)
  class << klass
    attr_accessor :dependents
  end
  klass.dependents = dependents
end

The Watch DSL looks like:

God.watch do |w|
   w.group         = "stooges_servers"
   w.name          = "moe_server"
   w.start         = "#{APP_ROOT}/bin/moe_server --log-file #{APP_ROOT}/log/moe.log"
   w.log           = "#{APP_ROOT}/log/moe.log"
   w.interval      = 30.seconds
   add_dependents w, "larry_server", "curly_server"
   .
   .
   .

Dependent Condition

Ensures that the middleware is up and running and any dependent processes are running before moving to the start state. A Condtion is associated with a Watch. Each Condition instance has access to the associated Watch instance.

class DependentsRunning < God::PollCondition
  def valid?
    super
  end

  def test
    middleware_up? ? all_dependents_up? : false
  end

  def all_dependents_up?
    gps = God.status
    @watch.dependents.each do |name|
      dependent = gps[name]
      if not dependent.nil?
        if dependent_up? dependent  
           applog @watch, :info, "Dependent process up: #{name}" 
         else 
            return false
          end
      else
        return false
      end
    end
    return true  # if for some reason the dependent list is empty
  end

  def dependent_up?(dependent)
    dependent[:state] == :up ? true : false
  end
end

Extra Code

In the sections on Task State Transitions only the transitions for :init to :start and :up to :init are discussed. Here are the other state transitions which are taken pretty much from the God documentation.

def life_cycle(w)
  # lifecycle
  w.lifecycle do |on|
    on.condition(:flapping) do |c|
      c.to_state = [:start, :restart]
      c.times = 5
      c.within = 5.minute
      c.transition = :unmonitored
      c.retry_in = 10.minutes
      c.retry_times = 5
      c.retry_within = 2.hours
    end
  end
end

def server_transitions(w)

  w.transition(:start, :up) do |on|
    on.condition(:process_running) do |c|
      c.running = true
    end
  end

  w.transition([:start, :restart], :up) do |on|
    on.condition(:process_running) do |c|
      c.running = true
    end

    # failsafe
    on.condition(:tries) do |c|
      c.times = 5
      c.transition = :start
    end
  end

  # init if process is not running
  w.transition(:up, :init) do |on|
    on.condition(:process_exits)
  end
  life_cycle w
end

Closing Notes

It was really easy using the God DSL to quickly deploy a process management infrastructure. I needed a quick answer to process management and how to ensure a start order. I do a lot of work on the frontend, backend and in many environments, I needed a low learning curve answer. I am not a God expert, so caveat emptor. Please, correct, share and improve.