Throttling parallel processes in your shell

Add the following bash/zsh function to your .bashrc or .zshrc:

function throttle() {
    local throttle_procs
    if [[ -z $1 ]] ; then
        throttle_procs=2
    else
        throttle_procs=$1
    fi

    while [[ $(jobs -p | wc -l) -ge $throttle_procs ]] ; do
        sleep 0.33;
    done
}

Then use it like so:

$ for i in $(seq 1 10); do
      echo $i: $(date)
      sleep $(shuf -i 1-5 -n 1) & throttle 3
   done

Your shell will spawn up to three processes (based on the argument passed to the throttle function) and then block until one or more of the backgrounded processes are completed.

(This is a somewhat modified example from here: http://prll.sourceforge.net/shell_parallel.html)