Using xargs and positional args

xargs can take a list of tokens and run a number of processes (say the number of CPUs) with one or more of those tokens until the list is empty.

If I have a command "process-image" that takes one photo filename argument, I could process the images on a four cpu computer with "process-image" and one argument at a time.

ls -d *.jpg | xargs -P 4 -n 1 -- ./process-image

This doesn't work well if I wanted to copy the files in multiple streams:

ls -d *.jpg | xargs -P 4 -n 1 -- scp ... dad@garage:images/

The ... is where the filename should be but xargs append the filename to the end of the scp command and any arguments.

Here's how to run any command with xargs with your choice of where the arguments are inserted.

ls -d *.jpg | xargs -P 4 -n 1 -- bash -c 'scp $0 dad@garages:images/'

The bash -c takes one string to run, and any arguments after become positional arguments starting at $0 (instead of $1).

To copy 8 files at a time, change the -n from 1 to 8, and include $@ for arguments $1 onwards:

ls -d *.jpg | xargs -P 4 -n 8 -- bash -c 'scp $0 $@ dad@garages:images/'

Here's a function to hide the arcana:

function runmany {
  cpu="$1"; shift; args="$1"; shift
  xargs -P $cpu -n $args -- bash -c "$*"
}

Run it like:

ls -d *.jpg | runmany  scp \$0 \$@ dad@garages:images/

xargs has an -I% which inserts the arguments into your command string. Using bash -c is good when you want to insert the arguments all over the place.