vacmaa
Last Updated: February 25, 2016
·
9.752K
· destructuring

Using xargs and positional args

xargs can take a list of tokens and run a number of processes (say the number of CPUs) with one or more of those tokens until the list is empty.

If I have a command "process-image" that takes one photo filename argument, I could process the images on a four cpu computer with "process-image" and one argument at a time.

ls -d *.jpg | xargs -P 4 -n 1 -- ./process-image

This doesn't work well if I wanted to copy the files in multiple streams:

ls -d *.jpg | xargs -P 4 -n 1 -- scp ... dad@garage:images/

The ... is where the filename should be but xargs append the filename to the end of the scp command and any arguments.

Here's how to run any command with xargs with your choice of where the arguments are inserted.

ls -d *.jpg | xargs -P 4 -n 1 -- bash -c 'scp $0 dad@garages:images/'

The bash -c takes one string to run, and any arguments after become positional arguments starting at $0 (instead of $1).

To copy 8 files at a time, change the -n from 1 to 8, and include $@ for arguments $1 onwards:

ls -d *.jpg | xargs -P 4 -n 8 -- bash -c 'scp $0 $@ dad@garages:images/'

Here's a function to hide the arcana:

function runmany {
  cpu="$1"; shift; args="$1"; shift
  xargs -P $cpu -n $args -- bash -c "$*"
}

Run it like:

ls -d *.jpg | runmany  scp \$0 \$@ dad@garages:images/

xargs has an -I% which inserts the arguments into your command string. Using bash -c is good when you want to insert the arguments all over the place.

3 Responses
Add your response

1114

Thanks, I was looking for a way to add an argument at the end of the command. I could only find stuff about the -I option but that allows only one argument at a time. This bash way works great and allows me to only type my password the minimum number of times when copying a lot of files.

The only change is that you should quote the arguments to properly handle arguments with spaces.

ls -1 *.jpg | xargs -d "\n" -P 4 -n 8 -- bash -c 'scp "$0" "$@" dad@garages:images/'

You may think that quoting the $@ would make it be treated as one argument but bash has a convenient special case. From the bash man page: "When the expansion occurs within double quotes, each parameter expands to a separate word."

Hope that helps someone.

over 1 year ago ·
20859

Thanks!

over 1 year ago ·
20896

GNU Parallel has built in support for positional arguments:

seq 12 | parallel -N4 echo arg4:{4} arg1:{1} arg3:{3} arg2:{2}

For minimizing number of times you have to enter password for scp try:

find ... | parallel -j1 -X scp {} other@server:

Learn more: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1 http://www.gnu.org/software/parallel/parallel_tutorial.html

over 1 year ago ·