sgjf8g
Last Updated: February 25, 2016
·
1.676K
· bassemdy

Some Unix Copy/Find/Replace magic

I've been working with a lot of flat files lately and found these commands quite handy when it comes to searching for specific text patterns, replacing those matches with alternatives and copying/deleting or executing any other action on the output.

Tested on OS X and openSuSE, changes might be necessary for different flavors of Linux

_ Note: Some of the commands below ( as flagged by @serverhorror ) can exceed the maximum allowed number of arguments for a given machine: getconf ARG_MAX resulting in the failure of execution. Use the below with care. _

Get the list of files with a matching pattern in a given directory

egrep [options] PATTERN [FILE...]

Example

egrep -Rl 'IP445A' ./*

-l, --files-with-matches
      Suppress normal output; instead print the name of  each input file  from  which output would normally have been printed. The scanning will stop on the first match.

-R, -r, --recursive 
      Read all files under each directory, recursively; this is equiv-alent to the -d recurse option.

Get the number of matching files

Returns the number of files containing text matching the pattern.

egrep [options] PATTERN [FILE...] | wc -l

-l 
      The number of lines in each input file is written to the standard output.

Example

egrep -Rl 'IP445A' ./* | wc -l

Store the names of each matching file into another text file

egrep [options] PATTERN [FILE...] > container.txt

Example

egrep -Rl 'IP445A' ./* > list.txt

Copy files with a matching pattern to a new directory

cp `egrep [options] PATTERN [FILE...]` <New Directory>

Example

cp `egrep -Rl "IP445A" ./*` ../matching/

The cp command will execute egrep first and copy all results to a new directory.

Note that egrep is wrapped with backticks and not single quotes.

Replace matching pattern with another string

grep -l '' <directory> | xargs sed -i "" 's/<regex>/<replacement>'

-i extension
         Edit files in-place, saving backups with the specified extension.  If a zero-length extension is given, no backup will be saved.  It is not recommended to give a zero-length extension when in-place editing files, as you risk corruption or par-tial content in situations where disk space is exhausted, etc.

Example

grep -l '' ./* | xargs sed -i "" 's/IP4445A/IP445A/'

Please refer to the man pages (in the sources section below) for detailed explanation of the parameters.

Read a list of files from a text file and copy each one to a new directory

#OSX use -I / for other Linux distros use -J
cat list.txt | xargs -I % cp % <absolute path>

Example

cat list.txt | xargs -I % cp % /Users/bassemd/newFolder

Sources

I recommend you go over the man pages for each command to have a deeper understanding of it and its diverse options

6 Responses
Add your response

18794

I think most of those are actually bad advice. for example the "Copy files with a matching pattern to a new directory" is subject to getconf ARG_MAX in the shell...

egrep ... | wc -l # useseless exec
egerp -c ....

egrep -Rl 'IP445A' ./*
find . -iregex 'IP445A' [-print|-ls]

cp `egrep -Rl "IP445A" ./*` ../matching/ # may exceep max command line arguments
find . -iname '*IP445A*' -exec cp {} ../matching/ + # not even xargs is needed + does the same
over 1 year ago ·
18796

@serverhorror thank you for the feedback! Greatly appreciated.
I will edit the post to include a notice about exceeding the max command line arguments.

However, all your alternatives for do not satisfy the requirements:

  1. of getting the total number of matching files and not matches inside a given file, hence the use of wc (that provides other options as well)
  2. of recursively looking for matches inside each file of a given parent directory and children
over 1 year ago ·
18811

What i usually use is grep -Ril "whatever" that will match anything recursively, case-insensitive and list the file names, when I'm not matching regular expressions, and it works well for finding files containing certain string

over 1 year ago ·
18817

@eksperimental you're right, that will work but there are subtle differences between egrep grep and fgrep which are perfectly explained in this Stackexchange answer (which I highly recommend going over) http://unix.stackexchange.com/a/17951

over 1 year ago ·
18821

In your sed example you give -l in the description, but -i in the example. I think -i is what you wanted.

over 1 year ago ·
18822

@psycotica0 Thank you for flagging the typo. It has been fixed. The -i option with blank extension forces sed to edit the files themselves and not create other versions.

over 1 year ago ·