Some Unix Copy/Find/Replace magic
I've been working with a lot of flat files lately and found these commands quite handy when it comes to searching for specific text patterns, replacing those matches with alternatives and copying/deleting or executing any other action on the output.
Tested on OS X and openSuSE, changes might be necessary for different flavors of Linux
_ Note: Some of the commands below ( as flagged by @serverhorror ) can exceed the maximum allowed number of arguments for a given machine: getconf ARG_MAX resulting in the failure of execution. Use the below with care. _
Get the list of files with a matching pattern in a given directory
egrep [options] PATTERN [FILE...]Example
egrep -Rl 'IP445A' ./*
-l, --files-with-matches
Suppress normal output; instead print the name of each input file from which output would normally have been printed. The scanning will stop on the first match.
-R, -r, --recursive
Read all files under each directory, recursively; this is equiv-alent to the -d recurse option.Get the number of matching files
Returns the number of files containing text matching the pattern.
egrep [options] PATTERN [FILE...] | wc -l
-l
The number of lines in each input file is written to the standard output.Example
egrep -Rl 'IP445A' ./* | wc -lStore the names of each matching file into another text file
egrep [options] PATTERN [FILE...] > container.txtExample
egrep -Rl 'IP445A' ./* > list.txtCopy files with a matching pattern to a new directory
cp `egrep [options] PATTERN [FILE...]` <New Directory>Example
cp `egrep -Rl "IP445A" ./*` ../matching/The cp command will execute egrep first and copy all results to a new directory.
Note that egrep is wrapped with backticks and not single quotes.
Replace matching pattern with another string
grep -l '' <directory> | xargs sed -i "" 's/<regex>/<replacement>'
-i extension
Edit files in-place, saving backups with the specified extension. If a zero-length extension is given, no backup will be saved. It is not recommended to give a zero-length extension when in-place editing files, as you risk corruption or par-tial content in situations where disk space is exhausted, etc.Example
grep -l '' ./* | xargs sed -i "" 's/IP4445A/IP445A/'Please refer to the man pages (in the sources section below) for detailed explanation of the parameters.
Read a list of files from a text file and copy each one to a new directory
#OSX use -I / for other Linux distros use -J
cat list.txt | xargs -I % cp % <absolute path>Example
cat list.txt | xargs -I % cp % /Users/bassemd/newFolderSources
I recommend you go over the man pages for each command to have a deeper understanding of it and its diverse options
Written by Bassem Dy
Related protips
6 Responses
I think most of those are actually bad advice. for example the "Copy files with a matching pattern to a new directory" is subject to getconf ARG_MAX in the shell...
egrep ... | wc -l # useseless exec
egerp -c ....
egrep -Rl 'IP445A' ./*
find . -iregex 'IP445A' [-print|-ls]
cp `egrep -Rl "IP445A" ./*` ../matching/ # may exceep max command line arguments
find . -iname '*IP445A*' -exec cp {} ../matching/ + # not even xargs is needed + does the same
@serverhorror thank you for the feedback! Greatly appreciated.
I will edit the post to include a notice about exceeding the max command line arguments.
However, all your alternatives for do not satisfy the requirements:
- of getting the total number of matching files and not matches inside a given file, hence the use of
wc(that provides other options as well) - of recursively looking for matches inside each file of a given parent directory and children
What i usually use is grep -Ril "whatever" that will match anything recursively, case-insensitive and list the file names, when I'm not matching regular expressions, and it works well for finding files containing certain string
@eksperimental you're right, that will work but there are subtle differences between egrep grep and fgrep which are perfectly explained in this Stackexchange answer (which I highly recommend going over) http://unix.stackexchange.com/a/17951
In your sed example you give -l in the description, but -i in the example. I think -i is what you wanted.
@psycotica0 Thank you for flagging the typo. It has been fixed. The -i option with blank extension forces sed to edit the files themselves and not create other versions.
