Last Updated: February 25, 2016
·
8.221K
· pmaoui

AWK to explore your access log files

Global usage

Line that match a particular word :

awk '/foo1/' foo.txt

Print only specific field (splitted by whitespace character) :

awk '{print $2,$4,$NF;}' file.txt

NF is the total number of fields in a record

Print only specific field (splitted by ":") :

awk -F : '{print $2,$4;}' file.txt

access.log of apache2

Display IP address for a specific resource :

"http://www.url.com/foo1" access.log.log|awk '{print $1;}'

To count unique access to a specific resource:

awk -F'[ "]+' '$7 == "/foo1/image.jpg" { ipcount[$1]++ }
    END { for (i in ipcount) {
        printf "%15s - %d\n", i, ipcount[i] } }' /var/log/apache2/access.log
    }
}

To detect flooded paths (count how much one ip try to request)

head -500 access.log | sed 's/[0-9]*//g' | awk '{url[$1$8]++} END{for (i in url) {print url[i], i}}' | sort -nr
tail -5000 access.log | sed 's/[0-9]*//g' | awk '{url[$1$8]++} END{for (i in url) {print url[i], i}}' | sort -n