Processing Apache Log Files in Graphite

After messing around looking for a tool chain that would let us import Apache access_log statistics from a growing number of web machines into Graphite I've settled on rsyslog and etsy's logster. It's really quite simple and applicable to all sorts of logging problems. Here's a quick rundown of how it works.

Your apache configuration needs to utilize the logger utility. You can read more about it here but its available all over the dang place. http://linux.die.net/man/1/logger

In your Apache conf you'll need to edit your logging to something like this:

CustomLog /path/to/access_log foobar
CustomLog "|/path/to/logger -t httpd -p local6.info" foobar
CustomLog /path/to/access_log foobar

In your /etc/rsyslog.conf all you need is a simple forwarder for httpd.

if $programname == 'httpd' then @@logging.meow.com:10512

If you notice /var/log/messages is filling up with your access_log messages you need to include 'local6.none' on the rsyslogl.conf line defining /var/log/messages.

Now you need to collect all the stats some place. Configure another machine running rsyslog (at logging.meow.com) to receive those log events.
Paste this at the end of your rsyslog.conf:

# process remote messages
# define new ruleset and add rules to it:
$ModLoad imtcp.so
$RuleSet remote
if $programname == 'httpd' then /path/to/my/unified.log
# bind ruleset to tcp listener
$InputTCPServerBindRuleset remote
# and activate:
$InputTCPServerRun 10512

Cool, that's pretty much it for pulling in the logs. Now you'll want to install etsy's logster on the log-capturing machine from their github repo. https://github.com/etsy/logster

And add this Apache log parser I modified from their example to parse your logs. https://gist.github.com/2997169

I hope this was easy to follow. If you're looking for any further reading you can try this loggly wiki page: http://wiki.rsyslog.com/index.php/Working_Apache_and_Rsyslog_configuration

or the rsyslog documentation: http://www.rsyslog.com/doc/manual.html

. Thanks!