Having log files aggregated to a single server (or even multiple servers) is important both for marketing (analytic) and for security. However, the reason why I am interested in aggregating log entries: I can avoid log rotation. Possible problems with log rotation:
- Need to make sure that half of your servers rotate their log files at one time, and the other half rotate their log files at another time so that your site is not down at any moment.
- Developers (without careful design) may use HTTP request to trigger long running operation. Rotating the log files cause these child processes to be killed off and it caused us some pain.
- Subversion servers should avoid log rotation. What happens when log rotation happen during a check-in?
- Sometimes log rotation is started, but never finish, or finished but left the server in bad state (not started). The cause of this is usually due to a very large log file (one that has not been rotated for weeks)
- Empty log files (does happen sometimes, related to log rotation, but not sure why)
- Having to log to local files means that we have to do log rotation on the box that apache is running. Log rotation usually restart apache. If we are not careful, all the servers may be restarted at the same time, which means that for some time (may be just for a minute), our entire production environment is down, which may or may not be acceptable. Having log files forwarded to a remote host, mean we don't have to do log rotation (avoid restarting apache). Log rotation can be done on remote host.
Edit your httpd.conf to include:
ErrorLog syslog:local1
This tells Apache to send the error log output to the syslog facility called local1. You need to tell your syslog server to send this log output off to another server. Exactly how you do this may vary from one syslog implementation to another. However, the normal Linux syslog uses a file called /etc/syslog.conf to configure how syslog streams.
Because the Apache error log uses syslog-standard severity ratings, you can use normal syslog configurations to split syslog output into different files based on severity. For example, if you wanted to log only critical errors to a particular file, add the line in your /etc/syslog.conf file:
local1.crit /var/log/apache.crit
This causes the logging of entries of level crit and higher to the file /var/log/apache.crit.
To send log entries to a remote server:
local1.* @192.168.200.12
You also need to configure the remote syslog server to accept these log entries. For most syslog servers, this involves adding the -r flag to the startup parameters for the syslog server. However, this may vary from one syslog implementation to another, so consult the documentation of whatever particular syslog server you're running.
Unfortunately the above mechanism only works for the error_log file. Check with apache documentation to see if streaming of access_log to a log server is supported in future version. For now, write a Perl script:
#!/usr/bin/perl
use Sys::Syslog qw( :DEFAULT setlogsock );
setlogsock('unix');
openlog('apache', 'cons', 'pid', 'local2');
while ($log = <STDIN>) {
syslog('notice', $log);
}
closelog
Put this script somewhere and make it executable.
Second, point your access log at this script using the piped logfile syntax:
CustomLog |/usr/local/apache/bin/apache_syslog combined
The apache_syslog script will launch when your web server starts up, and it processes logfile entries as it receives them from the server. The while loop will continue for the lifetime of the Apache server process.
As you can see in the script, it sends these entries to a syslog facility named local2, so configure your syslog server with an entry in /etc/syslog.conf like:
local2.* @192.168.200.12
You can have multiple CustomLog directives, so you can have multiple logfiles. If you want to have a local logfile in addition to a remote one, just add an extra CustomLog.
You can have one CustomLog directive for logging in the format supported by your log analyzer (analytic for marketing people), another CustomLog directive that log extra information for debugging, and another CustomLog directive for another purpose.
keep your clocks synchronized with NTP
Second, I noticed that my syslog server will condense several log entries into a single line in the logfile if they're the same. For example, a repeated request for a particular missing file resulted in:
Aug 31 20:18:05 192.168.200.10 last message repeated 19 times
Finally, it is worth mentioning that there are third-party modules available to do logging for multiple servers. The most popular of these modules is mod_log_spread, which will likely be a more efficient solution for very large (many servers) installations.
http://www.oreillynet.com/pub/a/sysadmin/2006/10/12/httpd-syslog.html
http://www.devshed.com/c/a/Apache/Logging-in-Apache/3/
#ErrorLog logs/error_log
ErrorLog syslog:local0
LogFormat "%t %a \"%r\" %>s %T %D %B %P %X %I %O \"%{Referer}i\" \"%{User-Agent}i\" %{memory_usage}n \"%{PHPSESSID}C\" \"%{Set-Cookie}o\" \"%{Content-Type}o\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent
#CustomLog logs/access_log combined
CustomLog "|/usr/bin/apache_syslog.pl" combined
/usr/bin/apache_syslog.pl:
#!/usr/bin/perl
use Sys::Syslog qw( :DEFAULT setlogsock );
setlogsock('unix');
openlog('httpd', 'pid', 'local1');
while ($log = <STDIN>) {
syslog('notice', $log);
}
closelog();
What is the flip side of forwarding log files to a remote host?
One of the reasons for forwarding log files to remote host is so that we can disable log rotation on local host (because log rotation restart apache). However, sometime we find it desirable / necessary to disable this remote logging, and log to local file. We need to remember to re-enable local log rotation. So to disable logrotate for apache, rather than removing /etc/logrotate.d/httpd, we should comment out its content. If we need to re-enable log rotation for apache, we can just uncomment lines in /etc/logrotate.d/httpd
