rsyslog

http://www.rsyslog.com/guides/
http://blog.gerhards.net/2007/08/why-does-world-need-another-syslogd.html
http://www.rsyslog.com/doc/manual.html
http://www.rsyslog.com/faq/
http://www.rsyslog.com/security-advisories/
http://www.rsyslog.com/video-tutorials/
http://www.rsyslog.com/doc/property_replacer.html
http://www.rsyslog.com/doc/rsyslog_conf_filter.html
http://www.rsyslog.com/doc/log_rotation_fix_size.html
http://wiki.rsyslog.com/index.php/Working_Apache_and_Rsyslog_configuration
http://www.rsyslog.com/doc/rsyslog_conf.html
http://www.rsyslog.com/doc-rsyslog_tls.html - printed
http://www.rsyslog.com/doc/rsyslog_secure_tls.html
http://www.howtoforge.com/logging_with_rsyslog_phplogcon_debian_etch
http://kb.monitorware.com/tutorial-rsyslog-server-phplogcon-on-fedora-core-4-t1555.html
http://docs.splunk.com/Documentation/Storm/latest/User/Howtosetuprsyslog — with Splunk
http://serverfault.com/questions/381353/rsyslogd-any-way-to-get-around-the-number-of-local-facilities
http://urbanairship.com/blog/2010/10/05/centralized-logging-using-rsyslog/
http://en.gentoo-wiki.com/wiki/Rsyslog

http://www.rsyslog.com/video-tutorials/
http://www.youtube.com/watch?v=8YkhLSbW7Wg

http://fog.ccsf.edu/~gboyd/cs260a/online/syslog/introduction.html

http://www.thegeekstuff.com/2010/07/logrotate-examples/#more-4826
http://www.ducea.com/2006/06/06/rotating-linux-log-files-part-1-syslog/
http://ubuntuforums.org/showthread.php?t=866853
http://www.linuxhomenetworking.com/wiki/index.php/Quick_HOWTO_:_Ch05_:_Troubleshooting_Linux_with_syslog
http://wpollock.com/AUnix2/Logging.htm
http://www.thegeekstuff.com/2010/07/logrotate-examples/
http://linuxers.org/howto/howto-use-logrotate-manage-log-files
http://rapidtechguide.blogspot.com/2010/08/basic-tutorial-on-logrotate.html
http://www.vanstormbroek.nl/blog/?p=5
http://www.amitnepal.com/tutorial-on-logrotate/
http://www.linux-tutorial.info/modules.php?name=ManPage&sec=8&manpage=logrotate
http://linuxconfig.org/logrotate

http://www.rsyslog.com/doc/rsyslog_conf_actions.html
http://www.rsyslog.com/doc/queues.html
http://www.rsyslog.com/sending-messages-to-a-remote-syslog-server/
http://www.thegeekstuff.com/2012/01/rsyslog-remote-logging/
http://techtots.blogspot.com/2011/12/rsyslogd-logging-to-remote-server.html
http://www.freeklijten.nl/home/2011/08/16/A-tutorial-on-remote-logging-with-rsyslog
http://serverfault.com/questions/396136/how-to-forward-specific-log-file-outside-of-var-log-with-rsyslog-to-remote-serv
http://www.linux.com/community/forums/productivity/prevent-rsyslog-from-writing-messages-from-remote-hosts-to-var-log-messages/16284
http://kb.monitorware.com/remote-logging-on-redhat-with-rsyslog-t1706.html
http://www.linuxquestions.org/questions/linux-newbie-8/syslog-remote-logging-with-rsyslog-server-903547/
https://gist.github.com/naokij/1846489
http://loggly.com/support/sending-data/logging-from/syslog/rsyslog/
http://www.9thport.net/articles/how-to-setup-central-logging-with-rsyslog
http://lists.adiscon.net/pipermail/rsyslog/2009-December/026304.html
http://users.telenet.be/mydotcom/howto/linux/syslogserver.html
http://www.gossamer-threads.com/lists/rsyslog/users/3135
http://kb.monitorware.com/tls-forwarding-inconsistent-t10559.html

Messages refer to a facility (auth, authpriv, daemon, cron, ftp, lpr, kern, mail, news, syslog, user, uucp, local0 … local7) and are assigned a severity (Emergency, Alert, Critical, Error, Warning, Notice, Info or Debug) by the sender of the message.

logger is a command line utility that can send message to syslog.

logger -p local0.notice 'This is a test from db1'

What RFCs are involved?

The Internet Engineering Task Force document syslog in RFC 3164, which was obsolete by RFC 5424.

What are the facilities defined?

What are the severity levels defined?

Rsyslog is an enhanced multi-threaded syslogd with a focus on security and reliability.

Features:

  • support for on-demand disk buffering
  • reliable syslog over TCP, SSL, TLS and RELP
  • writing to databases (MySQL, PostgreSQL, Oracle, and many more)
  • optional web interface phpLogCon
  • email alerting
  • fully configurable output formats (including high-precision timestamps)
  • the ability to filter on any part of the syslog message
  • on-the-wire message compression
  • ability to convert text files to syslog
  • ability to configure backup syslog/database servers - if the primary fails, control is switched to a prioritized list of backups
  • support for receiving messages via reliable RFC 3195 delivery (a bit clumpsy to build right now…)
  • ability to generate file names and directories (log targets) dynamically, based on many different properties
  • control of log output format, including ability to present channel and priority as visible log data
  • good timestamp format control; at a minimum, ISO 8601/RFC 3339 second-resolution UTC zone
  • ability to reformat message contents and work with substrings
  • support for log files larger than 2gb
  • support for file size limitation and automatic rollover command execution
  • support for TLS-protected syslog (both natively and via stunnel)
  • ability to filter on any part of the message, not just facility and severity
  • ability to use regular expressions in filters
  • support for discarding messages based on filters
  • ability to execute shell scripts on received messages
  • control of whether the local hostname or the hostname of the origin of the data is shown as the hostname in the output
  • ability to preserve the original hostname in NAT environments and relay chains

It is a drop-in replacement for stock syslogd and able to work with the same configuration file syntax.

Syslog has the concept of "channels". How can we log to different channels? How can we use log4j to log to different syslog channel and allow searching only on those channels. System logs should not be visible to developers.

How to configure rsyslog with logrotate? How to index log message? How to configure rsyslog so that log messages can be searched using Apache Solr or some other open source tool?

Perhaps within our application, whenever an exception is thrown, we can log exceptions to a separate channel.

Developer should be able to view the log entries chronologically (order by timestamp)

/etc/rsyslog.conf

On the sending end, how can we configure Apache to use rsyslog?

CustomLog "|/usr/bin/logger -t apache -i -p local6.notice" procurios-syslog
error_log syslog

On the sending end, how can we forward all log messages to the remote host (1.2.3.4)?

To forward all log message to a remote host, modify /etc/rsyslog.conf:

*.* @1.2.3.4:514

Now, if you restart rsyslog, every priority of every facility will be sent to the server with IP 1.2.3.4 over UDP. By adding a second @ in front of the first and changing your port you can send log message via TCP instead of UDP. The *.* may be a bit much. If you know that all you are going to do with specific logs is drop them on the receiving server, you might as well drop them on the sending servers so that you don't waste bandwidth.

On the receiving end, how can we configure rsyslog to use UDP?

$ModLoad imudp
$UDPServerAddress 1.2.3.4
$UDPServerRun 514

On the receiving end, how can we verify that rsyslog is listening on UDP port 514?

netstat -nlp

You should see something like:

udp        0      0 1.2.3.4:514           0.0.0.0:*          16637/rsyslogd

On the receiving end, how can we store all log messages in a single file?

This is not a good idea, but just to give an example.

*.* /var/log/oneGiantLogFile.log

On the receiving end, how can we configure rsyslog to store log file from different server in separate log file?

We will use templates:

$template syslog,"/var/log/external/%fromhost%/syslog.log"
$template apacheError,"/var/log/external/%fromhost%/apache/error.log"
$template apacheAccess,"/var/log/external/%fromhost%/apache/%msg:R,ERE,1,ZERO:imp:([a-zA-Z0-9\-]+)\.--end%-access.log"
$template mailError, "/var/log/external/%fromhost%/mail/error.log"

You'll notice the %fromhost%. This is a placeholder which is dynamically replaced with the DNS-resolved hostname of the machine the current log came from. Read more about property replacer

Now, for the actual filtering:

local7.* ?apacheError
& ~

local6.notice ?apacheAccess
& ~

*.* ?syslog

Apache uses local7 to send error logs and we told apache to use local6.notice for access logs, all we do now is put them in their dynamic files. The question mark is necessary to have rsyslog know a template is following. If an error log is coming from v004 it will be put into /var/log/external/v004/apache/error.log if it comes from v027 it will be stored in /var/log/external/v027/apache/error.log. On the next line (which seems to be necessary in this case) there are an ampersand and a tilde. The tilde tells rsyslog to drop all logs that were filtered out by the preceding command, the ampersand is merely used to connect the two lines.

How to configure rsyslog to store log messages in a MySQL database?

/etc/rsyslog.conf:

$ModLoad MySQL
*.*       >127.0.0.1,rsyslog,rsyslog,ENTER-YOUR-NEW-RSYSLOG-PASSWORD-HERE

…and remove the comments in front of the lines that deal with the TCP syslog reception:

# /etc/rsyslog.conf Configuration file for rsyslog v3.
#
# For more information see
# /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
$ModLoad MySQL
*.* >127.0.0.1,rsyslog,rsyslog,ENTER-YOUR-NEW-RSYSLOG-PASSWORD-HERE
#################
#### MODULES ####
#################
$ModLoad imuxsock # provides support for local system logging
$ModLoad imklog # provides kernel logging support (previously done by rklogd)
#$ModLoad immark # provides --MARK-- message capability
# provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514
# provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 514
###########################
#### GLOBAL DIRECTIVES ####
###########################

Now install loganalyzer:

wget http://download.adiscon.com/loganalyzer/loganalyzer-3.4.1.tar.gz
tar -xvzf loganalyzer-3.4.1.tar.gz
mv loganalyzer-3.4.1/ /var/www/
cd /var/www
chown www-data:www-data * . -Rf
mv loganalyzer-3.4.1/ loganalyzer
cd contrib/
cp * ./../src/
cd ./../src/
sh ./configure.sh

And we are ready to enter the final setup of LogAnalyzer using our web-browser. Point your web-browser to the fixed IP address of our rsyslog server that in this demo is http://192.168.0.15/loganalyzer/src/install.php. Go trough the simple setup script (it's pretty much… next -> next.) Now you should have working rsyslog server with LogAnalyzer up and running.

Now, on the sending end, create a spool directory:

mdir /rsyslog/work

that is used in case network connectivity is lost, and change the /etc/rsyslog.conf:

# provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514
$WorkDirectory /rsyslog/work # default location for work (spool) files
$ActionQueueType LinkedList # use asynchronous processing
$ActionQueueFileName srvrfwd # set file name, also enables disk mode
$ActionResumeRetryCount -1 # infinite retries on insert failure
$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down
*.* @@YOUR-RSYSLOG-SERVER-ADDRESS-HERE
###########################
#### GLOBAL DIRECTIVES ####
###########################

References:
http://www.freeklijten.nl/home/2011/08/16/A-tutorial-on-remote-logging-with-rsyslog
http://www.howtoforge.com/centralized-rsyslog-server-monitoring

Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License