sar (System Activity Reporter) is part of the systat package.
After installing the package, run the rc script to start the collection daemon for the first time. After that, it will be called by cron in crontabs installed by the package. The Red Hat and SuSE RPMs install the crontabs in /etc/cron.d/sysstat, so you can adjust the sampling rate there. You may want to increase the granularity for busy systems with performance issues. Reduce the frequency of sampling for systems that are cruising along.
The sysstat crontab invokes a shell script, sa1, which in turn calls a program named sadc (System Activity Data Collector) which gathers data samples on CPU, memory, io, and network devices, creating a snapshot of system performance for that moment in time. The data is written out, in a binary format, to /var/log/sa/saXX where the XX is the two digit day of the month. This scheme allows for the files to be overwritten as the month rolls over, so if they need to be archived, set up a shell script to rotate them elsewhere. Also, be aware that, on Red Hat at least, the /usr/lib/sa/sa2 script which creates the daily summaries each night, also removes any sar data files older than 7 days. This is configurable in /etc/sysconfig/sysstat.
sar, when invoked without any option, reads the file in /var/log/sa for today. It replays the recorded data, showing the historical data as of a certain time, not now. This is exactly what we wanted to accomplish the three objectives: getting historical data, finding usage patterns and understanding trends
To look at data from another day, use -f option:
sar -f /var/log/sa/sa08
sar can also be used to collect real time data at intervals. In this example, we sample at five second intervals three times:
sar 5 3
To cause these stats to be written to a file, use -o option.
The most common options are for CPU, memory, disk activity, network activity.
Tuning is a complex activity that requires a sysadmin to make a careful analysis of the hardware, users, and applications that comprise each unique system. One way to do this is to collect baseline data. sar reports (run when a performance problem rears its ugly head), is not much useful without baselines. So archive some performance data.
When called with no options or arguments, sar gives us a nice little summary of CPU activity. On multiprocessor systems, sar will report CPU activity for the systems as a whole, and for each processor if you use "sar -P ALL" This is useful to discover imbalances in CPU use which can then be tracked down to problems with applications or the operating system:
sar -P ALL
Look at these stats in conjunction with load and run queue stats, sar -q.
High loads can be caused by i/o bottlenecks and network problems such as unresponsive NFS or NIS servers or DNS problems.
The sar command writes to standard output the contents of selected cumulative activity counters in the operating system. The accounting system, based on the values in the count and interval parameters. For example display comparison of CPU utilization; 2 seconds apart; 5 times, use:
sar -u 2 5
To display memory utilization, use the -r option:
The 'kbmemfree' column shows the free memory available in KB at that time.
The 'kbmemused' column shows the memory used in KB at that time
The '%memused' column shows the percentage of memory used.
The 'kbbuffers' column shows the memory was used as buffers.
The 'kbcached' column shows the memory was used as cache
The 'kbswpfree' column shows the free swap space in KB at that time
The 'kbswpused' column shows swap space used in KB at that time
The '%swpused' column shows the percentage of swap used at that time
The 'kbswpcad' column shows cached swap in KB at that time.
To display paging related activity, use the -B option:
The 'pgpgin/s' column shows the amount of paging into the memory from disk, per second
The 'pgpgout/s' column shows the amount of paging out to the disk from memory, per second
The 'fault/s' column shows page faults per second
The 'majflt/s' column shows the major page faults per second.
To display swapping related activity, use the -W option:
The 'pswpin/s' column shows the number of pages of memory swapped back into the memory from disk, per second
The 'pswpout/s' column shows the number of pages of memory swapped out to the disk from memory, per second
To display disk device statistics, use the -d option:
The 'tps' column shows the number of transfers per second. Transfers are I/O operations. Note: this is just number of operations; each operation may be large or small. So, this, by itself, does not tell the whole story.
The 'rd_sec/s' column shows the number of sectors read from the disk per second.
The 'wr_sec/s' column shows the number of sectors written to the disk per second
The sar option for looking at block devices is -d (disk). The statistics given here are straightforward; transfers per second, 512 byte sectors read per second, and 512 byte sectors written per second.
The output from sar is different from that of df. sar list more devices than df. There are just two hard drives on this system, each formatted with a single file system. So what are all these devices? A look at /dev tells us that dev3-0 is hda, and dev3-64 is hdb. The devices labeled with major number 1 are all special block devices like /dev/null, ramdisk, /dev/zero, and so on. The 22's are cdrom drives, 2 is the floppy, and 9-0 is the software RAID device md0. So before these stats can be of much use to you, you'll have to identify the major and minor numbers of the drives you want to examine. ATA hard drives will have major number 3 on Linux systems, but the minor number will be different for each device. Check this by doing a listing on /dev:
ls -l /dev/hd*
The major and minor numbers are listed between the group ownership and the date. To cut through all of the extraneous listings for devices you don't care about, just run sar through grep with something like this:
sar -d | grep dev3-
Note, on old kernels, pre 2.5, use -b for the disk io report.
To display network statistics, you use the -n option: The -n option takes arguments DEV (devices), EDEV (error count for devices), SOCK (sockets), and ALL (all of the above).
sar -n DEV | more
To display network statistics from the 24th:
sar -n DEV -f /var/log/sa/sa24 | more
The output is pretty easy to parse visually. The three pairs of stats here are packets per second (received and transmitted), bytes per second (received and transmitted), compressed packets per second (received and transmitted), and the lone multicast packets received.
sar -n EDEV | more
The output for the EDEV option is wide, but it's often the first three numbers that you want — receive errors per second, transmit errors per second, and collisions per second.
Use the SOCK argument to see total number of sockets, and breakdowns of tcp, udp, raw, and the number of ip fragments in use. This is a nice quick way to see trends in socket useage and corralate with system changes.
To display run-queue statistics, use the -q option:
Guide to Advanced Linux Command Mastery, Part 3: Resource Management