// // Monitoring System Performance Using SAR

A technical troubleshooting blog about Oracle with other Databases & Cloud Technologies.

Monitoring System Performance Using SAR

4 min read
SAR is used to collect, report or save system activity.


SAR stands for System Activity Report. SAR is used to check Memory, CPU, system load, network, swap space utilization, etc. It uses a proc file system to collect the information.

The sar command writes to standard output the contents of selected cumulative activity counters in the operating system. All data is captured in binary form and saved to a file (data.file). The data can then be selectively displayed with the sar command using the -f option.

SAR is provided by sysstat package, which also provides other statistical reporting tools, such as vmstat, iostat, netstat etc. Please note that sysstat package is not installed by default.

Using sar command is best way check in case of any performance issue report:

* RAM
* Swap
* Buffer/Cache
* Run Queue
* CPU
* Network
  • Check if sysstat package is installed on the server.
# rpm -qa | grep sysstat
If package does not exist, use the below command to install sysstat package.
yum install sysstat
Verify the sar version.
# sar -V
sysstat version 11.7.3
Create sysstat file under /etc/default/sysstat directory and add ENABLED=”true”
# vi /etc/default/sysstat
Enabled="true"
Finally enable and start the sysstat service
# systemctl enable sysstat
# systemctl start sysstat
sar would be now be successfully installed the Linux machine. It’s time to check the real-time reports by the sar command.
SAR Command usage:

Usage of command #sar without any argument on the terminal will show the real-time output for any activity running on the Linux machine using the count and interval parameter.
# sar
Linux 5.4.17-2136.300.7.el8uek.x86_64              01/19/2023 _x86_64_ (4 CPU)

12:00:19 AM CPU %user %nice %system %iowait %steal %idle
12:10:24 AM all 1.90 0.00 0.52 0.02 0.00 97.55
12:20:32 AM all 1.72 0.00 0.44 0.01 0.00 97.82
12:30:20 AM all 1.84 0.19 0.53 0.04 0.00 97.40
12:40:18 AM all 1.63 0.00 0.43 0.02 0.00 97.92
12:50:32 AM all 1.74 0.00 0.47 0.01 0.00 97.77
01:00:32 AM all 1.78 0.00 0.45 0.01 0.00 97.76
01:10:06 AM all 1.68 0.00 0.45 0.01 0.00 97.86
01:20:32 AM all 1.82 0.00 0.45 0.03 0.00 97.71
01:30:25 AM all 1.81 0.00 0.46 0.01 0.00 97.72
01:40:32 AM all 1.86 0.00 0.48 0.02 0.00 97.65
01:50:19 AM all 1.71 0.00 0.43 0.01 0.00 97.85
02:00:03 AM all 1.79 0.02 0.46 0.02 0.00 97.71
02:10:11 AM all 1.76 0.00 0.48 0.02 0.00 97.73
02:20:32 AM all 1.68 0.00 0.43 0.04 0.00 97.86
02:30:32 AM all 1.83 0.00 0.46 0.01 0.00 97.70
02:40:32 AM all 1.78 0.00 0.46 0.04 0.00 97.72
Current data can also be obtained by use of below :

Where,
interval is an interval in seconds.
iterations is a number of times to make and output statistics.
# sar <interval> <iterations>

# sar 1 5
Linux 5.4.17-2136.300.7.el8uek.x86_64  01/19/2023 _x86_64_ (4 CPU)

03:05:06 PM CPU %user %nice %system %iowait %steal %idle
03:05:07 PM all 0.25 0.00 1.24 0.00 0.00 98.51
03:05:08 PM all 0.50 0.00 0.25 0.00 0.00 99.25
03:05:09 PM all 0.25 0.00 0.50 0.00 0.00 99.25
03:05:10 PM all 0.25 0.00 0.25 0.00 0.00 99.50
03:05:11 PM all 0.25 0.00 0.50 0.00 0.00 99.25
Average: all 0.30 0.00 0.55 0.00 0.00 99.15
Below table provides a few option used with sar command:

-A	All
-b	I/O
-d	I/O for each block device
-n	network statistics
-q	processor queue
-r	Memory and swap
-u	CPU details
-v	Kernel
-w	simplified swap details
-B	swap
Examples to check system performance statistics

Example 1: To View memory statistics
# sar -r 1 10
Linux 3.11.12.ol6.x86_64 (server1) 02/24/2014 _x86_64_ (4 CPU)

09:54:56 AM kbmemfree        kbmemused   %memused kbbuffers kbcached  kbcommit             %commit
09:54:57 AM    193624        7784500     97.57    389504    2280488   9644972               60.44
09:54:58 AM    193608        7784516     97.57    389504    2280456   9644972               60.44
09:54:59 AM    193312        7784812     97.58    389504    2280504   9644972               60.44
09:55:00 AM    193204        7784920     97.58    389508    2281068   9645356               60.45
09:55:01 AM    132436        7845688     98.34    389508    2281356   9659980               60.54
09:55:02 AM    127380        7850744     98.40    389516    2281440   9659520               60.53
09:55:03 AM    126976        7851148     98.41    389516    2281068   9659036               60.53
09:55:04 AM    126308        7851816     98.42    389516    2281068   9659036               60.53
09:55:05 AM    126744        7851380     98.41    389516    2281092   9657976               60.52
09:55:06 AM    187128        7790996     97.65    389516    2281092   9656576               60.52
Average:       160072        7818052     97.99    389511    2280963   9653240               60.49
To calculate free memory from Average value do:

kbmemfree + kbbuffers + kbcached = actual free memory on the system

Using above example: 
160072 + 389511 + 2280963 = 2830546KB that is around 2.69GB free memory.
Calculate memory usage using free command
# free -m
            total       used       free     shared    buffers     cached
Mem:          7791       7623        167          0        380       2227
-/+ buffers/cache:       5016       2775         
Swap:         7791        655       7136
From above example, to get free memory on the system use: -/+ buffers/cache - free column

Free column reports 2775 MB so that match as well SAR output calculation ( keeping in mind that in SAR Average value was used )

To get used memory on system use:  -/+ buffers/cache - used column
Check which most Top 10 users are consuming memory on system in percentage
# ps -eo user,pcpu,pmem | tail -n +2 | awk '{num[$1]++; cpu[$1] += $2; mem[$1] += $3} END{printf("NPROC\tUSER\tCPU\tMEM\n"); for (user in cpu) printf("%d\t%s\t%.2f\t%.2f\n",num[user], user, cpu[user], mem[user]) }'

NPROC  USER CPU MEM
15      gdm 0.00 0.10
6   oracle 2.10 18.00
1       rpc 0.00 0.00
1      dbus 0.00 0.00
2        68 0.00 0.00
1     rtkit 0.00 0.00
2   postfix 0.00 0.00
1   rpcuser 0.00 0.00
295  root 18.50 42.40
From the above example observation, root has 295 processes which are taking 42% of memory, user oracle has 6 processes which are taking 18% of memory, that in total around 60%

60% value can be also seen on sar 'commit' column.
Example 2: To view I/O statistics
#sar -b 1 6
Linux 4.1.12-124.67.3.el7uek.x86_64              01/19/2023 _x86_64_ (12 CPU)

09:51:49 AM tps rtps wtps bread/s bwrtn/s
09:51:50 AM 4.00 0.00 4.00 0.00 24.00
09:51:51 AM 4.00 0.00 4.00 0.00 18.00
09:51:52 AM 0.00 0.00 0.00 0.00 0.00
09:51:53 AM 0.00 0.00 0.00 0.00 0.00
09:51:54 AM 0.00 0.00 0.00 0.00 0.00
09:51:55 AM 24.00 0.00 24.00 0.00 184.00
Average: 5.33 0.00 5.33 0.00 37.67
Example 3: To view each Disk statistics
#sar -d 1 2
Linux 4.1.12-124.67.3.el7uek.x86_64  01/19/2023 _x86_64_ (12 CPU)

09:54:41 AM DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util
09:54:42 AM dev202-0 50.00 16.00 90.00 2.12 0.04 0.90 0.34 1.70
09:54:42 AM dev202-16 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
09:54:42 AM dev249-0 1.00 16.00 0.00 16.00 0.01 6.00 6.00 0.60
09:54:42 AM dev249-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
09:54:42 AM dev249-2 45.00 0.00 90.00 2.00 0.04 0.87 0.24 1.10
09:54:42 AM dev249-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
09:54:42 AM dev249-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
09:54:42 AM dev249-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
09:54:42 AM dev249-6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
 Example 4: To view CPU statistics:
#sar -u 2 5
Linux 4.1.12-124.67.3.el7uek.x86_64  01/19/2023 _x86_64_ (12 CPU)

10:04:49 AM CPU %user %nice %system %iowait %steal %idle
10:04:51 AM all 4.89 0.00 3.30 0.00 0.00 91.81
10:04:53 AM all 0.42 0.00 0.38 0.00 0.00 99.21
10:04:55 AM all 0.04 0.00 0.08 0.04 0.00 99.83
10:04:57 AM all 0.21 0.00 0.13 0.04 0.00 99.62
10:04:59 AM all 0.08 0.00 0.13 0.00 0.00 99.79
Average: all 1.13 0.00 0.80 0.02 0.00 98.06