Performance Analysis in solaris 10- Memory

In real world memory performance issues plays a major role in system performance. Unix system memory contains 2 types , one is physical memory which is attached to the DIMM modules of the hardware and second is swap space which is a dedicated space in the disk which is treated as a memory by OS ( since the disk I/O is much slower than the I/O to the memory in generic way we will prefer to use swap space less frequently as possible).
tivity
Swap space is only used when the physical memory is too small to accommodate system memory requirements . At that time space is freed in physical memory by paging (moving) it out to swap space ( also keep it in mind that if the increase in paging to swap space will degrade system CPU performance )

vmstat command
***********

vmstat reports virtual memory statics regarding kernel thread, virtual memory,disk,thread,cpu activity etc. Also note that in multiple CPU systems this will show the average of the number of CPU output. 

The details of the command is given below 

kthr - This indicates the kernel threads details in 3 states . 

    r - The number of kernel threads in run queue

    b - The number of blocked threads which are waiting for I/O paging 

    w - Number of swapped out lightweight processes (LWP - processes which is running under same kernel thread and shares its system resources and addresses with other lwp's)  that are waiting for resources to finish 

memory - Report the usage of real and virtual memory 

    swap - available swap space ( in kb)

    free - size of the free list (in kb)

page - Report about the page faults and paging activity . Details of this section is given below 

    re - page reclaims 

    mf - minor faults 

    pi - kilobytes paged in 

    po - kilobytes paged out

    fr - kilibytes freed

    de - anticipated short term memory shortfall ( in KB)

    sr- pages scanned by clock algorithm 

disk - Reports the number of disk operations per second . There are slots up to 4 disks with a letter and number ( letter indicates the disk type like scsi.ide ) and number is the logical number

faults - Reports trap/interrupt rates 

    in - interrupts

    sy - system calls

    cs- cpu context switches 

cpu - Breakdown usage of the cpu time. In multi processor systems it will be average of all the CPU's

    us - user time

    sys - system time

    id - idle time

swap usage analysis

****************

If you concentrate in swap analysis we need to use below mentioned two commands . 

bash-3.2# swap -s
total: 267032k bytes allocated + 86184k reserved = 353216k used, 964544k available

bash-3.2# swap -l
swapfile dev swaplo blocks free
/dev/zvol/dsk/rpool/swap 181,1 8 2097144 2097144

But there is major difference between these two commands , in the first one we are using 353216k of (964544 + 353216)k  which means of 26% in use. In the second one you can see as all the 2097144 is free means 0% is used.  In the first command (swap -s ) includes the  portion of physical memory also which is using as swap. The major difference in usage of these two commands are in generic if you are checking the swap usage over time you can use swap -s. (if the system performance is good). But if the system performance is degraded you need to concentrate more  about  the change in swap usage and what causes that change  ( also keep it in mind that swap -l displays output in 512 bytes and swap -s displays in 1024 byte blocks) .

If the system run's out of swap space it will show the error messages given below and we might think about expanding the same using creating the swap file. In general while creating the swap you have to provide size as half of the system physical memory

for example if the system memory is 8GB the ideal swap size should be 4GB

***********************************************************
application is out of memory

malloc error O

messages.1:Sep 21 20:52:11 mars genunix: [ID 470503 kern.warning]
WARNING: Sorry, no swap space to grow stack for pid 100295 (myprog)
***********************************************************

Creating the swap file

1.Login as super user

2. Create the swap file using mkfile <name> <size in k/m/g> filename

3. Activate the swap file using /usr/sbin/swap -a /path/filename

4. Add the entry at /etc/vfstab 

   /path/filename - - swap - no -

5. Verify the swap file using /usr/bin/swap -l

As a nutshell while configuring the swap , please keep it in below points 

• Never allocate swap with size less then 30% of RAM.
• Determine whether large applications (such as compilers and databases) will be using the /tmp  directory. If one or several of your application have a huge demand for swap space, use the swap -s  command to monitor swap resources on a similar existing system tro get estimate of the actual requirements.

Cache

If we check the free -m command in a unix box we can see major portion of the memory is in cached column. So what is mean by that cache, is it currently used by system?

[root@testserver ~]# free -m
             total       used       free     shared    buffers     cached
Mem:         15976      15195        781          0        167       9153
-/+ buffers/cache:       5874      10102
Swap:         2000          0       1999

In this case you can see 9GB is cached . These caches are called page caches / dirty caches which will act as a temporary memory for read and write process. During the write process the contents of these dirty cache will be periodically transferred to the system storage . Till 2.6.31 version , the process called pdflush will ensure that the data is transferring to system storage and clearing the dirty pages periodically. But after this kernel version there will be a thread for each device ( like sda/sdb) will monitor this mechanism 

root@pc:~# ls -l /dev/sda
brw-rw---- 1 root disk 8, 0 2011-09-01 10:36 /dev/sda
root@pc:~# ls -l /dev/sdb
brw-rw---- 1 root disk 8, 16 2011-09-01 10:36 /dev/sdb
root@pc:~# ps -eaf | grep -i flush
root       935     2  0 10:36 ?        00:00:00 [flush-8:0]
root       936     2  0 10:36 ?        00:00:00 [flush-8:16]

This same mechanism is applicable for reading also, file blocks will be transferred from disk to page cache for reading . For example if you access 100MB file twice , in second time the access will be faster as it is fetching from the cache. If Linux needs more memory for normal applications than is currently available, areas of the Page Cache that are no longer in use will be automatically deleted.

Mostly log files or database dump file ( data files) are mostly accumulated by page cache as it is accessed continuously . So configuring perfect log rotate or zipping it periodically will release the page cache when it will be really needed for system performance .