Veritas cluster server is the high availability cluster solution for open platform including Unix and MS. This can be used as high availability cluster and high performance cluster depending up on the requirement where high availability cluster (HAC) improve application availability by failing them over or switching them over in a group of systems and high performance cluster (HPC) which improve application performance by running them on multiple systems simultaneously.
VCS consist of multiple systems connected to a shared storage in different formations. VCS monitors and controls applications (resources) running in the system and restarts it by switching to defferent nodes to ensure minimum or zero downtime.
I am mentioning different VCS terms which will be used through out this blog for your understanding
Resources - Resources includes hardware and software entities which includes ip address, disks, scrips , network interfaces etc. controlling this resources means starting/stopping/monitoring the resources as per the requirement . These resources are controlled by resource dependencies where there will be specific procedure to make this resources online or offline.
Resources have 3 categories
on-off - These type of resources will be on or off with in the cluster as per the requirement . example scripts
on-only - This resources will be always on and VCS cannot stop them . example os NFS file systems
persistent - Persistent resources will not be make online or offline by VCS. example is network interface.
Service groups - Service groups are logical group of resources and dependencies for better management. Each node in the VCS might host multiple service groups which can be monitored separately. If any resource in a service group is failed VCS will restart the service group in same node itself or by moving to other node.
service groups are 3 types called
Failover Service group - running in one node at a time
Parallel Service group - running in multiple nodes at a time
Hybrid Service group - This is a combination of Failover & Parallel where with in the system zones it will act as Failover and Parallel across the zones .
Also there is one more service group called cluster service group which will contain main cluster resources like cluster manager,Notification,Wide area connectors (WAC) etc. This service group can switch to any nodes and it is the first service group came as online
High Availability Daemon (HAD)
HAD is the main daemon running in the VCS system which is responsible for running the cluster with cluster configuration , it will distribute the information when the nodes will join/leaves the cluster also it will respond to operator input and take corrective action when something fails . This daemon is generally known as VCS engine. HAD will use agents to understand the status of the resources and the daemon running in one system will replicate the details to all other nodes in VCS system
Low Latency Transport (LLT)
LLT provides faster communication between the cluster nodes which means kernel to kernel communication. LLT has two functions
Traffic distribution - LLT helps to load balance the traffic between the nodes and it supports maximum up to eight links. If one link is down communication will shift automatically to another node.
Heart Beat - LLT is responsible for sending and receiving heart beat communication between the nodes to make sure system health status is fine.This heartbeat is analysed by GAB to identify the system status
Main configuration files - /etc/llthosts & /etc/llttab
1. /etc/llthosts - This file contains hosts details with respective id's
ex: bash-3.00# cat /etc/llthosts
0 solaris-test2
1 solaris-test3
2. /etc/llttab - This file contains LLT network links related to each nodes in the system
ex:
bash-3.00# cat /etc/llttab
set-node solaris-test2
set-cluster 100
link e1000g0 /dev/e1000g:0 - ether - -
link e1000g1 /dev/e1000g:1 - ether - -
Group Membership Services/Atomic Broadcast (GAB)
GAB provides a global message order to provide synchronized system state. This also monitors disk communications which is required for VCS heartbeat utility . The main functions of GAB is to maintain cluster membership ( by receiving input of the status of the servers through LLT) and helps for reliable cluster communication
Main configuration file /etc/gabtab
This file contains the information needed by gabdriver and it is accessed by gabconfig utility
ex:
bash-3.00# cat /etc/gabtab
/sbin/gabconfig -c -n2
where 2 is the number of nodes available in the cluster system
Configuring disk heartbeat
This is same as qdisk configuration in RHEL cluster. We require at least 64kb of a disk for this configuration
Below commands will initialize the disk region (s - start block, S- signature)
#/sbin/gabdiskconf - i /dev/dsk/c1t2d0s2 -s 16 -S 1123
#/sbin/gabdiskconf - i /dev/dsk/c1t2d0s2 -s 144 -S 1124
Adding the disk configuration
#sbin/gabdiskhb -a /dev/dsk/c1t2d0s2 -s 16 -p a -s 1123
#sbin/gabdiskhb -a /dev/dsk/c1t2d0s2 -s 144 -p h -s 1124
configure the disks to use the driver and initialize
#sbin/gabconfig -c -n2
LLT commands
lltstat -n command will show the llt link status in each node
in server1-
******************************************************
bash-3.00# /sbin/lltstat -n
LLT node information:
Node State Links
* 0 solaris-test2 OPEN 2
1 solaris-test3 OPEN 2
**************************************************************
in server2
*****************************************************
bash-3.00# /sbin/lltstat -n
LLT node information:
Node State Links
0 solaris-test2 OPEN 2
* 1 solaris-test3 OPEN 2
***********************************************************
lltstat -nvv |more command will show in verbose format
***************************************************
bash-3.00# /sbin/lltstat -nvv
LLT node information:
Node State Link Status Address
* 0 solaris-test2 OPEN
e1000g0 UP 08:00:27:F2:FD:1B
e1000g1 UP 08:00:27:A4:52:AF
1 solaris-test3 OPEN
e1000g0 UP 08:00:27:19:48:CB
e1000g1 UP 08:00:27:76:37:96
2 CONNWAIT
e1000g0 DOWN
e1000g1 DOWN
3 CONNWAIT
e1000g0 DOWN
e1000g1 DOWN
***************************************************
To start & stop llc use below commands
***************************************************
#lltconfig -c
#lltconfig -U ( Keep it in mind that GAB has to be stopped before this command)
GAB commands
1. Define the group membership and atomic broadcast (GAB) configuration
# more /etc/gabtab
/sbin/gabconfig -c -n2
2. start GAB
# sh /etc/gabtab
starting GAB done.
3. Display the GAB details
bash-3.00# /sbin/gabconfig -a
GAB Port Memberships
===============================================================
Port a gen f2503 membership 01
Port h gen f2505 membership 01
VCS configuration
Configuring VCS means communicating VCS engine about the name & definition of the cluster,service groups, and resource dependencies The VCS configuration file is located /etc/VRTSvcs/conf/config named main.cf & types.cf .Here the main.cf file will define entire cluster where types.cf includes resource types .In VCS the first system comes online will reads the configuration file and keeps the entire configuration in memory and system which are online later will derive the information .
sample main.cf file
********************************************************************************
bash-3.00# cat /etc/VRTSvcs/conf/config/main.cf
include "OracleASMTypes.cf"
include "types.cf"
include "Db2udbTypes.cf"
include "OracleTypes.cf"
include "SybaseTypes.cf"
cluster unixchips (
UserNames = { admin = dqrJqlQnrMrrPzrLqo }
Administrators = { admin }
)
system solaris-test2 (
)
system solaris-test3 (
)
group unixchipssg (
SystemList = { solaris-test2 = 0, solaris-test3 = 1 }
AutoStartList = { solaris-test2 }
)
DiskGroup unixchipsdg (
Critical = 0
DiskGroup = unixchipsdg
)
Mount unixchipsmount (
Critical = 0
MountPoint = "/vcvol1/"
BlockDevice = "/dev/vx/dsk/unixchipsdg/vcvol1"
FSType = vxfs
FsckOpt = "-y"
)
Volume unixchipsvol (
Critical = 0
Volume = vcvol1
DiskGroup = unixchipsdg
)
unixchipsmount requires unixchipsvol
unixchipsvol requires unixchipsdg
// resource dependency tree
//
// group unixchipssg
// {
// Mount unixchipsmount
// {
// Volume unixchipsvol
// {
// DiskGroup unixchipsdg
// }
// }
// }
*********************************************************************************
Types. cf file
****************
bash-3.00# cat /etc/VRTSvcs/conf/config/types.cf
type AlternateIO (
static str AgentFile = "bin/Script51Agent"
static str ArgList[] = { StorageSG, NetworkSG }
str StorageSG{}
str NetworkSG{}
)
type Apache (
static boolean IntentionalOffline = 0
static keylist SupportedActions = { "checkconffile.vfd" }
static str ArgList[] = { ResLogLevel, State, IState, httpdDir, SharedObjDir, EnvFile, PidFile, HostName, Port, User, SecondLevelMonitor, SecondLevelTimeout, ConfigFile, EnableSSL, DirectiveAfter, DirectiveBefore }
static int ContainerOpts{} = { RunInContainer=1, PassCInfo=0 }
str ResLogLevel = INFO
str httpdDir
str SharedObjDir
str EnvFile
str PidFile
str HostName
int Port = 80
str User
boolean SecondLevelMonitor = 0
int SecondLevelTimeout = 30
str ConfigFile
boolean EnableSSL = 0
str DirectiveAfter{}
str DirectiveBefore{}
)
................. output is omitted
*********************************************************************************
The setup
As per the above architecture diagram we have 2 solaris servers and openfiler storage for the sample set up. Here i am not covering the installation part which will be explaining in a separate post by. soon. The sample service will be a vxfs mount point which is configured as a service in VCS setup .
1. Once the HA installation is completed we will get the status as both the nodes are online mode as given below
bash-3.00# hastatus -sum
-- SYSTEM STATE
-- System State Frozen
A solaris-test2 RUNNING 0
A solaris-test3 RUNNING 0
Now we need to create the diskgroup using the vxfs file system .
1. First check the available disks in the server using below command ( identical output for both the nodes)
bash-3.00# echo |format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c0d0 <DEFAULT cyl 2085 alt 2 hd 255 sec 63>
/pci@0,0/pci-ide@1,1/ide@0/cmdk@0,0
1. c0d1 <VBOX HAR-34e30776-506a21a-0001-1.01GB>
/pci@0,0/pci-ide@1,1/ide@0/cmdk@1,0
2. c1d1 <VBOX HAR-b00f936a-669c3f0-0001-1.01GB>
/pci@0,0/pci-ide@1,1/ide@1/cmdk@1,0
3. c2t0d0 <VBOX-HARDDISK-1.0-1.17GB>
/pci@0,0/pci1000,8000@14/sd@0,0
4. c2t1d0 <VBOX-HARDDISK-1.0-1.06GB>
/pci@0,0/pci1000,8000@14/sd@1,0
5. c3t3d0 <OPNFILER-VIRTUAL-DISK-0 cyl 44598 alt 2 hd 16 sec 9>
/iscsi/disk@0000iqn.2006-01.com.openfiler%3Atsn.7d10272607c20001,0
6. c3t4d0 <OPNFILER-VIRTUAL-DISK-0 cyl 63258 alt 2 hd 16 sec 9>
/iscsi/disk@0000iqn.2006-01.com.openfiler%3Atsn.0a610808dede0001,0
bash-3.00# vxdisk list
DEVICE TYPE DISK GROUP STATUS
c0d0s2 auto:none - - online invalid
c0d1 auto:ZFS - - ZFS
c1t1d0s2 auto - - error
c2t0d0 auto:ZFS - - ZFS
c2t1d0 auto:ZFS - - ZFS
disk_0 auto:cdsdisk iscsi1 ---------------- online invalid
disk_1 auto:cdsdisk iscsi2 ---------------- online invalid
As per the above command we can see 2 iscsi disks from the storage which is shown as online but invalid . These disks we will configure for the diskgroup .
2. Bring open filer disks under vxvm
bash-3.00 # /etc/vx/bin/vxdisksetup -i disk_0
bash-3.00# /etc/vx/bin/vxdisksetup -i disk_1
Now we can see both the disks are online status
3. bash-3.00# vxdisk list
DEVICE TYPE DISK GROUP STATUS
c0d0s2 auto:none - - online invalid
c0d1 auto:ZFS - - ZFS
c1t1d0s2 auto - - error
c2t0d0 auto:ZFS - - ZFS
c2t1d0 auto:ZFS - - ZFS
disk_0 auto:cdsdisk iscsi1 ---------------- online
disk_1 auto:cdsdisk iscsi2 ---------------- online
4. We have to create the diskgroup with thease online disks now
bash-3.00#vxdg init unixchipsdg iscsi1=disk_0 iscsi2=disk_1
We can see the disk group is created (unixchipsdg) for these iscsi disks
bash-3.00# vxdisk list
DEVICE TYPE DISK GROUP STATUS
c0d0s2 auto:none - - online invalid
c0d1 auto:ZFS - - ZFS
c1t1d0s2 auto - - error
c2t0d0 auto:ZFS - - ZFS
c2t1d0 auto:ZFS - - ZFS
disk_0 auto:cdsdisk iscsi1 unixchipsdg online
disk_1 auto:cdsdisk iscsi2 unixchipsdg online
5. We need to create the volume using this diskgroup
bash-3.00#vxassist -g unixchipsdg make vcvol1 2g
6. Format the volume to create the file system
bash-3.00#mkfs -F vxfs /dev/vx/rdsk/unixchipsdg/vcvol1
version 7 layout 6291456 sectors,
3145728 blocks of size 1024, log size 16384 blocks
largefiles supported
7.Now you can try to mount it for confirmation after creating the mount point
bash-3.00#mkdir /vcvol1
bash-3.00#mount -F vxfs /dev/vx/dsk/unixchipsdg/vcvol1 /vcvol1
bash-3.00#df -h /vcvol1
Filesystem size used avail capacity Mounted on
/dev/vx/dsk/unixchipsdg/vcvol12.0G 18M 1.8G 1% /vcvol1
8. The next step is to create the service group/volume/diskgroup . The order of creation will be service group/volume/mount
Creating the servicegroup
**********************
# haconf -makerw (this will make the configuration file ie main.cf to read write mode)
#hagrp -add unixchipssg
# hagrp -modify unixchipssg SystemList solar is-test2 0 solaris-test3 1
# hagrp -modify unixchipssg AutoStartList solaris-test2
# hagrp -display unixchipssg
#Group Attribute System Value
unixchipssg AdministratorGroups global
unixchipssg Administrators global
unixchipssg Authority global 0
unixchipssg AutoFailOver global 1
unixchipssg AutoRestart global 1
unixchipssg AutoStart global 1
unixchipssg AutoStartIfPartial global 1
unixchipssg AutoStartList global solaris-test2
unixchipssg AutoStartPolicy global Order
unixchipssg ClusterFailOverPolicy global Manual
unixchipssg ClusterList global
unixchipssg ContainerInfo global
unixchipssg DisableFaultMessages global 0
unixchipssg Evacuate global 1
unixchipssg ExtMonApp global
unixchipssg ExtMonArgs global
unixchipssg FailOverPolicy global Priority
unixchipssg FaultPropagation global 1
unixchipssg Frozen global 0
unixchipssg GroupOwner global
.........output will be omitted
#haconf -dump ( to update the configuration in main.cf file)
#view /etc/VRTSvcs/conf/config/main.cf
# cat /etc/VRTSvcs/conf/config/main.cf
include "OracleASMTypes.cf"
include "types.cf"
include "Db2udbTypes.cf"
include "OracleTypes.cf"
include "SybaseTypes.cf"
cluster unixchips (
UserNames = { admin = dqrJqlQnrMrrPzrLqo }
Administrators = { admin }
)
system solaris-test2 (
)
system solaris-test3 (
)
group unixchipssg (
SystemList = { solaris-test2 = 0, solaris-test3 = 1 }
AutoStartList = { solaris-test2 }
)
Adding the Diskgroup
********************
# hares -add unixchipsdg DiskGroup unixchipssg
# hares -modify unixchipsdg Critical 0
# hares -modify unixchipsdg DiskGroup unixchipsdg
# hares -modify unixchipsdg Enabled 1
# hares -online unixchipsdg -sys solaris-test2
# hares -state unixchipsdg
#Resource Attribute System Value
unixchipsdg State solaris-test2 ONLINE
unixchipsdg State solaris-test3 OFFLINE
#haconf -dump
Adding the volume resource
***********************
# hares -add unixchipsvol Volume unixchipssg
# hares -modify unixchipsvol Critical 0
# hares -modify unixchipsvol Volume vcvol1 ( this is vxfs volume we have created earler )
# hares -modify unixchipsvol DiskGroup unixchipsdg
# hares -modify unixchipsvol Enabled 1
# hares -display unixchipsvol (display the volume status with attributes)
#Resource Attribute System Value
unixchipsvol Group global unixchipssg
unixchipsvol Type global Volume
unixchipsvol AutoStart global 1
unixchipsvol Critical global 0
unixchipsvol Enabled global 1
unixchipsvol LastOnline global solaris-test3
unixchipsvol MonitorOnly global 0
unixchipsvol ResourceOwner global
unixchipsvol TriggerEvent global 0
unixchipsvol ArgListValues solaris-test2 Volume 1 vcvol1 DiskGroup 1 unixchipsdg
unixchipsvol ArgListValues solaris-test3 Volume 1 vcvol1 DiskGroup 1 unixchipsdg
unixchipsvol ConfidenceLevel solaris-test2 0
unixchipsvol ConfidenceLevel solaris-test3 100
unixchipsvol ConfidenceMsg solaris-test2
unixchipsvol ConfidenceMsg solaris-test3
unixchipsvol Flags solaris-test2
unixchipsvol Flags solaris-test3
unixchipsvol IState solaris-test2 not waiting
unixchipsvol IState solaris-test3 not waiting
unixchipsvol MonitorMethod solaris-test2 Traditional
unixchipsvol MonitorMethod solaris-test3 Traditional
................................out put is omitted
# hares -online unixchipsvol -sys solaris-test2
# hares -state unixchipsvol
#Resource Attribute System Value
unixchipsvol State solaris-test2 ONLINE
unixchipsvol State solaris-test3 OFFLINE
#haconf -dump
Adding the mount point resource
*******************************
# hares -add unixchipsmount mount unixchipssg
# hares -modify unixchipsmount Critical 0
# hares -modify unixchipsmount MountPoint /vcvol1
# hares -modify unixchipsmount BlockDevice /dev/vx/dsk/unixchipsdg/vcvol1
# hares -modify unixchipsmount FSType vxfs
# hares -modify unixchipsmount FSCKopt %-y
# hares -modify unixchipsmount Enabled 1
# hares -display unixchipsmount
#Resource Attribute System Value
unixchipsmount Group global unixchipssg
unixchipsmount Type global Mount
unixchipsmount AutoStart global 1
unixchipsmount Critical global 0
unixchipsmount Enabled global 1
unixchipsmount LastOnline global solaris-test2
unixchipsmount MonitorOnly global 0
unixchipsmount ResourceOwner global
unixchipsmount TriggerEvent global 0
unixchipsmount ArgListValues solaris-test2 MountPoint 1 /vcvol1/ BlockDevice 1 /dev/vx/dsk/unixchipsdg/vcvol1 FSType 1 vxfs MountOpt 1 "" FsckOpt 1 -y SnapUmount 1 0 CkptUmount 1 1 OptCheck 1 0 CreateMntPt 1 0 MntPtPermission 1 "" MntPtOwner 1 "" MntPtGroup 1 "" AccessPermissionChk 1 0 RecursiveMnt 1 0 VxFSMountLock 1 1
unixchipsmount ArgListValues solaris-test3 MountPoint 1 /vcvol1/ BlockDevice 1 /dev/vx/dsk/unixchipsdg/vcvol1 FSType 1 vxfs MountOpt 1 "" FsckOpt 1 -y SnapUmount 1 0 CkptUmount 1 1 OptCheck 1 0 CreateMntPt 1 0 MntPtPermission 1 "" MntPtOwner 1 "" MntPtGroup 1 "" AccessPermissionChk 1 0 RecursiveMnt 1 0 VxFSMountLock 1 1
...............................................output will be omitted
# hares -online unixchipsmount -sys solaris-test2
# hares -state unixchipsmount
#Resource Attribute System Value
unixchipsmount State solaris-test2 ONLINE
unixchipsmount State solaris-test3 OFFLINE
Linking the resources to the service group
********************************
# hares -link unixchipsvol unixchipsdg
# hares -link unixchipsmount unixchipsvol
9. Testing the resources
checking the status
# hastatus -sum
-- SYSTEM STATE
-- System State Frozen
A solaris-test2 RUNNING 0
A solaris-test3 RUNNING 0
-- GROUP STATE
-- Group System Probed AutoDisabled State
B unixchipssg solaris-test2 Y N ONLINE
B unixchipssg solaris-test3 Y N OFFLINE
checking the service status ( in this case mount point)
# df -h |grep vcvol1
/dev/vx/dsk/unixchipsdg/vcvol1 2.0G 19M 1.9G 1% /vcvol1
switching the service to another node
# hagrp -switch unixchipssg -to solaris-test3
# hastatus -sum
-- SYSTEM STATE
-- System State Frozen
A solaris-test2 RUNNING 0
A solaris-test3 RUNNING 0
-- GROUP STATE
-- Group System Probed AutoDisabled State
B unixchipssg solaris-test2 Y N OFFLINE
B unixchipssg solaris-test3 Y N ONLINE
(imp: the default location of the vcs commands will be /opt/VRTSvcs/bin/ and we need to export the path to avoid the full path details during execution in /etc/profile)
PATH=$PATH:/usr/sbin:/opt/VRTS/bin
export PATH
Thank you for reading the article. please feel free to post your comments and suggestions
